• English
  • 3.9.0

    • Overview

    In the Toolchain Overview section, we introduced the basic concepts of PTQ and QAT to help you build an initial understanding of these two quantization methods. Next, we will further walk you through the practical usage flow so that you can understand how model quantization and compilation are used in real projects.

    If this is your first time using the toolchain, we recommend that you choose the entry based on the type of model you currently have:

    • If you already have an ONNX model, we recommend that you start with the ONNX Model Quantization section.

    • If you already have a PyTorch model and training code, we recommend that you start with the PyTorch Model Quantization section.

    ONNX Model Quantization

    The overall ONNX model quantization pipeline is shown below:

    This pipeline is based on the horizon_tc_ui command-line tool or the HMCT + HBDK API, and completes graph optimization, calibration, quantization, and compilation in one streamlined flow. It is suitable for scenarios where the model has already been exported and you want to quickly complete structure verification, quantization and compilation, performance evaluation, and accuracy evaluation. It is also a good default entry for first-time users or models with relatively simple structures.

    We recommend that you first read ONNX Model Quick Start and ONNX Model Quantization Overview for a basic understanding.

    PyTorch Model Quantization

    The overall PyTorch model quantization pipeline is shown below:

    This pipeline is built on the open-source PyTorch framework. It provides quantization capability through the plugin and uses the HBDK compiler API to complete model fixed-point conversion and compilation. This pipeline also supports further finetuning after calibration. Although the entry cost and training cost are relatively higher, it also offers a higher upper bound on accuracy, making it more suitable for iterative optimization around model structure, quantization configuration, and training strategy.

    We recommend that you first read PyTorch Model Quick Start, QAT Conversion Introduction, and PyTorch Model Quantization Basic Process for a basic understanding.

    Basic Differences Between the Two Pipelines

    ItemONNX Model QuantizationPyTorch Model Quantization
    Main inputONNX modelPyTorch float model
    Typical goalQuickly verify whether the model can be quantized, compiled, and deployedContinuously optimize accuracy through quantization configuration, calibration, and training
    Common starting pointRun through the standard pipeline firstStart with a minimal Calibration loop
    Follow-up pathIf accuracy is not sufficient, continue with accuracy tuningIf Calibration is acceptable, export directly; otherwise continue with QAT
    More suitable rolesBeginners, deployment engineersAlgorithm engineers, quantization tuning engineers
    Note

    We recommend that you first read the Platform Differences section to understand the differences across platforms in quantization strategy, output precision, input/output handling, and deployment capability.

    For model quantization and deployment verification on different platforms, taking the original PyTorch framework model as an example, the overall recommended flow is shown below:

    You may choose your path according to the following suggestions:

    1. If your current goal is to complete model structure verification, quick performance validation, or deployment feasibility validation, we recommend that you choose the lighter path first.

    2. If accuracy is still not sufficient after quick validation, you can then continue with accuracy tuning or the QAT path for further optimization.

    3. No matter which path you choose, you should always return to deployment-side result verification in the end, including quantized.bc / hbm accuracy, performance, and consistency checks.

    Choose the lighter path for quick validation

    If your goal is to first confirm that the model can be quantized, compiled, and basically verified, we recommend that you choose the lighter path first:

    • For an ONNX model, start with the ONNX Model Quantization pipeline.

    • For a PyTorch model, start by completing one minimal Calibration loop.

    Move to QAT for formal accuracy iteration

    If accuracy is still not sufficient after quick validation, you can then move to a more complete optimization path:

    In the following sections, we will further introduce ONNX model quantization, PyTorch model quantization, model modification scenarios that may be involved in model conversion and compilation, as well as further operations in the X86 simulation environment.