To meet different deployment requirements and enable efficient model deployment and operation on edge computing platforms, Horizon provides two quantization solutions based on whether the quantized parameters need further adjustment: Post-training Quantization(PTQ) and Quantized Awareness Training(QAT).
The operational differences between these two methods are shown in the figure below (PTQ on the left, QAT on the right):

Post-training Quantization(PTQ) uses a batch of calibration data to calibrate a trained model, directly converting a trained FP32 model into a fixed-point model. No further training of the original model is required; only a few hyperparameters need to be adjusted to complete the quantization process. This method is simple, fast, and does not require retraining, so it has been widely used in many edge and cloud deployment scenarios. We recommend trying PTQ first to see if it meets your deployment accuracy and performance requirements.

Quantized Awareness Training(QAT) involves retraining the model after quantization. Since fixed-point values cannot be used for backward gradient calculation, in practice, fake quantization nodes are inserted before certain operators to collect truncated values of data flowing through these operators during training. This helps with quantization of nodes when deploying the quantized model. During training, accuracy is continuously optimized to obtain the best quantization parameters. As QAT requires model retraining, it has higher technical requirements for operators.

#PTQ vs QAT Pipeline Comparison

Although the J6 toolchain supports both PTQ and QAT model conversion pipelines, each has its own focus in terms of implementation tools, workflow, performance, and applicable scenarios. The core differences are as follows:

PTQ: Based on the hb_compile command-line tool, you only need to configure the yaml file and calibration dataset to complete model graph optimization, calibration, quantization, and compilation in one step. This quantization method is fast and easy to use, but offline quantization based solely on mathematical statistics is not conducive to model iteration and may cause difficult corner cases. Therefore, it is usually used for early model evaluation or quantization of simple model structures.

QAT: Based on the PyTorch open-source framework, model quantization is provided as a plugin, and the hbdk compiler API is called to implement model fixed-point conversion and compilation. This pipeline supports further finetune training after model calibration. Although it is more difficult to get started and has higher training costs, it also offers higher accuracy ceilings and is more suitable for iterative model optimization.

#Recommended Workflow

Based on the above summary and comparison of PTQ and QAT features and pros/cons, we recommend the following toolchain workflow for quantizing floating-point models on different platforms:

1. First, export the floating-point ONNX model (opset10~19). It is recommended to first use the PTQ pipeline for rapid model structure verification and full int8 performance evaluation. If both performance and accuracy meet expectations, you can proceed directly to board deployment; if accuracy does not meet expectations, you can further perform PTQ accuracy tuning.

   • On the J6 B/E/M platforms, perform accuracy tuning directly under int8+int16 mixed precision until accuracy meets expectations, and then proceed to board deployment.

   • On the J6 H/P platforms, perform accuracy tuning directly under int8+int16+float16 mixed precision until accuracy meets expectations, and then proceed to board deployment.

2. If the PTQ pipeline cannot solve special model accuracy issues, switch to the QAT pipeline for quantization. Similarly, it is recommended to first complete model structure verification and full int8 performance evaluation. If both performance and accuracy meet expectations, you can proceed directly to board deployment; if accuracy does not meet expectations, you can further perform QAT accuracy tuning.

   • On the J6 B/E/M platforms, first train to the expected accuracy under full int16 configuration, then gradually reduce the int16 ratio to achieve a balance of performance and accuracy under int8+int16 mixed precision, and finally proceed to board deployment.

   • On the J6 H/P platforms, first train to the expected accuracy under full int16+float16 configuration, then gradually reduce the int16+float16 ratio to achieve a balance of performance and accuracy under int8+int16+float16 mixed precision, and finally proceed to board deployment.

This is the recommended workflow. You can flexibly choose PTQ or QAT solutions according to your actual needs. In subsequent sections, we will provide detailed introductions to Post-training Quantization(PTQ), Quantized Awareness Training(QAT), common model modification scenarios during model conversion and compilation, and further operations in the x86 simulation environment.