#Float Model / Floating-point Model

A floating-point model that meets the requirements of quantization aware training.

#Calibration

The process of obtaining quantization parameters using calibration data.

#Calibration Model

A pseudo-quantized model obtained after Calibration.

#QAT / Quantized Awareness Training

Quantized Awareness Training, or QAT, refers to simulating low-precision computation during model training or fine-tuning in order to reduce the accuracy loss introduced by model quantization. Because fixed-point values cannot be used directly for backward gradient computation, actual training usually does not apply real quantization to the model. Instead, fake quantization nodes are inserted before certain operators to simulate quantization behavior during the forward pass, while also collecting truncation ranges and quantization parameters.

In this way, the model can be fine-tuned under low-precision conditions closer to actual deployment. By continuously optimizing accuracy, it can obtain better quantization parameters and improve the final result of the quantized model. Because this process involves joint optimization of model training and quantization parameters, it places higher technical demands on the user.

#QAT Model

A pseudo-quantized model obtained after quantization aware training.

#Pseudo-quantization

The process of first quantizing and then dequantizing floating-point data. In network models, this is generally implemented through fake quantization nodes.

#Pseudo-quantized Model

A model with fake quantization nodes, usually obtained through Calibration or QAT.

#Quantized Model / Fixed-point Model / Quantization Model

By converting the floating-point parameters in a pseudo-quantized model into fixed-point parameters, and converting floating-point operators into fixed-point operators, the resulting transformed model is called a Quantized model, a fixed-point model, or a quantization model.

#HBIR Model

A model exported for deployment purposes, usually exported from a QAT model, and typically used for accuracy simulation and board-side compilation.

#Quantization Principles

The core idea of model quantization is to use low-precision values, such as INT8, to represent and compute model parameters and activations that were originally trained with high-precision values, such as FP32.

The basic steps for quantizing a tensor are as follows:

• Collect the value range of the tensor, and use it to determine the corresponding scale.

• Scale the tensor with that scale so that the scaled values fall within the representable range of the low-precision dtype, and then cast the result to the low-precision dtype.

Using int8 quantization as an example, the computation is: $quantized\_x = clip(round(x / scale), -128, 127)$.

#Static Quantization VS Dynamic Quantization

Static quantization requires collecting the activation distributions at different positions in the model over a dataset, and computing a fixed scale for each activation. This process is called calibration. Dynamic quantization does not require a calibration step. Instead, it collects the value range of the current tensor during model inference, computes the scale in real time, and completes the conversion to low precision on the fly.

#Supported Operator Range

Due to BPU constraints on operators, only the operators listed in Supported Operator List, as well as certain internally defined special operators introduced to satisfy BPU constraints, are supported.