Starting from toolchain version OE3.5.0, we introduced a new qconfig template. The new qconfig unifies template configuration and fallback logic into a single workflow. Compared with the legacy approach, it is better suited for current use cases in terms of adapting to platform capabilities, improving quantization configuration efficiency, and supporting typical high-accuracy scenarios. We recommend migrating and upgrading as soon as possible.
In this section, we retain some introductory explanations of the legacy Qconfig usage principles to help you better understand it.
The qconfig refers to quantization configuration, which is a key set of parameters in the quantization process of deep learning models.
The quantization mode of the model is determined by qconfig, which needs to be set for the model before preparing the qat / calibration model.
Attention
Due to historical reasons, there are different definitions and usages of qconfig in the Plugin. Earlier versions of qconfig will be deprecated in the near future, and we only recommend that you use the qconfig usage described in this document.
A qconfig object can set three keywords: input, weight, and output, representing the quantization configuration of the operator's input, weight, and output respectively. When preparing model, these configurations determine whether to insert FakeQuantize or FakeCast nodes at the corresponding positions. None means no nodes will be inserted.
import torchfrom horizon_plugin_pytorch.quantization.qconfig import QConfigfrom horizon_plugin_pytorch.quantization.fake_quantize import FakeQuantizefrom horizon_plugin_pytorch.quantization.fake_cast import FakeCastfrom horizon_plugin_pytorch.quantization.observer_v2 import MinMaxObserverfrom horizon_plugin_pytorch.dtype import qint8qconfig = QConfig( input=None, weight=FakeQuantize.with_args( observer=MinMaxObserver, dtype=qint8, qscheme=torch.per_channel_symmetric, ch_axis=0, ), output=FakeCast.with_args(dtype=torch.float16), # activation=xxx Earlier usage, same as the output keyword. Still compatible, but it's recommended to use the output keyword.)
FakeQuantize is a fake quantization node that performs quantization and dequantization operations on the input.
Inserting fake quantization can simulate the errors caused by quantization in the forward pass of a floating-point model.
The horizon_plugin_pytorch supports three types of fake quantization: FakeQuantize, PACTFakeQuantize, and _LearnableFakeQuantize.
We recommend using the statistic-based FakeQuantize. The document won't introduce PACTFakeQuantize and _LearnableFakeQuantize.
If required, please read the papers before using them.
You can call the with_args method of FakeQuantize to get a constructor and use it to construct qconfig as shown in the previous section.
The parameters of with_args include parameters supported by FakeQuantize and observer, theoretically allowing configuration of all parameters declared in the __init__ method of the FakeQuantize and observer classes. However, to avoid unnecessary details, we recommend you to configure the observer-related parameters only.
Different observers have different parameters. Below are examples of constructing FakeQuantize with common used observers. For the specific usage of other observers, see the calibration section.
import torchfrom horizon_plugin_pytorch.quantization.qconfig import QConfigfrom horizon_plugin_pytorch.quantization.fake_quantize import FakeQuantizefrom horizon_plugin_pytorch.quantization.observer_v2 import MinMaxObserver, FixedScaleObserver, HistogramObserverfrom horizon_plugin_pytorch.dtype import qint8# The __init__ method of MinMaxObserver includes many parameters. The with_args method can control these parameters.# We only recommend you to set a few parameters as in the fq_constructor_1 example.# def __init__(# self,# averaging_constant: float = 0.01,# ch_axis: int = -1,# dtype: Union[torch.dtype, QuantDType] = qint8,# qscheme: torch.qscheme = torch.per_tensor_symmetric,# quant_min: int = None,# quant_max: int = None,# is_sync_quantize: bool = False,# factory_kwargs: Dict = None,# ) -> None:fq_constructor_1 = FakeQuantize.with_args( observer=MinMaxObserver, # Suitable for input/output/weight in qat and weight in calibration. averaging_constant=0.01, # When performing qat after calibration, the averaging_constant of input/output can be set to 0 to fix the scale. dtype=qint8, # Quantization type, set based on the support of the operator. qscheme=torch.per_channel_symmetric, # Only weight supports per-channel quantization. ch_axis=0, # Specify the channel for per-channel quantization.)# Similarly, you can check the __init__ method of FixedScaleObserver and HistogramObserver to learn the configurable parameters.fq_constructor_2 = FakeQuantize.with_args( observer=FixedScaleObserver, # Fixed scale, will not change in any conditions. dtype=qint8, # Quantization type, set based on the support of the operator. scale=INPUT_ABS_MAX / 128, # scale value, use maximum absolute value divided by the maximum quantization type value.)fq_constructor_3 = FakeQuantize.with_args( observer=HistogramObserver, # Suitable for input/output in calibration. dtype=qint8, # Quantization type, set based on the support of the operator.)qconfig = QConfig( weight=fq_constructor_x, ...)
FakeCast is a fake conversion node that converts the input to float32 data type. If the data type is float16, it also simulates the truncation error caused by converting value to float16. This node is mainly used to mark operators that require floating-point computation.
The method of using FakeCast to construct qconfig is similar to FakeQuantize, but it only has one parameter.
import torchfrom horizon_plugin_pytorch.quantization.qconfig import QConfigfrom horizon_plugin_pytorch.quantization.fake_cast import FakeCastqconfig = QConfig( input=FakeCast.with_args(dtype=torch.float16), # set based on the support of the operator. ...)
