#Build QConfig

Qconfig (Quantization Configuration) refers to the quantization configuration. It is a key set of parameters in the deep learning model quantization process. The model's quantization mode is determined by qconfig. Before preparing a calibration / QAT model, you need to set qconfig for the model first. In the previous section PyTorch Model Quantization Basic Process, we already introduced the position of QConfig in the overall pipeline. Next, this chapter will further walk you through how to get started with configuration, how to choose templates, and how to use them in typical scenarios based on the practical usage of the new qconfig templates.

#New qconfig

As the toolchain continues to improve in quantization accuracy, starting from toolchain version OE3.5.0, we support the new qconfig templates. The new qconfig unifies template configuration and fallback logic into one workflow. Compared with the legacy approach, it is better suited to current usage in terms of matching platform capabilities, improving quantization configuration efficiency, and supporting typical high-accuracy scenarios.

Compared with the legacy approach, the main changes in the new qconfig templates are as follows:

• Templates and fallback mechanisms are managed in a unified way, making the overall workflow clearer.

• Single / dual int16 input configuration is better supported for GEMM-type operators such as Conv / Matmul / Linear.

• Dtype is configured first, and then fuse decisions are made based on requirements, so the final result is closer to expectations.

• After prepare, qconfig_dtypes.pt, qconfig_dtypes.pt.py, and qconfig_changelogs.txt are saved automatically for reuse, inspection, and traceability.

With the new template-based workflow, you usually no longer need to focus on low-level definitions such as handwritten QConfig(...), nor do you need to manually write complex configurations one by one. Instead, you can focus on the main path of QconfigSetter + templates.

#QconfigSetter Description

QconfigSetter is currently the most recommended way to configure qconfig. It works with the model computation graph, automatically sets qconfig for the model in template order, and outputs the corresponding configuration result after prepare.

Using QconfigSetter depends on the graph mode in the prepare process. A typical usage is as follows:

from horizon_plugin_pytorch.quantization import prepare, PrepareMethod, get_qconfig
from horizon_plugin_pytorch.quantization.qconfig_setter import *
from horizon_plugin_pytorch.quantization.observer_v2 import HistogramObserver

my_qconfig_setter = QconfigSetter(
    reference_qconfig=get_qconfig(observer=HistogramObserver),   # qconfig used to provide the observer
    templates=[                   # configured templates
        ...
    ],
    enable_optimize=True,         # enable default optimizations
    save_dir=args.save_path,      # save path for qconfig-related configuration results and changelog
)

float_model.eval()
qat_model = prepare(
    float_model,
    example_inputs=example_input,     # graph mode requires model inputs to obtain the computation graph
    qconfig_setter=my_qconfig_setter,
    check_result_dir=args.save_path,
    method=PrepareMethod.JIT_STRIP,   # QconfigSetter depends on the computation graph
)

When using it, we recommend that you focus on the following parameters:

• reference_qconfig, a required configuration used to specify the observer in a unified way. The observer is used to observe tensor distributions during subsequent Calibration or QAT, and provides the basic information for later quantization parameter calculation.

• templates, a required configuration used to define dtype, threshold, and their coverage at different locations. The configurable templates currently include ModuleNameTemplate, ConvDtypeTemplate, MatmulDtypeTemplate, SensitivityTemplate, and LoadFromFileTemplate. Detailed descriptions are provided in the template sections below.

• enable_optimize, used to determine whether the default optimization template passes are enabled. In most cases, we recommend keeping it True.

• save_dir, which we recommend always enabling. After prepare is executed, qconfig-related configuration results and change records are saved here, which makes it easier for you to check whether the configuration is as expected and to reuse existing configurations later. For details on these artifacts, please refer to Prepare Description.

For the differences between observers, scale search strategies, and the usage of reset_scale, please refer to the relevant description in Model Calibration.

#Core Template Description

The templates you can currently configure include the following:

#ModuleNameTemplate

ModuleNameTemplate is used to specify dtype configuration or quantization threshold configuration by module name. It supports global, model-part, and operator-level configuration, and it is also the starting point of many qconfig configurations.

Its most common usage includes the following:

• Configure the global output dtype.

• Configure a group of modules under a certain prefix to use a different precision.

• Add threshold for a specific operator, so as to implement fix scale.

The detailed usage of ModuleNameTemplate mainly includes the following points:

1. The module name can be an operator name or a prefix. When different module names in one ModuleNameTemplate overlap, the longer name has higher priority. For example:

ModuleNameTemplate(
    {
        "": qint8,  # global qint8
        "head": qint16,  # higher priority than the global configuration, head is finally configured as int16
        "head.conv0": torch.float16,  # higher priority than the configuration on head, head.conv0 is finally configured to output float16
    }
)

2. You can specify threshold for an operator, provided that the operator contains the corresponding fake quantization node. In this case, the quantization scale is calculated as $scale = \frac{threshold}{-qdtype.min}$. For example:

ModuleNameTemplate(
    {
        "quant": {"dtype": qint8, "threshold": 1.0},  # the quantization scale of quant is 1.0/128
    }
)

3. dtype and threshold are configured on the operator output by default. You can configure input or weight by specifying the key. When an operator has multiple inputs, None can be used as a placeholder. For example:

ModuleNameTemplate(
    {
        "conv0": {"input": qint8, "weight": qint16},
        "conv1": {"dtype": {"input": qint16}, "threshold": {"weight": 1.0}},
        "matmul0": {
            "dtype": {"input": [qint16, None]},
            "threshold": {"input": [1.0, None]},  # configure the first input of matmul0 as int16 with a fixed scale of 1.0/32768, and leave the second input unchanged
        },
    }
)

4. In some locations of the model, it is difficult to obtain the optimal quantization scale through statistical methods alone, such as inputs or outputs related to physical quantities. When the value range of an operator is relatively clear, in the new qconfig templates, you can implement fix scale by configuring dtype + threshold, which helps avoid cases where the scale obtained later from statistics or moving average during Calibration or QAT does not fully cover the valid value range.

In the new qconfig templates, the scale is calculated as $scale = \frac{threshold}{-qdtype.min}$, where threshold is generally the maximum absolute value of the corresponding input or output, and qdtype is the corresponding quantized type. The configurations with threshold in points 2 and 3 above are common fix scale patterns in the new templates. For example:

my_qconfig_setter = QconfigSetter(
    reference_qconfig=get_qconfig(observer=HistogramObserver),
    templates=[
        ModuleNameTemplate({"": qint8}),
        ConvDtypeTemplate(input_dtype=qint8, weight_dtype=qint8),
        MatmulDtypeTemplate(input_dtypes=qint8),
        ModuleNameTemplate(
            {
                "backbone.quant": {"dtype": qint8, "threshold": 1.0},
                "backbone.conv1.conv1_2.conv": {
                    "dtype": {"weight": qint16},
                    "threshold": {"weight": 1.0},
                },
            }
        ),
    ],
    save_dir=args.save_path,
)

You can check whether fix scale takes effect after Prepare Description by examining the generated model_check_result.txt.

#ConvDtypeTemplate

ConvDtypeTemplate is used to batch configure the input and weight dtypes of Conv / Linear operators.

Its most common use cases include the following:

• Configure global Conv / Linear as int8.

• Configure single-int16 or dual-int16 input for convolution operators under a specific prefix.

• Improve precision for some sensitive operators in mixed-precision scenarios.

In actual use, ConvDtypeTemplate usually needs to be configured together with ModuleNameTemplate and MatmulDtypeTemplate. A common order is to first use ModuleNameTemplate to configure the global output dtype, then use ConvDtypeTemplate and MatmulDtypeTemplate to complete the basic GEMM configuration, and finally add a local ConvDtypeTemplate to override the earlier settings.

An example is shown below:

my_qconfig_setter = QconfigSetter(
    reference_qconfig=get_qconfig(observer=HistogramObserver),
    templates=[
        ModuleNameTemplate({"": qint8}),
        ConvDtypeTemplate(
            input_dtype=qint8,
            weight_dtype=qint8,
        ),
        MatmulDtypeTemplate(
            input_dtypes=qint8,
        ),
        ConvDtypeTemplate(
            input_dtype=qint8,
            weight_dtype=qint16,
            prefix=["backbone.conv1.conv1_3.conv"],
        ),
    ],
)

#MatmulDtypeTemplate

MatmulDtypeTemplate is used to configure the input dtype of Matmul operators.

It supports:

• dual int8 input.

• single int16 input.

• dual int16 input.

• batch configuration by prefix.

In Transformer or attention-related structures, this template is usually used quite often.

In actual use, MatmulDtypeTemplate also usually needs to be configured together with ModuleNameTemplate and ConvDtypeTemplate. The common order is also to complete the basic template configuration first, and then append a local MatmulDtypeTemplate later for overriding.

An example is shown below:

my_qconfig_setter = QconfigSetter(
    reference_qconfig=get_qconfig(observer=HistogramObserver),
    templates=[
        ModuleNameTemplate({"": qint8}),
        ConvDtypeTemplate(
            input_dtype=qint8,
            weight_dtype=qint8,
        ),
        MatmulDtypeTemplate(
            input_dtypes=[qint8, qint8],
        ),
        MatmulDtypeTemplate(
            input_dtypes=[qint16, qint8],
            prefix=["encoder.encoder.0.layers.0.self_attn.matmul"],
        ),
    ],
)

#SensitivityTemplate

SensitivityTemplate is used to raise the precision of highly ranked sensitive operators based on sensitivity analysis results.

It supports:

• raising activation-sensitive operators.
• raising weight-sensitive operators.
• handling both activation-sensitive and weight-sensitive operators at the same time.

More commonly, it is used after a basic qconfig configuration has been completed, and then the sensitivity template is used to raise key operators to int16.

SensitivityTemplate is usually placed after the basic templates. In other words, you first complete the basic configuration of ModuleNameTemplate, ConvDtypeTemplate, and MatmulDtypeTemplate, and then append sensitivity templates later to raise the precision.

An example is shown below:

table1 = torch.load("output1_ATOL_sensitive_ops.pt")
table2 = torch.load("output2_ATOL_sensitive_ops.pt")

my_qconfig_setter = QconfigSetter(
    reference_qconfig=get_qconfig(observer=HistogramObserver),
    templates=[
        ModuleNameTemplate({"": qint8}),
        ConvDtypeTemplate(input_dtype=qint8, weight_dtype=qint8),
        MatmulDtypeTemplate(input_dtypes=qint8),
        SensitivityTemplate(
            sensitive_table=table1,
            topk_or_ratio=10,
            sensitive_type="weight",
        ),
        SensitivityTemplate(
            sensitive_table=table2,
            topk_or_ratio=0.5,
            sensitive_type="activation",
        ),
    ],
)

#LoadFromFileTemplate

LoadFromFileTemplate is used to load previously saved quantization configuration from qconfig_dtypes.pt, which is suitable for reusing existing configuration results.

An example is shown below:

my_qconfig_setter = QconfigSetter(
    reference_qconfig=get_qconfig(observer=HistogramObserver),
    templates=[
        LoadFromFileTemplate("qconfig_dtypes.pt"),
    ],
    enable_optimize=False,
    save_dir=args.save_path,
)

#Typical Configuration Examples

#Global int8/int16

If you would like to quickly run through a basic configuration first, you can start with global int8:

QconfigSetter(
    reference_qconfig=get_qconfig(),
    templates=[
        ModuleNameTemplate({"": qint8}),
        ConvDtypeTemplate(input_dtype=qint8, weight_dtype=qint8),
        MatmulDtypeTemplate(input_dtypes=qint8),
    ],
)

This is a common starting point for many models, especially when you want to complete one round of Calibration verification first.

The configuration method for global int16 is similar to that for global int8. You only need to replace qint8 with qint16 in the example above.

#feature int16, weight int8

If you would like to start from a higher feature precision, you can also configure it as follows:

QconfigSetter(
    reference_qconfig=get_qconfig(),
    templates=[
        ModuleNameTemplate({"": qint16}),
        ConvDtypeTemplate(input_dtype=qint16, weight_dtype=qint8),
        MatmulDtypeTemplate(input_dtypes=qint16),
    ],
)

There are two practical points here that deserve special attention:

• When you configure global feat int8 / int16 / fp16, you must also configure the weight of Conv operators accordingly. Otherwise, weight may automatically use int16 computation and further introduce unexpected CPU operators.

• After you configure global int8, you may still see some operators using int16 in model_check_result.txt. This is usually an automatic handling strategy used by the tool to improve quantization accuracy. For example, some operators implemented through decomposition may use higher precision internally and still output int8.

If you would also like to batch switch modules under a certain prefix back to int8, you can continue stacking ModuleNameTemplate, ConvDtypeTemplate, and MatmulDtypeTemplate, for example:

my_qconfig_setter = QconfigSetter(
    reference_qconfig=get_qconfig(observer=HistogramObserver),
    templates=[
        ModuleNameTemplate({"": qint16}),
        ConvDtypeTemplate(input_dtype=qint16, weight_dtype=qint8),
        MatmulDtypeTemplate(input_dtypes=qint16),
        ModuleNameTemplate({"backbone": qint8}),
        MatmulDtypeTemplate(
            input_dtypes=[qint8, qint8],
            prefix=["backbone"],
        ),
        ConvDtypeTemplate(
            input_dtype=qint8,
            weight_dtype=qint8,
            prefix=["backbone"],
        ),
    ],
)

#Dual-int8 GEMM operators, fp16 for other operators

This type of configuration is one of the common starting points on J6P/H. Since the platform provides floating-point capability, keeping vector-type computation in fp16 is often a practical way to get the pipeline running first, while still constraining Conv / Matmul GEMM operators to int8 computation. This makes it a common starting configuration for balancing accuracy and performance.

QconfigSetter(
    reference_qconfig=get_qconfig(),
    templates=[
        ModuleNameTemplate({"": torch.float16}),  # configure all operators to output fp16
        ConvDtypeTemplate(input_dtype=qint8, weight_dtype=qint8),  # configure the input and weight of conv as int8
        MatmulDtypeTemplate(input_dtypes=qint8),  # configure both inputs of matmul as int8
    ],
)

#Dual-int8 GEMM operators, fp16 for other operators, and configure high-sensitivity GEMM operators as int16

If you already have sensitivity analysis results, you can continue to use SensitivityTemplate for batch promotion and raise high-sensitivity GEMM operators to int16, for example:

table1 = torch.load("output1_ATOL_sensitive_ops.pt")
table2 = torch.load("output2_ATOL_sensitive_ops.pt")

my_qconfig_setter = QconfigSetter(
    reference_qconfig=get_qconfig(observer=HistogramObserver),
    templates=[
        ModuleNameTemplate({"": torch.float16}),
        ConvDtypeTemplate(input_dtype=qint8, weight_dtype=qint8),
        MatmulDtypeTemplate(input_dtypes=[qint8, qint8]),
        SensitivityTemplate(
            sensitive_table=table1,
            topk_or_ratio=10,
        ),
        SensitivityTemplate(
            sensitive_table=table2,
            topk_or_ratio=0.1,
        ),
    ],
)

#Dual-int8 GEMM operators, int16 for other operators, and configure high-sensitivity GEMM operators as int16

If you would like to further validate a higher-precision configuration, you can also start from "int16 for other operators and dual int8 for GEMM operators", and then batch-raise high-sensitivity GEMM operators to int16. This configuration is more suitable for first validating what can be achieved with a higher-precision setup, and then gradually rolling back some high-precision GEMM operators according to deployment performance requirements. An example is shown below:

table1 = torch.load("output1_ATOL_sensitive_ops.pt")
table2 = torch.load("output2_ATOL_sensitive_ops.pt")

my_qconfig_setter = QconfigSetter(
    reference_qconfig=get_qconfig(observer=HistogramObserver),
    templates=[
        ModuleNameTemplate({"": qint16}),
        ConvDtypeTemplate(input_dtype=qint8, weight_dtype=qint8),
        MatmulDtypeTemplate(input_dtypes=[qint8, qint8]),
        SensitivityTemplate(
            sensitive_table=table1,
            topk_or_ratio=10,
        ),
        SensitivityTemplate(
            sensitive_table=table2,
            topk_or_ratio=0.1,
        ),
    ],
)

#Combined Example Using Four Common Templates

In general, ModuleNameTemplate, ConvDtypeTemplate, MatmulDtypeTemplate, and SensitivityTemplate are the four templates most commonly used together. The following integrated example helps you understand how these templates work together as a whole:

from horizon_plugin_pytorch.quantization.qconfig_setter import *
import torch

table1 = torch.load("output1_ATOL_sensitive_ops.pt")
table2 = torch.load("output2_ATOL_sensitive_ops.pt")

my_qconfig_setter = QconfigSetter(
    reference_qconfig=get_qconfig(observer=HistogramObserver),
    templates=[
        ModuleNameTemplate({"": qint8}),
        ConvDtypeTemplate(input_dtype=qint8, weight_dtype=qint8),
        MatmulDtypeTemplate(input_dtypes=qint8),
        ModuleNameTemplate(
            {"quant": {"dtype": qint8, "threshold": 1.0}}
        ),
        MatmulDtypeTemplate(
            input_dtypes=[qint16, qint8],
            prefix=["head"],
        ),
        ConvDtypeTemplate(
            input_dtype=qint8,
            weight_dtype=qint16,
            prefix=["backbone"],
        ),
        SensitivityTemplate(
            sensitive_table=table1,
            topk_or_ratio=10,
            sensitive_type="weight",
        ),
        SensitivityTemplate(
            sensitive_table=table2,
            topk_or_ratio=0.5,
            sensitive_type="activation",
        ),
    ],
)

In this example, the first three templates are used to complete the basic configuration, and then fix scale, local high precision, and sensitivity-based precision raising are stacked later. In actual use, you can remove, add, or replace templates according to the model structure and tuning goal.

#Default Optimization Template Pass Description

In addition to the templates you configure explicitly, QconfigSetter also integrates a series of default optimization templates that are used to legalize and optimize qconfig results. When enable_optimize is set to True, the default optimization templates take effect. You do not necessarily need to configure these templates one by one manually, but it is important to understand what they do, because this helps you understand what kinds of optimizations the default optimization templates will perform.

They mainly include the following:

• CanonicalizeTemplate: legalizes dtype configuration by operator type. The current default rules include:

  • GEMM-type operators do not support float input, including weight.

  • Interpolation operators still prefer int8 by default. Current interpolation operators on nash-e/m/p/h platforms support fp16 and int16, but their performance is relatively poor, so the templates still default to int8. If an interpolation operator needs to use another dtype, please use ModuleNameTemplate(..., freeze=True) to configure the output of its previous operator to the required dtype.

  • Special operators such as DPP and RPP only support int8.

  • The general rule for other operators is that input dtype and output dtype cannot contain qint and float at the same time.

• EqualizeInOutScaleTemplate: for operators such as relu, relu6, concat, and stack, scale should be collected after the operator. Otherwise, accuracy or performance may be affected. To do this:

  • The output dtype of the previous operator is set to float32.

  • During export to HBIR, pseudo quantization is inserted at the input of relu, concat, and stack, and the scale reuses the output scale.

• FuseConvAddTemplate: handles hardware-supported conv + add fuse scenarios. To do this:

  • The output dtype of conv is set to float32.

  • The corresponding input dtype of add is set to float32.

• GridHighPrecisionTemplate: based on experience, the grid computation process of grid sample does not have sufficient precision with qint8, so related operators are automatically configured to higher precision.

• InternalQuantsTemplate: in model partition deployment scenarios, QuantStub is inserted at partition points to record dtype and scale at that location. The dtype configuration of this type of QuantStub must remain consistent with the input.

• OutputHighPrecisionTemplate: when a GEMM-type operator is used as the model output, it is configured as high-precision output.

• PropagateTemplate: for operators implemented as decomposed subgraphs, there are empirical configurations. For example, internal small operators in LayerNorm and Softmax should use higher precision.

• SimpleIntPassTemplate: for performance optimization, in computation graphs such as op0 -> op1 -> op2, the output type of op1 is changed to int when all the following conditions are met:

  • op2 requires int input configuration.

  • op0 can output int.

  • op1 currently outputs float16, and belongs to one of the following types:

    • cat and stack.

    • mul_scalar.

    • lookup-table operators without accuracy risk, that is, operators that are implemented with lookup tables by default under fp16.

• SimplifyTemplate: removes redundant quantization node configuration by changing the corresponding dtype to None.

If you find that the final qconfig result is not fully aligned with your intuitive understanding of the templates, in many cases these default optimization templates are taking effect.

#Common Usage Pitfalls

#Continue modifying dtype after using LoadFromFileTemplate

This is not a recommended usage pattern.

After loading qconfig_dtypes.pt, you should generally not continue modifying dtype. If you keep modifying it, the result may not only become unstable, but may also break the original optimization result.

#Only configure the global dtype without explicitly configuring Conv / Matmul

This is another common pitfall.

For example, if you only use ModuleNameTemplate({"": qint8}) for global configuration, but do not also configure ConvDtypeTemplate and MatmulDtypeTemplate, the final result may differ from your expectations.

Therefore, in most scenarios, ModuleNameTemplate, ConvDtypeTemplate, and MatmulDtypeTemplate should be considered together, rather than configuring only one global template.

If you encounter dtype fallback, weight dtype that differs from your expectations, or a final result that does not match your intuitive understanding of the templates during actual usage, we recommend that you refer to Common Failures and continue troubleshooting together with model_check_result.txt and qconfig_changelogs.txt.