#Common Failure Resolutions

If obvious abnormalities have already appeared during quantized training, export, or deployment validation, or if you have already encountered direct environment, framework, or export-related errors, this page is the right place to start troubleshooting.

If you are still trying to understand what a symptom usually means and what to check first, it is better to start with PyTorch Model Quantization FAQ.

#First Identify Which Stage the Problem Belongs To

When the issue is about accuracy or abnormal results, it is usually best to narrow it down in the following order:

1. Whether accuracy has already dropped significantly after Calibration.

2. Whether abnormal loss, NAN, INF, or long-term non-convergence appears during QAT training.

3. Whether QAT accuracy is normal, but quantized.bc accuracy is abnormal.

4. Whether quantized.bc accuracy is normal, but hbm or on-board results still drop.

5. Whether int16 / fp16 settings do not take effect as expected.

If the issue is a direct error, such as an import error, a deepcopy error, or an export error, you can jump directly to the second half of this page under Direct Error Troubleshooting.

#Accuracy and Result Troubleshooting

#Accuracy Drops Significantly After Calibration

If the accuracy drop already appears at the Calibration model stage, the troubleshooting focus is usually not on export, convert, or compile, but on the calibration data, quantization configuration, and model structure itself.

Confirm first:

• First compare the evaluation results of the float model and the Calibration model to confirm whether the issue already appears at the Calibration stage.

• If fixed-point direction validation has already been done, then confirm whether export, convert, or compile further amplifies the error.

Check these artifacts first:

• Calibration data and preprocessing configuration, to confirm whether they are close enough to the real inference distribution.

• model_check_result.txt, especially shared modules, unfused patterns, per-layer qconfig, and high-precision output settings.

• If you still need to narrow it down further, continue with Calibration and Accuracy Tuning Guide.

What to do next:

• If the issue is clearly in the Calibration stage, first adjust the Calibration data, steps, batch size, and quantization configuration.

• If the issue appears only after export, convert, or compile, move to Deployment Consistency Analysis for further troubleshooting.

#Abnormal Loss During QAT Training

This type of issue usually appears as an excessively large initial QAT loss, NAN / INF during training, or loss that fails to converge for a long time. In this case, it is better to first confirm that the training pipeline and quantization states are correct before continuing to adjust hyperparameters.

Confirm first:

• First check whether the input data contains abnormal values.

• Then check whether the floating-point model has already converged.

• Then check whether Calibration was enabled correctly and whether the Calibration model parameters were loaded correctly.

Check these artifacts first:

• Training logs, to determine whether the anomaly appears at the beginning of training, after some number of steps, or after a state switch.

• model_check_result.txt and qconfig_dtypes.pt.py, to confirm that there are no obvious problems in graph structure, dtype, or quantization configuration.

• If you need to rule out pipeline issues, continue with the methods in Accuracy Tuning Guide, such as disabling fake quant or setting lr to 0.

What to do next:

• If the issue is mainly inside the training stage itself, first adjust the learning rate, gradient clipping, BN strategy, and fake quant states.

• If the abnormal loss is amplified only in the fixed-point direction, continue with Deployment Consistency Analysis to determine whether the issue occurs after export or convert.

#QAT Accuracy Is Normal, But quantized.bc Accuracy Is Abnormal

This usually means the training stage looks normal on the surface, but the abnormality has already appeared in the export or convert direction. The key is to identify the first stage where the deviation appears instead of continuing to look only at the QAT curve.

Confirm first:

• First confirm that QAT model accuracy is normal, and then validate the accuracies of qat.export.pt, qat.bc, and quantized.bc.

• Use pre_export to determine whether the issue already appears at the lookup-table fixed-point conversion stage.

Check these artifacts first:

• compare_per_layer_out.txt, for narrowing down the issue layer by layer.

• Bad cases, per-layer comparison results, and consistency sensitivity results produced by QuantAnalysis.

• Deployment Consistency Analysis, to distinguish whether the issue occurs in export or convert.

What to do next:

• If qat.pt and qat.export.pt are already inconsistent, handle it first as an export-stage issue.

• If qat.bc is normal but quantized.bc is abnormal, handle it first as a convert-stage issue.

• If the issue still cannot be located with regular methods, use bc_editor for segmented troubleshooting.

#quantized.bc Accuracy Is Normal, But hbm or On-board Results Drop

This kind of problem is usually more related to the compile or deployment path than to the training stage itself. It is recommended to verify it in the order quantized.bc -> hbm -> on-board, instead of treating it as a generic “QAT accuracy drop” from the beginning.

Confirm first:

• First confirm whether quantized.bc accuracy is normal.

• If quantized.bc is normal but hbm or on-board results still drop, then continue by determining whether the problem appears after compile or during board-side input preparation.

Check these artifacts first:

• Compile configuration, board-side input/output adaptation, and deployment logs.

• Preprocessing, postprocessing, and input data format configuration, to confirm whether input domains such as RGB, centered YUV444, or nv12 are consistent with the training stage.

• Deployment Consistency Analysis, to further determine where the problem lies in the compile or deployment path.

What to do next:

• It is recommended to validate artifacts and inputs/outputs stage by stage before deciding whether to return to the training side or continue investigating the deployment side.

• If the issue is already clearly in compile or board-side adaptation, do not keep retrying QAT parameter adjustments.

#int16 / fp16 Settings Do Not Take Effect as Expected

This type of problem is better checked through actual artifacts than through configuration code alone. Writing the configuration only proves that you tried to set it. Whether it actually takes effect should be judged from the final dtype results in the debug artifacts.

Confirm first:

• First determine whether the issue is that the configuration did not hit the target module at all, or that it did hit the target but the final dtype still does not match expectations.

• Then determine whether the issue is limited to a single layer or whether an entire class of template configurations is not taking effect.

Check these artifacts first:

• model_check_result.txt, qconfig_dtypes.pt.py, and qconfig_changelogs.txt, to confirm whether the final operator dtype matches expectations.

• The module_name in the configuration code, to confirm that the correct module name is used. This field supports only string values and does not support index-based configuration.

• If the platform strategy or template configuration itself is still unclear, continue with Build QConfig.

What to do next:

• First confirm that the configuration really hits the intended module, and then check the template override order and fallback logs.

• If the configuration code looks correct but the final dtype still does not match expectations, continue troubleshooting based on the final results in the debug artifacts rather than the configuration code alone.

#Direct Error Troubleshooting

#import Errors

#Cannot find the extension library(_C.so)

Recommendation:

• Make sure the horizon_plugin_pytorch version matches the CUDA version.

• In python3, locate the installation path of horizon_plugin_pytorch and check whether .so files exist under that directory. There may be multiple versions of horizon_plugin_pytorch installed at the same time, so it is recommended to uninstall them and keep only the required version.

#RuntimeError: Cannot load custom ops. Please rebuild the horizon_plugin_pytorch

Recommendation: Confirm that the local CUDA environment is correct, such as the path and version.

#Module Does Not Support deepcopy

Some frameworks, such as PyTorch Lightning, add wrappers around native torch modules but do not support deepcopy.

Recommendation: Implement the __deepcopy__ method for the model.

class Model(Module):
    ...
    def __deepcopy__(self, memo):
        new_model = Model()
        new_model.xxx = self.xxx
        return new_model

#RuntimeError: Only Tensors created explicitly by the user (graph leaves) support the deepcopy protocol at the moment

This error usually occurs when deepcopy torch.nn.Module is called. A common cause is that intermediate results are saved into the model during forward, and these intermediate results are non-leaf tensors, which then causes the deepcopy failure.

class Model(torch.nn.Module):
    ...
    def forward(self, x):
        # x is an input that requires gradient. x.requires_grad = True
        y = x + 1

        # y is a non-leaf tensor. Assigning it to a model attribute causes deepcopy model to fail.
        self.yy = y

Recommendation: Check whether any attributes in the model, including those left after forward, are non-leaf tensors. Before saving these tensors, make sure they are detached first.

class Model(torch.nn.Module):
    ...
    def forward(self, x):
        # x is an input that requires gradient. x.requires_grad = True
        y = x + 1

        # y is a non-leaf tensor. Detach it before saving it in the model.
        self.yy = y.detach()

#Error During Export: IndexError: list index out of range

This error is usually caused by QTensor appearing in the model outputs.

Recommendation: Check the model outputs and make sure all outputs have gone through dequant.