In previous chapters, we have repeatedly mentioned some debug artifacts, such as model_check_result.txt, fx_graph.txt, qconfig_dtypes.pt, and compare_per_layer_out.txt.

These files themselves are not complicated. What usually blocks users is not the files, but questions like the following:

• Which file should be checked first at the current stage?

• What is this file mainly used to determine?

• After an abnormality is found, what should be checked next?

Therefore, this chapter organizes the most commonly used debug artifacts across the entire workflow in one place, so that you can more quickly answer the following questions:

• Whether operators were fused successfully.

• Whether there are shared modules.

• Whether dtype settings took effect as expected.

• Whether scale is abnormal.

• How to determine which layer is sensitive.

• Whether the error looks more like truncation error or rounding error.

#Debug Artifacts Overview

From the perspective of when they are used, these artifacts can generally be divided into two categories:

• Prepare-stage inspection artifacts: used to confirm whether model structure, graph capture results, and qconfig are basically correct.

• Error-analysis and consistency-analysis artifacts: used to further locate where the error is concentrated and which operators are most sensitive.

Their correspondence is as follows:

If you only want to understand the troubleshooting order, it can usually be simplified into two steps:

1. First check Prepare-stage artifacts to confirm whether the workflow has been set up correctly.

2. Then check analysis artifacts to confirm where the error mainly occurs and what should be adjusted next.

#Common Questions and the Corresponding Artifacts

If you currently just want to quickly decide which artifact should be checked first, you can refer to the following table:

#Recommended Reading Order

If you are still not sure how to use these artifacts, they can usually be understood in the following order:

1. Right after Prepare, first check model_check_result.txt to confirm whether model structure, fusion, and qconfig inspection results are generally normal.

2. If you suspect that the graph structure itself is not as expected, then check fx_graph.txt to confirm whether the actual captured execution path, module replacement, and quantization-node insertion are correct.

3. If the issue mainly lies in whether dtype settings took effect, then continue with qconfig_dtypes.pt.py and qconfig_changelogs.txt to confirm both the final result and the change process.

4. If Calibration, QAT, or deployment-side results are unsatisfactory, then start checking output_xxx_sensitive_ops.txt and compare_per_layer_out.txt to determine which outputs are most sensitive and from which layer the error starts to grow.

5. Finally, use abnormal_layer_advisor.txt to help confirm whether there are obviously abnormal layers, and determine which local issue should be inspected first next.

If these artifacts already suggest that the issue is more likely to occur in the Export, Convert, or Compile stage, it is recommended that you continue reading Deployment Consistency Analysis and proceed with stage-by-stage troubleshooting.

#Prepare-Stage Artifacts

Prepare-stage artifacts are most suitable when the workflow has just been set up. If there are already obvious issues in structure, graph, or qconfig at this stage, later Calibration, QAT, Export, Convert, and consistency-analysis results will usually all be affected.

#model_check_result.txt

model_check_result.txt is automatically generated after Prepare is completed, and can also be generated separately through check_qat_model. It puts model structure inspection, operator fusion inspection, call-count inspection, and qconfig inspection results together, and is the most important overview artifact in the Prepare stage.

If you only want to do one overall check right after Prepare, this file is usually the first one to inspect.

In actual use, it is recommended that you focus on the following kinds of information:

• Whether there are patterns that could be fused but were not fused.

• Whether the call count of each module is normal, and whether there are shared modules.

• Whether the input / weight / output dtype of each layer is generally consistent with the target configuration.

• Whether there are abnormal qconfig hints that require extra attention, such as fixed scale, configuration conflicts, or suspicious settings.

This file is most suitable for answering the following questions:

• Whether operators were fused successfully.

• Whether there are shared modules.

• Whether qconfig obviously deviates from expectations.

If you have already found obvious issues here, it is usually recommended that you first go back to Prepare Description and Build QConfig, instead of continuing directly to Calibration or consistency analysis.

#fx_graph.txt

fx_graph.txt is generated after the graph-mode execution of Prepare, and is used to save the actual captured FX graph. Compared with model_check_result.txt, it does not directly tell you what is wrong, but it is more suitable for confirming what graph the tool is ultimately using to process the model.

If you suspect that the training path and deployment path are not completely consistent, or suspect that certain modules or branches did not really enter the graph, then this file should usually be checked first.

In actual use, key points to focus on include:

• Whether the final captured execution path is consistent with expectations.

• Whether certain modules, branches, or loops really entered the graph.

• Whether module replacement, operator fusion, and quantization-node insertion positions are generally reasonable.

In the following scenarios, fx_graph.txt is often especially worth checking first:

• There are dynamic branches or a deploy switch in forward.

• Some modules are defined in the code, but later results suggest that they did not take effect.

• You suspect that the training path and deployment path are not completely consistent.

If fx_graph.txt itself is already obviously inconsistent with the expected structure, it is usually not recommended to continue troubleshooting accuracy issues. Instead, you should first go back to the Prepare stage and adjust the model implementation, example_inputs, or related configuration.

#qconfig_dtypes.pt

qconfig_dtypes.pt is automatically saved after Prepare is completed. It is the saved file of operator-level quantization configuration, and is mainly used for reusing existing configuration.

If you later want to continue using the current configuration through LoadFromFileTemplate, or want to preserve the current quantization result as the starting point for subsequent tuning, this file becomes important.

However, if your current goal is only to quickly inspect whether the configuration is as expected, this file is usually not the first one to check. qconfig_dtypes.pt.py is more intuitive for direct inspection.

#qconfig_dtypes.pt.py

qconfig_dtypes.pt.py is also automatically saved after Prepare is completed. It is the readable version of qconfig_dtypes.pt, and is usually the most suitable one for direct manual inspection.

If what you care about most right now is “what dtype each key module was actually configured to use,” this file is usually more useful than the .pt file.

In actual use, key points to focus on include:

• The input / weight / output dtype of key modules.

• Whether key operators such as Conv / Linear / Matmul were really configured as qint8, qint16, or torch.float16.

• Whether the current mixed-precision configuration generally falls into the region you expect.

This file is most suitable for the following scenarios:

• Manually checking whether key modules such as backbone, neck, and head use the expected dtype.

• Judging whether the basic mixed-precision configuration has generally taken effect.

If you find that the result does not completely match your intuition from the template, it is not recommended to directly infer that “I only need to manually add a few dtype lines.” Instead, continue combining it with qconfig_changelogs.txt to determine which template, override relationship, or default optimization changed the result.

#qconfig_changelogs.txt

qconfig_changelogs.txt is automatically saved after Prepare is completed, and is used to record the qconfig change process. If qconfig_dtypes.pt.py is more suitable for answering “what the final result is,” then qconfig_changelogs.txt is more suitable for answering “why it finally became this result.”

When you suspect that the final dtype of an operator is inconsistent with your intuition from the template, or want to confirm whether the template override order is as expected, this file is usually the most worth checking.

It is especially suitable for helping you determine the following kinds of problems:

• Which template changed the qconfig of a specific module.

• Whether the template override order is as expected.

• Whether there are unexpected overrides, rollbacks, or simplifications.

It is especially useful in the following scenarios:

• You clearly configured higher precision, but it did not finally take effect.

• After combining global and local templates, the result is inconsistent with your intuition.

• After loading qconfig_dtypes.pt, you continue modifying dtype, and the result becomes unstable.

If you eventually confirm that the problem mainly comes from the template configuration relationship itself, it is recommended that you go back to Build QConfig and adjust the template, rather than staring only at the final artifact and patching the result temporarily.

#Error-Analysis and Consistency-Analysis Artifacts

When the Prepare stage is already basically normal, but Calibration, QAT, or deployment-side results are still unsatisfactory, what you usually need to focus on next is no longer “whether the workflow was built correctly,” but rather “where the error is concentrated, which operators are most sensitive, and what should be adjusted first next.”

At this point, artifacts related to QuantAnalysis are usually more helpful.

#compare_per_layer_out.txt

compare_per_layer_out.txt is generated after running QuantAnalysis.compare_per_layer(), and is the core artifact for per-layer comparison. It provides information in tabular form for each layer, including shape, quantized dtype, qscale, Cosine, L1, Atol, max_qscale_diff, and the statistics range before and after comparison. It is usually one of the most important files for locating issues layer by layer.

If Calibration accuracy is unsatisfactory, QAT accuracy is abnormal, or consistency issues occur in the Export / Convert stage, you should usually go back to this file to determine from which layer the error starts to amplify.

In actual troubleshooting, the following kinds of information deserve particular attention:

• From which layer onward the output metrics of the two models begin to deteriorate obviously.

• Whether the quant_dtype and qscale of the abnormal layer are reasonable.

• Whether the minimum, maximum, and mean values of base_model_* and analy_model_* deviate obviously.

• Whether max_qscale_diff is abnormally large.

This file is especially suitable for helping you answer the following questions:

• From which layer the issue starts to amplify clearly.

• Whether the scale of a certain layer may be unreasonable.

• Whether the current error looks more like truncation error or rounding error.

When judging whether the issue looks more like truncation or rounding, you can first use the following rules of thumb:

• If the numeric range of the model under analysis is obviously compressed, or compared with the baseline model you observe boundary shrinkage, saturation, clipping, and similar effects, then the issue usually looks more like truncation error or unreasonable scale setting.

• If the overall numeric ranges of the two models are close, but Atol / L1 persist while Cosine does not collapse significantly, then the issue usually looks more like rounding error, lookup-table approximation, or implementation-detail differences.

#output_xxx_sensitive_ops.pt/.txt

These files are generated after running QuantAnalysis.sensitivity(), and are among the most commonly used artifacts for mixed-precision tuning and key-layer localization.

If the model has multiple outputs, files are usually generated separately for each output, such as output_0_ATOL_sensitive_ops.txt. In actual use, it is recommended that you first choose the output file most relevant to your current business metric, and then start the analysis.

Among them:

• The txt file is more suitable for manual reading.

• The pt file is more suitable for script loading, or for subsequent automated configuration together with templates.

These files are usually most suitable for the following scenarios:

• During mixed-precision tuning, judging which operators are most worth upgrading first to qint16 or torch.float16.

• When the model has multiple outputs, determining which output branch is more strongly affected by the current accuracy drop.

In actual inspection, key points to focus on include:

• The current quantized dtype used by the highly ranked sensitive operators.

• Whether these operators are still qint8, and whether it is worth trying to promote them to qint16 or torch.float16.

• Whether the proportion of computation in these operators allows you to pay the cost of higher precision for them.

• Whether sensitivity mainly comes from activation, weights, or a specific input.

If an operator is already high precision but still ranks near the top, this often means that the issue is not simply “insufficient precision type,” and you still need to continue combining it with compare_per_layer_out.txt to see how its numeric distribution and error are produced.

#abnormal_layer_advisor.txt

abnormal_layer_advisor.txt is generated after running QuantAnalysis.compare_per_layer(), and is more like a list of abnormal-layer hints. It helps you quickly scan out layers that deserve attention first, such as layers with excessively large data range, inf or NaN, or outputs without high-precision protection.

Therefore, it is more suitable as an entry point for “quickly narrowing the scope,” rather than as the final conclusion.

This file is most suitable for the following scenarios:

• There are too many per-layer results, and you do not yet know which layers should be checked first.

• You want to quickly filter out obviously abnormal layers first, and then go back to the per-layer comparison results for further checking.

In actual use, it can usually be understood as follows:

• If it already clearly indicates that certain layers are obviously abnormal, it means that these layers are worth going back to compare_per_layer_out.txt to inspect first.

• If it does not indicate any obvious abnormality, that still does not mean that the model definitely has no problem. You still need to continue judging together with per-layer comparison and sensitivity results.