In previous chapters, we have repeatedly mentioned some debug artifacts, such as model_check_result.txt, fx_graph.txt, qconfig_dtypes.pt, and compare_per_layer_out_0.txt.

These files themselves are not complicated. What usually blocks users is not the files, but questions like the following:

• Which file should be checked first at the current stage?

• What is this file mainly used to determine?

• After an abnormality is found, what should be checked next?

Therefore, this chapter organizes the most commonly used debug artifacts across the entire workflow in one place, so that you can more quickly answer the following questions:

• Whether operators were fused successfully.

• Whether there are shared modules.

• Whether dtype settings took effect as expected.

• Whether scale is abnormal.

• How to determine which layer is sensitive.

• Whether the error looks more like truncation error or rounding error.

#Debug Artifacts Overview

From the perspective of when they are used, these artifacts can generally be divided into two categories:

• Prepare-stage inspection artifacts: used to confirm whether model structure, graph capture results, and qconfig are basically correct.

• Error-analysis and consistency-analysis artifacts: used to further locate where the error is concentrated and which operators are most sensitive.

Their correspondence is as follows:

If you only want to understand the troubleshooting order, it can usually be simplified into two steps:

1. First check Prepare-stage artifacts to confirm whether the workflow has been set up correctly.

2. Then check analysis artifacts to confirm where the error mainly occurs and what should be adjusted next.

#Common Questions and the Corresponding Artifacts

If you currently just want to quickly decide which artifact should be checked first, you can refer to the following table:

#Recommended Reading Order

If you are still not sure how to use these artifacts, they can usually be understood in the following order:

1. Right after Prepare, first check model_check_result.txt to confirm whether model structure, fusion, and qconfig inspection results are generally normal.

2. If you suspect that the graph structure itself is not as expected, then check fx_graph.txt to confirm whether the actual captured execution path, module replacement, and quantization-node insertion are correct.

3. If the issue mainly lies in whether dtype settings took effect, then continue with qconfig_dtypes.pt.py and qconfig_changelogs.txt to confirm both the final result and the change process.

4. If Calibration, QAT, or deployment-side results are unsatisfactory, then start checking output_xxx_case_xxx_sensitive_ops.csv and compare_per_layer_out_0.txt to determine which outputs are most sensitive and from which layer the error starts to grow.

5. Then use the advice column in the per-layer comparison results to help confirm whether there are obviously abnormal layers, and determine which local issue should be inspected first next.

6. If you already have the sensitivity ranking but are still unsure how many operators should actually be promoted to int16, check output_xxx_tuned_case_xxx_sensitive_ops.csv to confirm the cumulative gain.

If these artifacts already suggest that the issue is more likely to occur in the Export, Convert, or Compile stage, it is recommended that you continue reading Deployment Consistency Analysis and proceed with stage-by-stage troubleshooting.

#Prepare-Stage Artifacts

Prepare-stage artifacts are most suitable when the workflow has just been set up. If there are already obvious issues in structure, graph, or qconfig at this stage, later Calibration, QAT, Export, Convert, and consistency-analysis results will usually all be affected.

#model_check_result.txt

model_check_result.txt is automatically generated after Prepare is completed, and can also be generated separately through check_qat_model. It puts model structure inspection, operator fusion inspection, call-count inspection, and qconfig inspection results together, and is the most important overview artifact in the Prepare stage.

If you only want to do one overall check right after Prepare, this file is usually the first one to inspect.

In actual use, it is recommended that you focus on the following kinds of information:

• Whether there are patterns that could be fused but were not fused.

• Whether the call count of each module is normal, and whether there are shared modules.

• Whether the input / weight / output dtype of each layer is generally consistent with the target configuration.

• Whether there are abnormal qconfig hints that require extra attention, such as fixed scale, configuration conflicts, or suspicious settings.

This file is most suitable for answering the following questions:

• Whether operators were fused successfully.

• Whether there are shared modules.

• Whether qconfig obviously deviates from expectations.

If you have already found obvious issues here, it is usually recommended that you first go back to Prepare Description and Build QConfig, instead of continuing directly to Calibration or consistency analysis.

#fx_graph.txt

fx_graph.txt is generated after the graph-mode execution of Prepare, and is used to save the actual captured FX graph. Compared with model_check_result.txt, it does not directly tell you what is wrong, but it is more suitable for confirming what graph the tool is ultimately using to process the model.

If you suspect that the training path and deployment path are not completely consistent, or suspect that certain modules or branches did not really enter the graph, then this file should usually be checked first.

In actual use, key points to focus on include:

• Whether the final captured execution path is consistent with expectations.

• Whether certain modules, branches, or loops really entered the graph.

• Whether module replacement, operator fusion, and quantization-node insertion positions are generally reasonable.

In the following scenarios, fx_graph.txt is often especially worth checking first:

• There are dynamic branches or a deploy switch in forward.

• Some modules are defined in the code, but later results suggest that they did not take effect.

• You suspect that the training path and deployment path are not completely consistent.

If fx_graph.txt itself is already obviously inconsistent with the expected structure, it is usually not recommended to continue troubleshooting accuracy issues. Instead, you should first go back to the Prepare stage and adjust the model implementation, example_inputs, or related configuration.

#qconfig_dtypes.pt

qconfig_dtypes.pt is automatically saved after Prepare is completed. It is the saved file of operator-level quantization configuration, and is mainly used for reusing existing configuration.

If you later want to continue using the current configuration through LoadFromFileTemplate, or want to preserve the current quantization result as the starting point for subsequent tuning, this file becomes important.

However, if your current goal is only to quickly inspect whether the configuration is as expected, this file is usually not the first one to check. qconfig_dtypes.pt.py is more intuitive for direct inspection.

#qconfig_dtypes.pt.py

qconfig_dtypes.pt.py is also automatically saved after Prepare is completed. It is the readable version of qconfig_dtypes.pt, and is usually the most suitable one for direct manual inspection.

If what you care about most right now is “what dtype each key module was actually configured to use,” this file is usually more useful than the .pt file.

In actual use, key points to focus on include:

• The input / weight / output dtype of key modules.

• Whether key operators such as Conv / Linear / Matmul were really configured as qint8, qint16, or torch.float16.

• Whether the current mixed-precision configuration generally falls into the region you expect.

This file is most suitable for the following scenarios:

• Manually checking whether key modules such as backbone, neck, and head use the expected dtype.

• Judging whether the basic mixed-precision configuration has generally taken effect.

If you find that the result does not completely match your intuition from the template, it is not recommended to directly infer that “I only need to manually add a few dtype lines.” Instead, continue combining it with qconfig_changelogs.txt to determine which template, override relationship, or default optimization changed the result.

#qconfig_changelogs.txt

qconfig_changelogs.txt is automatically saved after Prepare is completed, and is used to record the qconfig change process. If qconfig_dtypes.pt.py is more suitable for answering “what the final result is,” then qconfig_changelogs.txt is more suitable for answering “why it finally became this result.”

When you suspect that the final dtype of an operator is inconsistent with your intuition from the template, or want to confirm whether the template override order is as expected, this file is usually the most worth checking.

It is especially suitable for helping you determine the following kinds of problems:

• Which template changed the qconfig of a specific module.

• Whether the template override order is as expected.

• Whether there are unexpected overrides, rollbacks, or simplifications.

It is especially useful in the following scenarios:

• You clearly configured higher precision, but it did not finally take effect.

• After combining global and local templates, the result is inconsistent with your intuition.

• After loading qconfig_dtypes.pt, you continue modifying dtype, and the result becomes unstable.

If you eventually confirm that the problem mainly comes from the template configuration relationship itself, it is recommended that you go back to Build QConfig and adjust the template, rather than staring only at the final artifact and patching the result temporarily.

#Error-Analysis and Consistency-Analysis Artifacts

When the Prepare stage is already basically normal, but Calibration, QAT, or deployment-side results are still unsatisfactory, what you usually need to focus on next is no longer “whether the workflow was built correctly,” but rather “where the error is concentrated, which operators are most sensitive, and what should be adjusted first next.”

At this point, artifacts related to QuantAnalysis are usually more helpful.

#compare_per_layer_out_0.txt

compare_per_layer_out_0.txt is generated after running QuantAnalysis.compare_per_layer(), and is the core artifact for per-layer comparison. It provides information in tabular form for each layer, including shape, quantized dtype, qscale, COSINE, L1, ATOL, max_qscale_diff, and the statistics range before and after comparison. It is usually one of the most important files for locating issues layer by layer.

If Calibration accuracy is unsatisfactory, QAT accuracy is abnormal, or consistency issues occur in the Export / Convert stage, you should usually go back to this file to determine from which layer the error starts to amplify.

In actual troubleshooting, the following kinds of information deserve particular attention:

• From which layer onward the output metrics of the two models begin to deteriorate obviously.

• Whether the quant_dtype and qscale of the abnormal layer are reasonable.

• Whether the minimum, maximum, and mean values of base_model_* and analy_model_* deviate obviously.

• Whether max_qscale_diff is abnormally large.

This file is especially suitable for helping you answer the following questions:

• From which layer the issue starts to amplify clearly.

• Whether the scale of a certain layer may be unreasonable.

• Whether the current error looks more like truncation error or rounding error.

When judging whether the issue looks more like truncation or rounding, you can first use the following rules of thumb:

• If the numeric range of the model under analysis is obviously compressed, or compared with the baseline model you observe boundary shrinkage, saturation, clipping, and similar effects, then the issue usually looks more like truncation error or unreasonable scale setting.

• If the overall numeric ranges of the two models are close and none of the truncation symptoms above appear, then the issue usually looks more like rounding error, lookup-table approximation, or implementation-detail differences.

#output_xxx_case_xxx_sensitive_ops.txt/.csv/.pt

These files are generated after running QuantAnalysis.sensitivity(), and are among the most commonly used artifacts for mixed-precision tuning and key-layer localization.

Files are generated separately for each output and each badcase, such as output_0_case_3_sensitive_ops.csv. In actual use, it is recommended that you first choose the output file most relevant to your current business metric, and then start the analysis.

For accuracy tuning, it is recommended to generate results through sensitivity(use_sensitivity_v2=True). In this case, each case generates txt, csv, and pt formats at the same time, plus two summary tables and one HTML report:

• csv: most suitable for analysis, since you can filter and sort by column in software such as Excel.

• txt: more suitable for quick manual reading.

• pt: more suitable for script loading, or for subsequent automated configuration together with SensitivityTemplate.

• output_xxx_case_mean_sensitive_ops.*: the average sensitivity across multiple badcases.

• output_xxx_case_worst_sensitive_ops.*: the worst sensitivity across multiple badcases.

• sensitivity_v2_report.html: view the trend after sensitivity ranking as charts, which makes it easier to decide from which operator in the ranking you should start setting qint8 to qint16.

It is recommended to check the csv first, and judge with the single-case, average, and worst views together, which gives a more stable conclusion than looking at only a single badcase.

These files are usually most suitable for the following scenarios:

• During mixed-precision tuning, judging which operators are most worth upgrading first to qint16 or torch.float16.

• When the model has multiple outputs, determining which output branch is more strongly affected by the current accuracy drop.

In actual inspection, key points to focus on include:

• The current quantized dtype used by the highly ranked sensitive operators.

• Whether these operators are still qint8, and whether it is worth trying to promote them to qint16 or torch.float16.

• With v2 enabled, the value in the qint8-qint16 column. It indicates how much sensitivity can be reduced by switching this fake stub from int8 to int16. The larger the reduction, the greater the accuracy gain of setting it to qint16.

• With v2 enabled, the relative magnitude of the truncation (clip) and rounding (round) columns, which helps determine which of the two the error of this layer mainly comes from.

• Whether the proportion of computation in these operators allows you to pay the cost of higher precision for them.

• Whether sensitivity mainly comes from activation, weights, or a specific input.

If an operator is already high precision but still ranks near the top, this often means that the issue is not simply “insufficient precision type,” and you still need to continue combining it with compare_per_layer_out_0.txt to see how its numeric distribution and error are produced.

#output_xxx_tuned_case_xxx_sensitive_ops.txt/.csv/.pt

These files are generated after running QuantAnalysis.tune_bad_case(), and are used to verify the real gain of the sensitivity conclusion.

On top of the sensitivity table, they add a tuned_cumulative_{metric} column, which indicates the cumulative improvement of the model output error after changing the current fake stub from qint8 to qint16. Only nodes that actually bring a gain have a value; nodes without gain are rolled back and keep their original state.

This artifact is most suitable for answering one question: when promoting precision in sensitivity order, up to which operator is enough.

#The advice Column in Per-Layer Comparison Results

The last column advice of compare_per_layer_out_0.txt gives diagnostic suggestions for the layer, which works as a built-in abnormal-layer hint. It helps you quickly scan out layers that deserve attention first, such as layers with excessively large data range, inf or NaN, outputs without high-precision protection, detected truncation error, or inconsistent inputs between the two models.

Therefore, it is more suitable as an entry point for “quickly narrowing the scope,” rather than as the final conclusion.

This column is most suitable for the following scenarios:

• There are too many per-layer results, and you do not yet know which layers should be checked first.

• You want to quickly filter out obviously abnormal layers first, and then go back to the per-layer comparison results for further checking.

In actual use, it can usually be understood as follows:

• If the advice of certain layers already gives an explicit hint, it means that these layers are worth checking first together with the scale, numeric range, and error metrics in the same row.

• If advice is empty, that still does not mean that the model definitely has no problem. You still need to continue judging together with per-layer comparison and sensitivity results.