When you use the PTQ workflow to compile ONNX models for quantization, the flow from original_float.onnx to the final hbm used for deployment goes through multiple steps: optimize, calibrate, ptq, quantize, compile, and deployment adaptation. If any of these steps introduces additional error, you may observe the situation of "results are normal on the training side, but do not meet expectations on the deployment side".

This section describes how to avoid such issues on the ONNX model quantization link, and, when they occur, how to perform stage-by-stage initial localization and prepare artifacts before contacting Horizon technical support.

#Model Artifacts Involved in the ONNX Quantization Link

When you use the PTQ workflow to compile ONNX models for quantization, the following artifacts are generated stage by stage (where xxx represents your model name):

File name	Description
xxx_original_float_model.onnx	Original floating-point model. Performs infer shape on your model and applies legality checks on operators and tensors. Serves as the baseline model for subsequent processing.
xxx_optimized_float_model.onnx	Optimized floating-point model. Applies a series of equivalent model optimizations to improve on-board execution performance, including but not limited to: Constant Folding Operator Replacement Operator Fusion Operator Elimination Operator Reordering
xxx_calibrated_model.onnx	Fake-quantized model. Applies fake quantization to the model according to your quant_config, simulating quantization behavior of different bit widths in the floating-point domain.
xxx_ptq_model.onnx	PTQ model. Applies fixed-point processing to lookup-table (LUT) operators to more closely approximate real fixed-point execution behavior.
xxx_quantized_model.bc	Fixed-point model (bc format). Converts the ONNX model into an HBIR model (bc format) and completes full-model fixed-pointization.
xxx_quantized_removed_model.bc	Fixed-point model (bc format, node-removal scenario). When model quantization involves removing input/output-side nodes, this model is saved automatically. In this scenario, when performing consistency comparison against hbm, this model should be used to align with hbm instead of xxx_quantized_model.bc.
xxx.hbm	On-board executable model. Compiles the HBIR model into an instruction file that can be directly loaded and executed on the board.

#Quick Judgment of the Consistency Analysis Path

In practice, we recommend not running the full tool set from the start. Use the observed symptom to first judge which stage the issue is more likely to fall in, then enter the detailed analysis of the corresponding stage below.

Current symptom	Suspected stage	Recommended first action
optimized_float and original_float outputs do not match	Model optimization stage	Equivalent optimizations should not introduce deviations in theory. Contact Horizon technical support directly and provide original_float.onnx and optimized_float.onnx
calibrated and optimized_float outputs do not match	Model calibration stage	This stage introduces quantization error and inevitably has some accuracy loss, which is different from the concept of consistency. If the quantized accuracy does not meet expectations, prefer using the accuracy_debug tool for layer-by-layer sensitivity analysis, and refer to the ONNX Model Accuracy Tuning Guide section to adjust quant_config. Contact Horizon technical support if the accuracy still does not meet expectations
ptq and calibrated outputs do not match	Lookup-table operator fixed-pointization stage	Typically contact Horizon technical support, and prepare the full set of original / optimized / calibrated / ptq onnx files
quantized.bc and ptq.onnx outputs do not match	ONNX to HBIR fixed-pointization stage	First export a version of quantized.bc and ptq.onnx without any model modification for comparison, to rule out deviations introduced by the model modification itself. If they still do not match, first investigate input-preparation deviations caused by structural differences such as input_type_rt / mean-scale / Pyramid or Resizer insertion. Contact Horizon technical support if it still does not meet expectations
hbm and quantized.bc (or quantized_removed_model.bc) outputs do not match	Compilation / on-board deployment stage	If the model contains floating-point operators (fp16, fp32), quantized.bc and hbm cannot be bit-aligned. Judge consistency based on dataset accuracy, stable reproducible bad cases, and multi-frame visualization rather than single-frame values. Aside from this special case, if the model has no CPU operators, first use hb_verifier to verify overall output consistency. If it passes, investigate on-board pre/post-processing code yourself. If it fails, contact Horizon technical support directly. If the model contains CPU operators, first suspect input preparation / on-board pre/post-processing / stride alignment / tail output type platform differences (int32 vs fp32). Refer to the Impact of Platform Differences on Consistency Interpretation subsection below. Contact Horizon technical support when the root cause cannot be located

#Impact of Platform Differences on Consistency Interpretation

When performing consistency comparison, common platform difference points are as follows (for detailed entries, refer to the Platform Differences section):

• Tail high-precision output type difference: On J6E/M, the tail high-precision operator output type is int32; on J6B/H/P, the tail high-precision operator output type is fp32. On-board post-processing code needs to be adapted per platform; otherwise the consistency comparison will show a difference.

• nv12 stride alignment differences: On J6H/P, nv12 input must satisfy stride 64 alignment. On J6E/M/B, nv12 input must satisfy stride 32 alignment. When on-board input preparation is not aligned per platform, hbm output and quantized.bc output will be inconsistent.

• Impact of input_no_padding / output_no_padding: Whether these two switches are enabled directly affects on-board input preparation and hbm output form. Input data preparation for consistency comparison needs to match the actual enabled state.

• Alignment between quantized_removed_model.bc and hbm: When model quantization involves removing input/output-side nodes, consistency comparison should align hbm with quantized_removed_model.bc rather than quantized_model.bc.

#Tools Available for Consistency Analysis

Before stage-by-stage investigation, align on the tools available for consistency analysis and their intended usage:

Tool	Applicable scope	When to use
hb_verifier	ONNX vs ONNX / ONNX vs HBIR / HBIR vs HBIR cosine similarity comparison, HBIR vs HBM output consistency comparison	Used for per-node cosine similarity comparison between any two stages of models to locate the source of error. HBM only supports final-output consistency comparison; per-node analysis is not supported
hb_analyzer	Model structure and operator deployment distribution analysis	Used to view CPU / BPU operator distribution, and to help judge whether structural differences affect consistency interpretation
hbm_infer	X86 + on-board joint HBM inference	Used to obtain HBM output with pre/post-processing on the X86 side and actual inference executed by the on-board Server, serving as one of the data sources for on-board consistency comparison. Note that actual inference still runs on the board, so it is affected by the on-board environment
hrt_model_exec	On-board HBM inference	Used to load HBM on-board for real I/O measurement, serving as the on-board data source for consistency comparison

#Common Analysis Results and Handling Suggestions

#Case 1: Cosine similarity of all operator layers and the final output is normal

• No anomaly is found in per-layer cosine similarity.

• Suggested handling:

  1. First check whether the input data pre-processing flow in the accuracy verification pipeline (such as normalization, color space, dimension order, etc.) is consistent with the training stage.

  2. If data processing is confirmed correct, contact Horizon technical support for problem analysis.

#Case 2: Cosine similarity of all operator layers is abnormal

• Usually caused by input data anomalies or format mismatches.

• Suggested handling:

  1. Check the source, format, and pre-processing logic of the input data.

  2. If input data is confirmed correct, contact Horizon technical support for problem analysis.

#Case 3: Cosine similarity of some operator layers is significantly low

• Indicates that the error may be introduced at a specific model stage or a specific operator.

• Suggested handling: Based on intermediate artifacts, use hb_verifier to perform stage-by-stage consistency comparison analysis. For the specific method, refer to the detailed description of the per-stage consistency analysis below.

#Per-Stage Consistency Analysis

The following provides, for each stage, the typical symptoms and theoretical expectations, the tools available at this stage, the impact of structural changes on consistency interpretation, and a checklist of artifacts to keep when contacting Horizon technical support.

#xxx_original_float_model.onnx → xxx_optimized_float_model.onnx

Typical symptoms and theoretical expectations

• This stage contains only equivalent model optimizations (constant folding, operator replacement, operator fusion, redundant operator elimination, operator reordering).

• The expected output cosine similarity should be 100%, and model accuracy should remain identical.

Available tools

• hb_verifier: performs per-node cosine similarity comparison for ONNX → ONNX.

• hb_analyzer: confirms whether the operator deployment distribution before and after optimization meets expectations.

Structural change impact reminders

• This stage does not involve input-side structural changes such as input_type_rt / Pyramid / Resizer.

Artifacts to keep when contacting Horizon

• xxx_original_float_model.onnx.

• xxx_optimized_float_model.onnx.

• hb_compile.log.

• The input.npy used for reproduction.

#xxx_optimized_float_model.onnx → xxx_calibrated_model.onnx

This stage introduces fake quantization simulation. According to your quant_config, operators are simulated to the corresponding bit width. This process inevitably introduces numerical error, which belongs to the quantization error category and is not a consistency issue. If xxx_calibrated_model.onnx accuracy drops, refer to the ONNX Model Accuracy Tuning Guide section for accuracy tuning. This is not covered in the current section.

#xxx_calibrated_model.onnx → xxx_ptq_model.onnx

Typical symptoms and theoretical expectations

• This stage applies fixed-point processing to lookup-table (LUT) operators, which may further introduce error.

• If xxx_ptq_model.onnx accuracy drops, it usually needs to be located by contacting Horizon technical support.

Available tools

• hb_verifier: performs per-node cosine similarity comparison for ONNX → ONNX to locate similarity changes at lookup-table operators.

Structural change impact reminders

• This stage does not involve input-side structural changes.

Artifacts to keep when contacting Horizon

• xxx_original_float_model.onnx.

• xxx_optimized_float_model.onnx.

• xxx_calibrated_model.onnx.

• xxx_ptq_model.onnx.

• hb_compile.log.

• The input.npy used for reproduction.

#xxx_ptq_model.onnx → xxx_quantized_model.bc

Typical symptoms and theoretical expectations

• Converting the ONNX model to the HBIR model (bc format) should be logically equivalent, but operator fixed-pointization may further introduce error.

• If xxx_quantized_model.bc and xxx_ptq_model.onnx outputs do not match, first investigate structural differences on the input-preparation side.

Available tools

• hb_verifier: performs per-node cosine similarity comparison for ONNX → HBIR.

• hb_analyzer: confirms the operator deployment distribution after conversion.

Structural change impact reminders

• When input_type_rt is nv12, the input layout of the corresponding quantized.bc is NHWC, which differs from the input form of ptq.onnx. Input data for consistency comparison must be prepared according to the corresponding layout. Refer to the inference-preparation subsection in the Data Preparation section.

• The color conversion from input_type_rt to input_type_train and the mean/scale handling are baked into quantized.bc. The input preparation for consistency comparison must match accordingly.

• Pyramid / Resizer input pipelines insert corresponding nodes at the model input side; directly using ptq.onnx input data cannot reproduce them. Refer to the inference-preparation subsection in the Data Preparation section to configure inputs.

Artifacts to keep when contacting Horizon

• xxx_ptq_model.onnx.

• xxx_quantized_model.bc.

• The quant_config.yaml / quant_config.json used.

• hb_compile.log.

• The input.npy used for reproduction (including the version that matches input_type_rt).

#xxx_quantized_model.bc → xxx.hbm

Typical symptoms and theoretical expectations

• Compilation from the HBIR model (bc format) to the HBM model should not, in theory, introduce numerical inconsistency for integer fixed-point operators.

• If bc and hbm outputs do not match, first investigate on-board input preparation, pre/post-processing, stride alignment, and cross-platform output type differences.

Available tools

• hb_verifier: performs HBIR → HBM final-output consistency comparison (HBM does not support per-node comparison).

• hbm_infer: obtains HBM output with pre/post-processing on the X86 side and actual inference executed by the on-board Server, serving as one of the data sources for on-board consistency comparison.

• hrt_model_exec: on the board, loads HBM to measure real I/O, serving as the on-board data source for consistency comparison.

Structural change impact reminders

• If model quantization involves removing input/output-side nodes, consistency comparison should align hbm with quantized_removed_model.bc instead of quantized_model.bc.

• On J6H/P, on-board nv12 input must satisfy the stride 64 alignment requirement. Whether input_no_padding / output_no_padding is enabled affects on-board padding handling and the input data preparation for consistency comparison. See the Impact of Platform Differences on Consistency Interpretation subsection.

• Tail high-precision output type platform differences (J6E/M is int32; J6B/H/P is fp32) manifest in the deployment-side output parsing code. If on-board post-processing is not adapted, this will appear as a consistency difference.

Artifacts to keep when contacting Horizon

• xxx_quantized_model.bc (or xxx_quantized_removed_model.bc).

• xxx.hbm.

• The quant_config.yaml / quant_config.json used.

• hb_compile.log.

• The output of hbm_infer.

• The on-board measurement output of hrt_model_exec.

• The input.npy used for reproduction and the corresponding on-board input.

#Suggestions After the Analysis

• If the consistency comparison result points to quantization configuration or calibration data issues, refer to the platform experience template subsection in the quant_config Description section, as well as the ONNX Model Accuracy Tuning Guide section.

• If the consistency comparison result points to structural differences or tool-side errors, organize artifacts per the "Artifacts to keep when contacting Horizon" checklist for each stage, and then contact Horizon technical support.