Convert to Fixed-Point Model

After the series of steps described earlier, if the pseudo-quantization accuracy already meets the requirement, you can proceed with the model deployment flow, which mainly includes exporting the HBIR model (export) and converting it to a fixed-point model (convert).

You can first understand these two steps as follows:

export is responsible for exporting the pseudo-quantized model on the torch side into HBIR representation.
convert is responsible for further converting the exported HBIR model into a true fixed-point model.

Stage	Main Input	Main Output	Responsibility	Has Entered True Fixed-Point
`Export`	`qat_model / calib_model` `example_inputs`	`qat.bc`	Exports the HBIR computation graph according to the `forward` logic required for deployment	No, it is still in the export stage
`Convert`	`qat.bc`	`quantized.bc`	Further converts the pseudo-quantized HBIR into a true fixed-point model	Yes, it has entered the true fixed-point stage

The relationships among several common and easily confused artifacts in the export and convert stages are as follows:

Artifact	Type	Obtained At	Description	Common Use
`qat.pt`	torch model	after `prepare` / `calibration` / `qat`	The pseudo-quantized model on the torch side	Continue with Calibration, QAT, and Export
`qat.export.pt`	torch model	after `pre_export`	A torch model that only completes lookup-table-based fixed-point conversion, and its computation logic is closer to `qat.bc`	Used for export consistency analysis and lookup table issue localization
`qat.bc`	HBIR model	after `export`	The HBIR model exported from `qat.pt`, which is still an artifact of the export stage	Used for export result verification and as convert input
`quantized.bc`	HBIR model	after `convert`	The model obtained by fixed-point converting `qat.bc`	Used for fixed-point accuracy verification before compilation

In the examples below in this chapter, qat_model is consistently used as the sample input. If you are currently following a Calibration-only route, you can also directly replace qat_model here with calib_model.

Export: Export HBIR Model

Before converting to a true fixed-point model, you first need to export the pseudo-quantized model as an HBIR model. During export, you need to provide a set of example_inputs. The tool uses these inputs to execute the complete forward logic of the model, and rewrites each executed operator into an HBIR node or subgraph.

An example is shown below:

from horizon_plugin_pytorch.quantization import FakeQuantState, set_fake_quantize
from horizon_plugin_pytorch.quantization.hbdk4 import export

base_model = qat_model

base_model.eval()
set_fake_quantize(base_model, FakeQuantState.VALIDATION)

hbir_qat_model = export(base_model, (example_input,))

Attention

The exported HBIR model contains not only quantized operators, but all operators actually executed in the model forward. Therefore, when exporting HBIR, make sure that the model forward logic is consistent with the final deployment logic.
HBIR does not support dynamic shape. The shapes throughout the computation graph are determined by the tensor shapes in the model at export time. If you need to use different batch sizes for simulation and on-board deployment, please export separate HBIR models with different data.
If you currently mainly want to verify whether the deployment pipeline can run through, you can also skip quantization accuracy tuning for now. After the initial Calibration is completed, directly run the Export / Convert flow first, so you can confirm as early as possible whether the model contains operations that cannot be exported or compiled.

Note

Accuracy Notes

The HBIR model obtained after export is theoretically consistent with the torch model in computation logic, but the following differences may prevent numerical alignment:

Most nonlinear elementwise operators in the torch model are converted into lookup table implementations in HBIR.
Operators with accumulation computations such as reduce_sum and gemm may show numerical fluctuations because the accumulation order is different.

If you find that qat.pt / qat_model is normal but qat.bc produces abnormal results, you should usually first suspect that the problem occurs in the export stage.

Convert: Convert to Fixed-Point Model

After obtaining the exported HBIR model, you can continue converting the model to a fixed-point model. Convert is the step where the model truly enters the fixed-point stage, and the output artifact is quantized.bc. In terms of accuracy, we generally consider the result of the fixed-point model to be fully consistent with the result of the compiled model.

Note

Before and after Convert, the model changes from floating-point pseudo-quantized computation to true fixed-point computation, so numerical fluctuations are normal.

In general, before compilation, quantized.bc is the fixed-point accuracy baseline that is most worth checking first.

An example is shown below:

from hbdk4.compiler import convert

quantized_bc = convert(
    qat_bc,
    "nash-e",
)

Attention

HBIR model inputs support only a single Tensor or Tuple[Tensor], and outputs support only Tuple[Tensor].
A fixed-point model and a pseudo-quantized model cannot be numerically identical, so please use the accuracy of the fixed-point model quantized.bc as the reference. If the accuracy of quantized.bc does not meet the requirement, you need to continue quantization aware training or proceed with consistency analysis until the accuracy meets your requirement.

Fixed-Point Model Accuracy Verification

After convert is completed, we recommend that you first verify the accuracy of quantized.bc, and then decide whether to continue with compile.

During verification, we recommend that you pay special attention to the following points:

The batch size of eval_hbir_data_loader needs to stay consistent with the example_inputs used during export.
The default input and output names are usually _input_{n} and _output_{n}. If you customized them during export, you also need to update them here accordingly.

An example is shown below, using the quantized_bc obtained from the convert step above:

from typing import Tuple

import hbdk4.compiler as hb4
import torch.utils.data as data

# Dataloader used for HBIR accuracy evaluation. Note that the batch_size here
# must be the same as the example_input used when exporting HBIR.
_, eval_hbir_data_loader = prepare_data_loaders(
    data_path, train_batch_size, 1
)

def evaluate_hbir(
    model: hb4.Module, data_loader: data.DataLoader
) -> Tuple[AverageMeter, AverageMeter]:
    top1 = AverageMeter("Acc@1", ":6.2f")
    top5 = AverageMeter("Acc@5", ":6.2f")

    for image, target in data_loader:
        image, target = image.cpu(), target.cpu()

        # The default input and output names follow the _input_{n} / _output_{n}
        # pattern. If you customized them during export, update them here too.
        output = model["forward"].feed({"_input_0": image})["_output_0"]
        acc1, acc5 = accuracy(output, target, topk=(1, 5))
        top1.update(acc1, image.size(0))
        top5.update(acc5, image.size(0))

    return top1, top5


# Test the fixed-point model accuracy
top1, top5 = evaluate_hbir(
    quantized_bc,
    eval_hbir_data_loader,
)
print(
    "Quantized model: evaluation Acc@1 {:.3f} Acc@5 {:.3f}".format(
        top1.avg, top5.avg
    )
)

If the accuracy of quantized.bc already meets the requirement, you can continue reading Model Compilation and Performance Evaluation.

Consistency Risk Reminder

During the Export / Convert stage, if the result does not match your expectation, we recommend that you first use the following approach to quickly determine where the problem occurs:

If qat.pt / qat_model is normal but qat.bc is abnormal, first determine whether the issue occurs in the export stage. Focus on lookup-table-based fixed-point conversion, whether the exported graph is consistent, and whether the deployment logic matches the training-side logic.
If qat.bc is normal but quantized.bc is abnormal, first determine whether the issue occurs in the convert stage. Focus on the numerical differences introduced during fixed-point conversion.
If quantized.bc is normal but hbm is abnormal, first determine whether the issue occurs in the compile / deployment stage. Focus on input and output handling, pre/post-processing adaptation, and on-board deployment differences.

If you need to continue troubleshooting consistency issues in the export / convert / compile stages, you can continue to refer to Deployment Consistency Analysis.

#Convert to Fixed-Point Model

#Export: Export HBIR Model

#Convert: Convert to Fixed-Point Model

#Fixed-Point Model Accuracy Verification

#Consistency Risk Reminder

Convert to Fixed-Point Model

Export: Export HBIR Model

Convert: Convert to Fixed-Point Model

Fixed-Point Model Accuracy Verification

Consistency Risk Reminder