• English
  • 3.8.1

    • Convert to Fixed-Point Model

    After the pseudo-quantization accuracy meets the standard through the series of processes described above, the relevant processes for model deployment can be executed, mainly including operations such as exporting the HBIR model (export) and converting to a fixed-point model (convert).

    Export HBIR Model

    Model deployment requires exporting the pseudo-quantization model as a HBIR model firstly. A set of example_inputs needs to be provided. The tool executes the complete forward logic of the model using these inputs, and converts each operator encountered during execution into an HBIR node or subgraph one by one.

    Attention
    • The exported HBIR does not only contain quantized operators, but all operators executed in the model’s forward pass. Therefore, when exporting HBIR, please ensure that the model’s forward logic is consistent with the required deployment logic.
    • HBIR does not support dynamic shapes. The shape of each part in the computation graph is determined by the shape of the tensors in the model at the time of export. If you need to use different batch sizes for simulation and on-board deployment, please export HBIR models separately using different example_inputs.
    • You can also choose to skip the quantization accuracy tuning process. After the initial completion of calibration, first directly verify the model deployment process to ensure that there are no operations in the model that cannot be exported or compiled.
    Note

    Accuracy Description

    The exported hbir model is theoretically consistent with the torch model in terms of calculation logic, but there are the following differences that lead to numerical inconsistency:

    1. Most nonlinear elementwise operators in the torch model are converted to a lookup table implementation in hbir.

    2. Operators with accumulation calculations, such as reduce_sum and gemm, will cause numerical fluctuations due to different accumulation orders.

    ######################################################################
# The user can modify the following parameters as required.
# 1. Which model to use as input for the process, you can choose either calib_model or qat_model.
base_model = qat_model
######################################################################

from horizon_plugin_pytorch.quantization import FakeQuantState, set_fake_quantize
from horizon_plugin_pytorch.quantization.hbdk4 import export

qat_model.eval()
set_fake_quantize(qat_model, FakeQuantState.VALIDATION)
hbir_qat_model = export(base_model, (example_input,))

    Convert to Fixed-point Model

    After exporting the model to the HBIR model, the model can be converted to a fixed-point model. The results of the fixed-point model are generally considered to be identical to those of the compiled model.

    Attention
    • HBIR models support only a single Tensor or Tuple[Tensor] as input, and only Tuple[Tensor] as output.
    • It is not possible to achieve complete numerical agreement between the fixed-point model and the pseudo-quantization model, so please take the accuracy of the fixed-point model as the standard. If the fixed-point accuracy is not up to standard, you need to continue the quantized awareness training.
    Note

    Accuracy Description

    After convert, the model changes from floating-point pseudo-quantization computation to int computation, which will lead to numerical fluctuations.

    from hbdk4.compiler import convert

# Transform the model to a fixed-point state, note that the march here needs to be distinguished from nash-e/m/p.
hbir_quantized_model = convert(
    hbir_qat_model,
    "nash-e",
)

# Dataloader for test accuracy of HBIR model. Please note that the batch size
# should be same as the example input when exporting HBIR.
_, eval_hbir_data_loader = prepare_data_loaders(
    data_path, train_batch_size, 1
)

def evaluate_hbir(
    model: hb4.Module, data_loader: data.DataLoader
) -> Tuple[AverageMeter, AverageMeter]:
    top1 = AverageMeter("Acc@1", ":6.2f")
    top5 = AverageMeter("Acc@5", ":6.2f")

    for image, target in data_loader:
        image, target = image.cpu(), target.cpu()
        # Default inpujt/output names are _input_{n}, _output_{n}，users can
        # modify them by params when export HBIR.
        output = model["forward"].feed({"_input_0": image})["_output_0"]
        acc1, acc5 = accuracy(output, target, topk=(1, 5))
        top1.update(acc1, image.size(0))
        top5.update(acc5, image.size(0))

    return top1, top5


# Test the accuracy of fixed-point models.
top1, top5 = evaluate_hbir(
    hbir_quantized_model,
    eval_hbir_data_loader,
)
print(
    "Quantized model: evaluation Acc@1 {:.3f} Acc@5 {:.3f}".format(
        top1.avg, top5.avg
    )
)