#ONNX Model Quantization Principle and Process

#Overview

The process of the model from training to conversion to running on the development board is shown below:

Model Training: It is the process of getting a usable model by using public deep learning frameworks such as TensorFlow, PyTorch, Caffe, etc. The trained model will serve as the input for the model conversion stage. The toolchain does not provide training-related libraries or tools. For the detailed supported public learning frameworks, please refer to the instructions in the Floating-point Model Preparation.

Model Conversion: Taking the floating-point model obtained from model training as input, this stage transforms floating-point models into models that can run efficiently on the Horizon computing platform through important steps such as model structure optimization, model calibration quantization and model compilation. For more information, please refer to Model Quantization and Compilation section.

Embedded Application development: The toolchain supports application development in both X86 simulation environment and real embedded environment. In case you are not convenient to use the development board, you can debug the program and verify the calculation results in the simulation environment. In order to reduce the cost of simulation verification, the toolchain provides the exact same simulation library interface as the embedded interface, only with different compilation configurations. For more information, please refer to Embedded Application Development section.

#ONNX Model Quantization Terminology and Artifact Reference

Before diving into each stage, use the table below to align on the key terminology and artifacts involved in the ONNX model quantization pipeline:

#ONNX Model Quantization Full-pipeline Stage Table

Combined with the Horizon toolchain, the ONNX model quantization pipeline is split into the following stages in actual usage order: Floating-point Model Preparation → Model Checking → Calibration Data Preparation → quant_config Configuration (yaml: Model Quantization and Compilation, json: quant_config specification) → Model Quantization and Compilation → Performance Evaluation → Accuracy Evaluation → Deployment Consistency Verification. The overall flowchart is shown below:

The target, input, output, checkpoints and troubleshooting jumps of each stage are summarized in the following table:

#Preliminary Notes

Before entering the detailed stages, please note the following:

• The quantized fixed-point model represents the final on-board accuracy. quantized.bc and hbm are theoretically consistent in accuracy, the final accuracy shall be based on either quantized.bc or hbm.

• The recommended quant_config strategy varies by platform. Please refer to the unified Platform Difference Description section for details.

• If the floating-point accuracy itself does not meet the target, please first go back to the training framework for investigation instead of tuning during the quantization stage. The ONNX model quantization pipeline cannot fix issues that originate from the floating-point model itself.

• Only when the ONNX model quantization accuracy remains below target for a long time should you consider switching to the pytorch QAT pipeline. The two pipelines are not replacements for each other.

#Model Conversion Process Detailed Explanation

So how to convert the floating-point models trained by using the opensource ML frameworks (such as Caffe, TensorFlow, PyTorch, etc.) to the Horizon hardware supported fixed-point models?

In most cases, the threshold values and weights of the floating-point models obtained from either the opensource ML frameworks or trained by yourself are floating-point numbers (float32) and each number occupies 4 bytes.

However, by converting the floating-point numbers to lower-bit computational numbers, thus the computation operations in the embedded runtime can be dramatically reduced.

Therefore, it brings significant performance boost by converting the floating-point models to lower-bit computational models with no loss or very small loss.

Typically, model conversion can be divided into the following steps:

1. Check if there are unsupported OPs in the models to be converted.

2. Prepare 20~100 images for calibration use at the conversion stage.

3. Convert the floating-point models to fixed-point models using the floating-point conversion tools.

4. Evaluate the performance and accuracy of the converted models to ensure that there isn't huge difference in model accuracy before and after the conversion.

5. Run models in simulator/dev board to validate model performance and accuracy.

#Three Usage Modes of hb_compile

The model checking, quantization and compilation tasks in the steps above can be performed by the hb_compile tool. For the tool's usage and detailed configuration/parameters, please refer to the Model Quantization and Compilation section. Under the ONNX model quantization pipeline, you can choose from the following three usage modes based on your target:

#Model Checking

Before converting the floating-point models into the fixed-point models, we should check if there are Horizon hardwares unsupported OPs in the floating-point models using the hb_compile tool. If yes, the tool will report the unsupported OP(s). The usage of the hb_compile tool for checking the model can be found in Model Checking section.

#Calibration Image Preparation

When converting the floating-point models, you need to prepare 20~100 images for calibration use at the calibration stage.

Input image formats may vary by input type and layout. In this stage, because both original (JPG, etc.) and the processed images are valid, you can either feed the calibration images used in the model training or feed your own processed images.

We recommend you preprocessing the calibration images on your own, you need to complete the operations such as image channel (BGR/RGB), data layout (NHWC/NCHW), and image resizing/padding (Resize&Padding). The tool will then feed the images to the calibration stage after loading them as npy format.

Taking Resnet50 as an example, the required transformer operations are as follows:

    transformers = [
        PILResizeTransformer(size=256),                              # Adjust the short side to 256 with bilinear interpolation method
        PILCenterCropTransformer(size=224),                          # Perform center cropping with 224*224 with PIL method
        HWC2CHWTransformer(),                                        # Convert NHWC to NCHW
        ScaleTransformer(scale_value=1 / 255),                       # Scale pixel value ranges to [0,1]
        MeanTransformer(means=np.array([0.485, 0.456, 0.406])),      # Normalize the data by the average of the data set
        ScaleTransformer( 
            scale_value=np.array([1 / 0.229, 1 / 0.224, 1 / 0.225])) # Normalize the data by the standard deviation of the data set
    ]

#Model Conversion

When you confirm that the floating-point model can be successfully converted, you can then convert the floating-point model to a Horizon hardware supported fixed-point model by using the hb_compile tool.

This process requires you to pass a configuration file(*.yaml) containing conversion requirements. For specific configuration file settings and the insturctions of each parameter, refer to the descriptions in sections Specific Parameter Information and Configuration File Template .

When the model conversion process ends, it also prints the level of similarity between the floating-point model and fixed-point model to the log, you can therefore judge the similarity before and after conversion according to the Quantized Cosine field.

If the value of Quantized Cosine is very close to 1, so the performance of the fixed-point model should be very close to that of the floating-point model before the conversion.

After a successful model conversion, the model outputs, information files and the model static performance evaluation files for each phase will be generated in the generated folder (model_output by default), where the model output files will be used in subsequent phases.

#Single Image Inference

The accuracy of the fixed-point model generated by the floating-point conversion must be evaluated.

You should have good understanding of the input/output structures of the model. You should also be able to accurately preprocess the input images of the model, postprocess the model outputs, and write the model execution scripts on your own.

You can refer to the sample code in 04_inference.sh of the samples/onnx_ptq/resnet50_mnist example in the delivery package.

The code logic is as follows:

import numpy as np
# Load Horizon dependency library
from horizon_tc_ui.hb_runtime import HBRuntime
from postprocess import postprocess

# Prepare the input for model running, here the `input.py` is the processed data
data = np.load("input.npy") 
# Load model file
sess = HBRuntime("model.bc")
# Obtain the model input & output node information
input_names = sess.input_names
output_names = sess.output_names

# Model inference, here we assume the model has only one input
# The return value is a list that corresponds in order to the names specified by output_names
output = sess.run(output_names, {input_names[0]: data})

# Postprocessing
top_five_label_probs = postprocess(output)

Once the script used, you can verify its own accuracy by inputting a single image.

For example, the input to this script is a picture of a zebra, preprocessing the image data from rgb to the data type configured by input_type_rt(for informaiton about intermediate types, refer to the Model Conversion Interpretation ). Then, infer the model by passing the above image data by using the HB_HBIRRuntime command, post-processing after inference, and finally print out 5 of its most likely types.

The output of the script is shown as follows with the most possible class being label: 340:

The input picture is classified to be:
label 340: prob 0.97
label 292: prob 0.02
label 282: prob 0.00
label 83: prob 0.00

label uses the ImageNet label classes, where the corresponding class of 340 is zebra, so the inference result is correct.

#Model Accuracy Verification

It's insufficient to determine the model accuracy by single image inference, so you still need to use scripts to evalute the model accuracy after the conversion.

To do so, you need some coding work to enable the model to loop the image inference and compare the inference results with standard results to get model accuracy results.

In model accuracy evaluations, images must be pre-processed and the model data must be post-processed, so here we provide a Python script as a sample.

The logic of this script is the same as that of single image inference, yet it must run on the entire dataset.

The script can evaluate the model output results and generate evaluation results.

You can refer to the sample code in 05_evaluate.sh of the samples/onnx_ptq/resnet50_mnist example in the delivery package, which calls the eval.py evaluation script.

Because it takes a long time to run the script, eval.py provides a --parallel_num option to specify the number of parallel processes, which defaults to 1. You can append this option to the command to raise the parallelism as appropriate.

After the execution of this script, you can get the accuracy of the converted fixed-point model from the output of the script.

#[Reference] Supported Calibration Methods

We currently support the following calibration methods:

1. Default

Default is a strategy that automatically searches the calibrated quantization parameters to obtain a relatively good combination.

2. Mix

Mix is a search strategy that integrates multiple calibration methods, which automatically identifies quantization-sensitive nodes, selects the best from a group of calibration methods at node granularity, and finally build a hybrid calibration method absorbing the advantages of multiple calibration methods.

3. KL

KL learns from the Solution proposed by TensorRT, uses the KL entropy value to traverse the data distribution of each quantized layer, and determines threshold value by searching for the lowest KL entropy value.

As this method can cause more data saturation and smaller data quantization granularity, it more suitable than max for those neural network models with more concentrated data distribution.

4. Max

Max refers to a calibration method that automatically selects the max value in quantized layer as the threshold.

This method can cause oversized quantization granularity. However, it also causes fewer saturated points than the KL method, which makes it suitable for those neural network models with more discrete data distribution.

#[Reference] OP List

For more information about the operators and corresponding constraints currently supported by Horizon Algorithm Toolchain, please refer to Toolchain Operator Support Constraint List.