Horizon J6 algorithm toolchain (hereinafter referred to as the toolchain) is a complete set of artificial intelligence edge algorithm solution, It is compatible with Horizon J6B, J6E, J6M, J6H, and J6P series edge computing platforms, and helps you quantify floating-point models into fixed-point models for rapid deployment of custom algorithm models on Horizon computing platforms.

For details on the supported ONNX/Torch operators, operator constraints, and general usage limitations, please refer to the Toolchain Supported Operator List section.

Currently, most of the models trained on GPUs are floating-point models, that is, the parameters are mainly stored in the float32 data type. The Horizon BPU computing platform uses lower-bit computations (e.g., int8, int16, fp16) as appropriate to reach a higher energy-efficiency performance for deployment.

The process of compressing a float32 model into a lower-bit computational model, while ensuring the correctness of the model's inference, we call it quantization. In addition, model quantization can effectively reduce the model size, accelerate the speed of deep-learning inference, therefore, it is also widely studied and applied in academia and industry.

Depending on whether to adjust the parameters after quantization, we can classify the quantization methods into post-training quantization (PTQ) and quantized awareness training (QAT).

The difference in operation between these two methods is shown in the following diagram (Left: PTQ; Right: QAT).

The PTQ uses a batch of calibration data to calibrate the trained models, which converts the trained FP32 model directly into a fixed-point computational model without any training of the original model. As the quantization process is simple and fast, requiring a few adjustments to the hyperparameters and no training, this method has been widely used in a large number of end-side and cloud-side deployment scenarios. We recommend that you to try the PTQ method first to see if it meets your requirements on the deployment accuracy and performance.

The QAT is to quantize the trained model before training it again. Since the fixed-point values cannot be used for backward gradient calculation, the actual procedure is to insert fake quantization nodes in front of some operators to obtain the truncated values of the data flowing through the op during the training, so that they can be easily used when quantizing the nodes during the deployment of the quantization models. We need to continuously optimize the accuracy during training to obtain the best quantization parameters. Since the model training is involved, it requires the developers to have higher levels of technical skills.

The schematic diagram of the toolchain composition is as follows:

The Runtime SDK provides runtime libraries that support key functionalities such as model deployment and evaluation. You can use the interfaces within the SDK to streamline the deployment of models onto Horizon Robotics' computing platforms. For the development of embedded applications, please refer to the Inference Application Development Guide chapter. In addition to on-board deployment, the Runtime SDK also offers an x86 simulation feature, supporting inference verification in scenarios where on-board deployment is not feasible. You can refer to the provided Sample to familiarize yourself with the deployment process for Horizon Robotics' platforms, quickly grasp the relevant methods, and master the techniques of model deployment.

For more information about embedded application development, please read Embedded Application Development.

In addition, the toolchain provides rich development tools, samples, and model releases with a large number of built-in algorithmic models to help you get started and to improve your development efficiency.