Whether you have performed PTQ Model Performance Evaluation or QAT Model Performance Evaluation, based on the performance analysis work mentioned earlier, you may find that the performance results are not as expected. This chapter introduces the suggestions and measures proposed by Horizon Robotics to improve model performance.

#Metric Interpretation of Performance Analysis Files

#Performance Analysis Tools

You can use the performance analysis tool hb_analyzer provided by us to analyze the performance of HBIR models at different stages as well as HBM models. For detailed usage instructions and explanations of the corresponding analysis files, please refer to The hb_analyzer Tool section.

#Development Machine-side Simulation Performance Evaluation

#Metric Interpretation of Simulation Performance Evaluation

In addition to the hb_analyzer tool, after completing the performance evaluation described earlier, we will obtain HTML and JSON files containing the estimated model performance metrics. The HTML file offers better readability, with the following key sections:

1. The Summary tab provides the estimated BPU performance metrics of the model:

   • FPS (1 core): Estimated FPS of the model running on a single core.

   • Latency: Estimated latency of the model execution.

   • BPU conv original OPs per run: Total computational load of all Conv operators in the model.

   • Compiling options: Model compilation parameters.

   • Compiling HBDK version: HBDK version used for model compilation.

   • Model minimum memory requirement: Static memory + Dynamic memory.

   • Static memory: Memory footprint of the HBM model file.

   • Dynamic memory: Model input memory + Model output memory + BPU shared temporary memory + Exclusive intermediate memory.

   • Model input memory: Memory footprint of the model input.

   • Model output memory: Memory footprint of the model output.

   • Bpu shared temporary memory: Memory that can be reused across models.

   • Exclusive intermediate memory: Memory that need to be applied for each frame.

2. The Temporal Statistics tab provides the bandwidth utilization during one frame of model inference.

3. The Layer Details tab provides information on the computational load, computation latency, and data transfer latency of each layer of BPU operators, as well as the active time window of each layer after compilation (this does not represent the execution time of the layer, as multiple layers usually execute alternately or in parallel).

4. The Timeline tab provides the estimated latency of the instruction set and information of the corresponding computing units during one frame of model inference (the actual displayed metrics are related to the configuration):

   • TAE: Tensor Acceleration Engine, which is the engine module responsible for accelerating tensor computations in the BPU. It is mainly used for various conv computations and also supports some matrix computations.

   • VAE: Vector Acceleration Engine, which is the engine module responsible for accelerating vector computations in the BPU. It is mainly used for various element-wise operations in neural networks, such as A+B, A*B, and LUT computations.

   • AAE: Auxiliary Acceleration Engine, which is the engine module responsible for auxiliary computation acceleration in the BPU. It is mainly used for auxiliary acceleration beyond tensors, vectors, and scalars, such as Pooling, Resize, and Warp function computations.

   • VPU: Vector Processing Unit, which is the unit responsible for vector computations inside the BPU. It is more flexible than the VAE but has lower computing power.

   • SPU: Scalar Processing Unit, which is the unit responsible for scalar computations inside the BPU.

   • TRANS: The computing unit in the BPU that handles data layout transformation.

   • STORE: Represents writing data from internal cache/registers to memory (or outside the computing platform).

   • LOAD: Represents loading data from memory (possibly outside the computing platform) into the computing platform cache.

#Bandwidth Analysis

By observing the curve in the Temporal Statistics tab of the HTML file and checking the computing bandwidth of each type of operator parsed from the JSON file, you can identify which structural segments or operators of the model are bandwidth-bound. Where the computing curve is relatively low but the load and store metrics are high, it indicates that the computing capability is limited due to bandwidth constraints.

In the Layer Details tab, you can also find the operators where the load/store overhead is much higher than the computing overhead.

#Utilization Rate Analysis of Timeline

Continuing to observe the curves in the Temporal Statistics tab of the HTML file, it can be seen that the computing curve only accounts for the computations performed by the TAE unit. Therefore, a drop in the computing curve may also be caused by the excessively long execution time of other computing units that cannot run in parallel with the TAE.

At this point, we can analyze whether a specific computing unit has an excessively long execution time and low parallelism with the TAE by observing the timeline chart in the Timeline tab.

#Others

If it is already difficult to identify obvious bottlenecks from the Temporal Statistics and Timeline, it is then necessary to directly visualize the quantized.bc file and identify optimizable structures based on experience.

#On-board Side Actual Measurement Metrics of Model Performance Data

For the metric interpretation of the on-board actual measurement data of model performance, you can refer to the introduction in Actual Measurement of Model Performance Data on the Board Side section.

#Performance Tuning Recommendations

#Check the YAML parameters that affect model performance

Some parameters in the YAML configuration file for model conversion will actually affect the final performance of the model. You can first check whether they have been configured correctly as expected. For the specific meaning and function of each parameter, please refer to Specific Parameter Information section.

• debug_mode: This parameter is used for precision debugging and analysis. If dump_all_layers_output is configured, a dequantization output node will be added for each convolution and matrix multiplication operator to output the intermediate results of the model, which will significantly reduce the performance of the model after deployment on the board. Therefore, during performance evaluation, it is imperative to remove the dump_all_layers_output parameter from debug_mode.

• compile_mode: This parameter is used to select the optimization direction (bandwidth or latency) for model compilation. When focusing on performance, configure it as latency.

• optimize_level: This parameter is used to select the optimization level of the compiler. O0 performs no optimization, offering the fastest compilation speed but the lowest optimization level. From O1 to O2, as the optimization level increases, the compiled model is expected to run faster, but the compilation time required will also be longer.

• max_time_per_fc: This parameter is used to control the execution duration of the function-call of the compiled model's data instructions, thereby implementing the model priority preemption function. Changing the function-call execution duration of the preempted model by setting this parameter will affect the on-board performance of the model.

#Process CPU operators

To determine whether there are CPU operators in the model, you can refer to the section Whether There Are CPU Operators in the Model. If the evaluation confirms that the prominent performance bottleneck is caused by the current operators running on the CPU, in this case, we recommend that you first check the section Toolchain Operator Support Contraint List to confirm whether the operators currently running on the CPU are supported by the BPU.

For the PTQ pipeline, if the parameters of the operators used exceed the constraints supported by the BPU, it is recommended to first adjust the calculation parameters of the original floating-point model to be within the BPU constraints. To help you quickly identify the specific parameters that exceed the constraints, we recommend that you perform another check using the method described in the section Model Checking. The tool will directly give prompts about the parameters that exceed the BPU-supported range.

If the operator does not have BPU support capability, you need to perform corresponding optimization operations according to the following scenarios:

• CPU operators are located in the middle of the model

  In this case, it is recommended that you first try parameter adjustment, operator replacement, or model modification.

• CPU operators are located at the head or tail of the model

  In this case, if the nodes are connected to the model input or output.

  • If you perform model quantization and compilation through the PTQ pipeline using the hb_compile tool, you can add the remove_node_type parameter to the model_parameters configuration group (model parameter group) in the YAML file, then recompile the model. For detailed usage, please refer to the section Model Parameters.

  • If you perform model quantization and compilation through the PTQ pipeline API or the QAT pipeline, you can delete the operators by calling the remove_io_op interface, then recompile the model. For detailed usage, please refer to the section Operator Deletion.

Since J6P supports extensive floating-point calculations and mutual switching between different computing precisions, not all fp32 operators will fall back to the CPU. Except for the operators explicitly unsupported in the operator constraints, common CPU-bound operators mainly include the following categories:

• ScatterND

  Currently, the BPU toolchain supports the ScatterND operator with fp16 and fp32 inputs on the J6P, J6H, and J6B platforms. Except for the above cases, this operator will severely affect model latency when running on the CPU. For methods to remove the ScatterND operator, you can refer to How to Remove the scatterND Operator in the ONNX Model and Generation and Removal of ScatterND in ONNX.

• int64 index

  Operators that output int64 type in the model are typically topk and argmax. For methods to avoid int64 cast in the outputs of topk and argmax, you can refer to 【Horizon J6 Toolchain Advanced Tutorial】Collection of Operator Optimization Solutions. Additionally, if there are no strict latency requirements, you do not need to modify the source code. After exporting qat.bc, you can use the following code to replace int64 data in the model with int32, which also avoids CPU cast introduced by int64 calculations:

  model = load("qat.bc")
  model.replace_index_tensor_type()
  quantized_model = convert(model, march="nash-p")

• High-precision operators (such as resize and warp) that are not supported by hardware

  If unexpected CPU-based resize and warp with int16/fp16 inputs are found in the model, it is most likely that the enable_optimize switch was not turned on when using the new version of Qconfig configuration, or there was a misconfiguration of other Qconfig settings that did not meet expectations.

  For other operators, you can judge the root cause by referring to the operator constraint information in the section Toolchain Operator Support Contraint List, as well as the reasons for operators falling back to the CPU prompted by enabling the advice parameter during conversion.

#Building a Quantization-Friendly Floating-Point Model

In actual production, we find that some models are inherently unsuitable for quantization, resulting in the investment of substantial resources and time in resolving quantization accuracy issues during model deployment. You can refer to the section Build Quantization Friendly Model to learn how to make appropriate adjustments to the floating-point model to improve its quantization compatibility.

In addition, for J6 platform, we provide the following optimization recommendations:

1. Perform zero-symmetric normalization on inputs. For radar point clouds and certain inputs with clear physical meanings, the original values are usually large. It is recommended to normalize such inputs before feeding them into the model, which can effectively reduce the difficulty of quantization to a certain extent.

2. Add batch normalization (bn) after every conv-like operator as much as possible to avoid numerical value amplification.

3. Replace some ReLU layers with ReLU6 to further control the numerical range.

4. Avoid using excessively large values such as 1e10 and 65536 for masks. Generally, a value of 100 (sufficient to achieve differentiation without affecting floating-point accuracy) or 30000 (to ensure the value after softmax is 1) is recommended. The value should not exceed the range of int16/fp16; otherwise, issues may arise during model export or QAT.

#Improving Parallelism and Utilization of Various Components in BPU

1. Common causes of bandwidth bottlenecks include operator-related factors and model structure issues:

   • Operator-related factors:

     • For example, the warp operator has poor data locality (the accessed input data may not be contiguous in memory), making it difficult to perform tiling optimization.

     • Excessive use of operators with high layout requirements such as concat, pad, and slice can easily lead to data volume expansion.

   • Model structure issues:

     • A "wide" model structure: requiring a large number of simultaneously active tensors that cannot fit into the L1 memory (L1M).

     • An overly "deep" model structure: such as shortcuts spanning too many layers, resulting in data being unable to reside in L1M for an extended period.

2. Optimization recommendations for the above issues are as follows:

   • For the warp operator, if there are multiple parallel structures with the same size, it is recommended to merge them along the batch dimension. This is because the warp operator can only be equivalently split along the batch dimension, but not along the height (h) or width (w) dimensions. Merging along the batch dimension can improve utilization. In addition, reducing the size of feature channels while satisfying 32-channel alignment can also significantly reduce the latency of the warp operator.

   • For concat, pad, and slice operators, it is recommended to minimize or avoid their use as much as possible. For example, leverage the broadcasting feature of elementwise operations to reduce manual use of expand for tensor shape alignment.

   • Significant bottlenecks in VPU/VAE:

     • softmax: Replace self-attention with sima. Reference: Relevant Paper.

     • layernorm: Replace layernorm with Dyt. Reference: Relevant Paper.

     • Avoid using fp32 precision as much as possible. Fp32 can only run on the VPU, whose computing power is several orders of magnitude lower than that of VAE and TAE. Moreover, fp32 doubles the bandwidth requirement compared to fp16/int16.

#BPU-Oriented Optimization for High-Efficiency Models

The academic community continuously optimizes the computational efficiency (the lower the theoretical computation required for the same algorithm accuracy, the higher the efficiency) and parameter efficiency (the fewer the parameters used for the same algorithm accuracy, the higher the efficiency) of algorithm models. Representative works in this field include EfficientNet and ResNeXt, which adopt Depthwise Convolution and Group Convolution respectively. However, such high-efficiency models have low compatibility with GPU/TPU and fail to fully leverage their algorithmic advantages. As a result, the academic community has been compelled to optimize EfficientNet V2/X and NFNet specifically for GPU/TPU. These optimizations mainly involve reducing the use of Depthwise Convolution and significantly increasing the group size in Group Convolution, which compromise the original computational and parameter efficiency of the models.

For more guidelines on high-efficiency model design, please refer to the section Efficient Model Design Advice. Meanwhile, we provide a rich set of models for your direct reference. The source code of reference algorithm models is available in the path samples/ai_toolchain/horizon_model_train_sample of the OpenExplorer package. For instructions on using the algorithm package, please refer to the section Reference Algorithm.

For other optimization recommendations, you can refer to the relevant content in 【Horizon J6 Toolchain Advanced Tutorial】Collection of Operator Optimization Solutions.