This section introduces the ONNX model accuracy tuning pipeline using the precision problems encountered in actual use as an example. Please read the Model Accuracy Tuning chapter firstly to Understand relevant theoretical knowledge and tool usage.
Typical accuracy issues include:
The all node type int16 quantization accuracy meets the requirements and Accuracy Debug Tool can provide relatively accurate sorting of sensitive nodes;
The all node type int16 quantization accuracy meets the requirements, but setting a large number of sensitive nodes to higher precision cannot effectively improve quantization accuracy;
The all node type int16 quantization accuracy does not meet the standard, under the premise of full BPU quantization of the model, we hope to further improve the quantization accuracy.
Accuracy Debug Tool provides an interface for calculating node quantization sensitivity. It can calculate the impact of each operator's quantization on the output results, set nodes with high quantization loss to higher precision, and complete accuracy tuning. The tuning pipeline is described using the HybridNets model as an example.
Using HMCT default INT8 quantization, percentile is selected as the calibration algorithm, the calibration accuracy does not meet the requirements (the accuracy of det and ll_seg decreases by more than 1%):
First, set all_node_type to INT16 and select percentile for the calibration algorithm. At this point, the calibration accuracy meets the requirements, and we can use INT8+INT16 mixed precision to complete the tuning:
Compile the INT8 calibration model based on the percentile calibration algorithm selected for INT16 model, configure debug_mode: "dump_calibration_data" in the yaml file to save the calibration data, and output the node quantization sensitivity through get_sensitivity_of_nodes:
Sorting by cosine similarity from front to back, gradually set the operator INT16 quantization, and the calibration model accuracy will increase until it meets the requirements:
Serial Number
Cosine Similarity Value(<=value will be set to INT16)
Accuracy
det
da_seg
ll_seg
1
None
0.75562(97.85%)
0.89675(99.12%)
0.81813(95.83%)
2
0.999
0.76531(99.11%)
0.90274(99.79%)
0.83874(98.24%)
3
0.9998
0.76545(99.12%)
0.90340(99.86%)
0.83961(98.34%)
4
0.9999
0.76613(99.21%)
0.90420(99.95%)
0.84216(98.64%)
5
0.99992
0.76712(99.34%)
0.90356(99.88%)
0.84397(98.85%)
6
0.99993
0.76781(99.43%)
0.90374(99.90%)
0.84484(98.95%)
7
0.99994
0.76811(99.47%)
0.90344(99.86%)
0.84528(99.01%)
From the above test table, we can see that if the sensitive nodes with a sensitivity value less than or equal to 0.99994 are set as INT16 nodes, the calibration accuracy meets the requirements:
If using the Accuracy Debug Tool to set sensitive nodes to higher precision fails to effectively improve model accuracy, we can first try specifying output nodes to filter out irrelevant nodes. Additionally, observe the model output error and select other metric to improve the correlation between sensitivity ranking and precision. Furthermore, by analyzing the model structure, typical substructures with a higher risk of quantization loss (such as model outputs, inputs, and structures with specific physical meaning) can be set to higher precision to complete accuracy tuning. This tuning pipeline is described using the YoloP model as an example.
Using HMCT default INT8 quantization, percentile is selected as the calibration algorithm, the calibration accuracy does not meet the requirements (the accuracy of det decreases by more than 1%):
First, set all_node_type to INT16 and select percentile for the calibration algorithm. At this point, the calibration accuracy meets the requirements, and we can use INT8+INT16 mixed precision to complete the tuning:
Compile the INT8 calibration model based on the percentile calibration algorithm selected for INT16 model, configure debug_mode: "dump_calibration_data" in the yaml file to save the calibration data, and output the node quantization sensitivity through get_sensitivity_of_nodes:
Sorting by cosine similarity from front to back, gradually set the operator INT16 quantization. However, even with a large number of sensitive nodes set to INT16, the accuracy still failed to meet the requirements:
Serial Number
Cosine Similarity Value(<=value will be set to INT16)
Accuracy
det
da_seg
ll_seg
1
None
0.61507(80.46%)
0.88863(99.84%)
0.65357(100.19%)
2
0.9999
0.60956(79.74%)
0.88911(99.89%)
0.65925(101.07%)
3
0.99996
0.60978(79.76%)
0.88933(99.92%)
0.66112(101.35%)
4
0.99998
0.60956(79.73%)
0.88931(99.91%)
0.66125(101.37%)
5
0.99999
0.66426(86.89%)
0.88958(99.94%)
0.66065(101.28%)
Observing the accuracy results of the INT8 calibrated model, only det branch does not reach 99%. When calculating nodes sensitivity through get_sensitivity_of_nodes interface, we can use the -o option to specify det output, so that a better sensitivity ranking can be obtained:
Sorting by cosine similarity from front to back, setting the operator to INT16 quantization, and focusing only on the det output can filter out useless nodes, but the final accuracy still does not meet the requirements:
Serial Number
Cosine Similarity Value(<=value will be set to INT16)
Accuracy
det
da_seg
ll_seg
1
None
0.61507(80.46%)
0.88863(99.84%)
0.65357(100.19%)
2
0.9999
0.60868(79.62%)
0.88836(99.81%)
0.65300(100.11%)
3
0.99997
0.60961(79.74%)
0.88902(99.88%)
0.65664(100.66%)
4
0.99999
0.66461(86.94%)
0.88932(99.91%)
0.65876(100.99%)
Observe the output similarity of the INT8 calibrated model. The L1 and L2 distances of the det branch deviate greatly from the floating point. Try replacing cosine similarity with other metrics:
When calculating nodes sensitivity through the get_sensitivity_of_nodes interface, we can specify mse as the metric to improve the discrimination between different nodes:
Sort by mse similarity from front to back, set the operator to higher precision, and finally achieve the required accuracy after adding a large number of INT16 nodes:
Even if we only focus on improving the det output based on mse metric, setting a large number of sensitive nodes for higher precision still cannot effectively improve the accuracy. Furthermore, considering that only the det task currently does not meet the requirements, it is inferred that quantization loss is less likely to occur in nodes in the da_seg and ll_seg branches, as well as in the shared backbone. Focus on the accuracy tuning of det branch, combined with model structure analysis, try to specify the det output position subgraph to use higher precision and test the calibration accuracy:
When the sorting of sensitive nodes is inaccurate, the source of the loss can be preliminarily determined by configuring the subgraph for higher precision. If the latency of the subgraph with higher precision increases significantly, sensitivity analysis can be performed within the subgraph, so that fewer nodes with higher precision will be configured.
The accuracy tuning pipeline requires constant modification of node configurations, model compilation and accuracy verification. The entire process is time-consuming and expensive to debug. Based on this, we provide the IR interface to support you to directly modify quantization parameters in calibrated_model.onnx for rapid verification.
from hmct.ir import load_model, save_modelfrom hmct.common import find_input_calibration, find_output_calibrationmodel = load_model("calibrated_model.onnx")# Modify specific activation or weight calibration nodes to use specific qtypenode = model.graph.node_mappings["ReduceMax_1317_HzCalibration"]print(node.qtype) # Support qtype readingnode.qtype = "float32" # Support int8, int16, float16, float32 configured# Configure all activation or weight calibration nodes to use int16 qtypecalibration_nodes = model.graph.type2nodes["HzCalibration"]# Configure all activation calibration nodes to use int16 qtypefor node in calibration_nodes: if node.tensor_type == "feature": node.qtype = "int16"# Configure all weight calibration nodes to use int16 qtypefor node in calibration_nodes: if node.tensor_type == "weight": node.qtype = "int16"# Configure all calibration nodes to use int16 qtypefor node in calibration_nodes: node.qtype = "int16"# Configure a node with int16 input qtypefor node in model.graph.nodes: if node.name in ["Conv_0"]: for i in range(len(node.inputs)): input_calib = find_input_calibration(node, i) # It is required to be able to find HzCalibration in the # input, and tensor_type is feature. if input_calib and input_calib.tensor_type == "feature": input_calib.qtype = "int16"# Configure a node with int16 output qtypefor node in model.graph.nodes: if node.name in ["Conv_0"]: output_calib = find_output_calibration(node) # It is required to be able to find HzCalibration in the # input, and tensor_type is feature. if output_calib and output_calib.tensor_type == "feature": input_calib.qtype = "int16"# Configure nodes with specific op_type to int16for node in model.graph.nodes: if node.op_type in ["Conv"]: for i in range(len(node.inputs)): input_calib = find_input_calibration(node, i) # It is required to be able to find HzCalibration in the # input, and tensor_type is feature. if input_calib and input_calib.tensor_type == "feature": input_calib.qtype = "int16"# Modify specific activation or weight calibration nodes with specific thresholdsnode = model.graph.node_mappings["ReduceMax_1317_HzCalibration"]print(node.thresholds) # Support thresholds readingnode.thresholds = [4.23] # Support np.array, List[float]save_model(model, "calibrated_model_modified.onnx")
