This tutorial mainly tells everyone how to train a BevSparseMultiTask model from scratch on the dataset nuscenes using HAT, including floating-point, quantized, and fixed-point models.
This model is a bev multi-task perception model, which contains detection heads for three major perception tasks: sparse dynamic detection head, sparse static element detection head, and general obstacle detection head.
If you just want to simply train the BevSparseMultiTask model, you can first read the content of this chapter.
Similar to other tasks, for all training and evaluation tasks, HAT uniformly adopts the tools + config form to complete.
After preparing the original dataset, you can conveniently complete the entire training process through the following process.
For this example, using the nuScenes dataset, you can download the dataset from https://www.nuscenes.org/nuscenes.
For the Occupancy prediction task, you will also need to download the OCC ground truth data,
which can be obtained from https://github.com/CVPR2023-3D-Occupancy-Prediction/CVPR2023-3D-Occupancy-Prediction.
It is recommended to unzip the downloaded dataset into the occ3d/gts directory within the nuScenes dataset folder.
Additionally, to improve training speed, we have packaged the original JPEG format dataset into an LMDB format dataset.
The previous command will generate two files, nuscenes_infos_train.pkl and nuscenes_infos_val.pkl,
in the "./tmp_data/nuscenes/occ3d directory".
Next, execute the following command to perform the packaging:
The above two commands correspond to converting the training dataset and the validation dataset, respectively. After the packaging is complete, the file structure in the data directory should be as follows:
train_lmdb and val_lmdb are the training dataset and validation dataset after packaging, which are also the final dataset read by the network. The meta contains initialization information needed by the evaluation script, with specific information copied from the original nuscenes dataset.
After the dataset is prepared, you can start training the floating-point BevSparseMultiTask network. Before starting network training, you can use the following command to test the computation amount and parameter count of the network:
Due to the clever registration mechanism used by the HAT algorithm package, each training task can be started in the form of train.py plus a config configuration file.
train.py is a unified training script, task-independent. What kind of task we want to train, what dataset to use, and training-related hyperparameter settings are all contained in the specified config configuration file.
The config file provides key dicts for model construction, data reading, etc.
After the model training is completed, we can also verify the performance of the trained model. Since we provide a two-stage training process of float, calibration, qat, accordingly we can verify the model performance trained in these three stages. Just run the following three commands respectively:
The accuracy displayed here is the actual accuracy of the final quantized model. Of course, this accuracy should be very close to the accuracy in the qat validation phase.
If you hope to see the visualization effect of the trained model, the tools folder below also provides a visualization script. You just need to run the following script:
The quantization training toolchain integrated in HAT is mainly prepared for Horizon's computing platform. Therefore, the inspection and compilation of quantized models are necessary.
We provide a model inspection interface in HAT, which can check whether it can run normally on the BPU after defining the quantized model:
After training is completed, the compile_perf_hbir tool can be used to compile the quantized model into an hbm file that can run on the board. At the same time, this tool can also estimate the running performance on the BPU. The following script can be used:
Among them, the type under model represents the defined model name, and the remaining variables represent other components of the model. The advantage of defining a model in this way is that we can easily replace the structure we want.
After the training script starts, it will call the build_model interface to turn such a dict-type model into a model of type torch.nn.Module.
Similar to the definition of model, the data augmentation process is implemented by defining data_loader and val_data_loader these two dicts in the config configuration file, corresponding to the processing procedures of training set and validation set respectively.
The Nuscenes dataset simultaneously contains annotations for dynamic, static, and general obstacles. We can train three tasks on one dataset.
In the specific code implementation, NuscenesBevDataset can provide dynamic and occ annotations, and NuscenesSparseMapDataset inherits from it and can provide dynamic, static, and occ annotations simultaneously.
The dynamic detection head uses temporal information, so data_loader adopts DistStreamBatchSampler to ensure that data is returned in sequence.
To train a model with high accuracy, a good training strategy is indispensable. For each training task, the corresponding training strategy is also defined in the config file, which can be seen from the variable float_trainer.
float_trainer defines our training method on a large scale, including using multi-GPU distributed training (distributed_data_parallel_trainer), the number of epochs for model training, and the choice of optimizer.
At the same time, callbacks reflects the small strategies used in the training process and operations you want to implement, including exponential moving average training (ExponentialMovingAverage), learning rate transformation method (CosineAnnealingLrUpdater),
Validating model metrics during the training process (Validation), and saving (Checkpoint) model operations. Of course, if you have operations you hope the model to implement during the training process, you can also add them in the form of this dict.
float_trainer is responsible for linking the entire training logic together.
Note
If you need to reproduce the accuracy, it is best not to modify the training strategy in the config. Otherwise, unexpected training situations may occur.
Through the above introduction, you should have a relatively clear understanding of the functions of the config file. Then, through the training script mentioned earlier, you can train a high-accuracy pure floating-point detection model.
Of course, training a good detection model is not our ultimate goal; it just serves as a pretrain to serve our later training of fixed-point models.
After we have a pure floating-point model, we can start quantized model training. Similar to floating-point training, we can use the following command to calibrate the model and obtain a pseudo-quantized model.
It can be seen that our configuration file has not changed, only the stage type has changed. At this time, the training strategy we use comes from calibration_trainer in the config file.
When we train the quantized model, we need to set model_convert_pipeline. It first loads the model's floating-point weights and then uses Float2Calibration to convert the model from a floating-point model to a calibration model.
Among them, calibration_qconfig_setter defines the quantization configuration of the model. The current configuration is:
sensitivity_templates = []table_dir = os.path.join(ckpt_dir, "quant_analysis")sensitive_setting_dict = { "output_det.classification_L1_sensitive_ops.pt": 20, "output_det.prediction_L1_sensitive_ops.pt": 20, "output_det.quality_L1_sensitive_ops.pt": 10, "output_om.all_pts_preds_L1_sensitive_ops.pt": 5, "output_om.all_bbox_preds_L1_sensitive_ops.pt": 5, "output_om.all_cls_scores_L1_sensitive_ops.pt": 5,}for k, v in sensitive_setting_dict.items(): sensitive_table = load_ckpt(os.path.join(table_dir, k)) sensitivity_templates.append( SensitivityTemplate( sensitive_table, topk_or_ratio=v, ) )backbone = ["backbone"]int16_max_point_muls = [ f"head.det.layers.{layer}.point_mul" for layer in range(3, 39, 7)] + [f"head.om.decoder.layers.{layer}.point_mul" for layer in range(2, 33, 6)]int16_max_reciprocal_op = [ f"head.det.layers.{layer}.reciprocal_op" for layer in range(3, 39, 7)]int16_add_ops = [f"head.det.layers.{layer}.add2" for layer in range(6, 35, 7)]# int16_add_ops += [f"head.om.layers.{layer}._generated_add_1" for layer in range(5, 35, 6)]float32_ops = [ f"head.det.layers.{layer}.cls_layers" for layer in range(6, 35, 7) # 6, 13, 20, 27, 34] + [f"head.det.layers.{layer}.quality_layers" for layer in range(6, 35, 7)]q_template = [ ModuleNameTemplate({"": qint8}), MatmulDtypeTemplate( input_dtypes=[qint8, qint8], # prefix=[""], ), ConvDtypeTemplate( input_dtype=qint8, weight_dtype=qint8, # prefix=[""], ), ModuleNameTemplate( { m: {"dtype": qint8, "threshold": 1.0} for m in ["backbone.quant"] }, ), ModuleNameTemplate( { m: {"dtype": qint16, "threshold": 1.1} for m in int16_max_point_muls }, ), ModuleNameTemplate( { m: {"dtype": qint16, "threshold": 10} for m in int16_max_reciprocal_op }, ), ModuleNameTemplate( { m: {"dtype": qint16, "threshold": 50} for m in ["head.det.fc_after"] }, ), ModuleNameTemplate( { m: {"dtype": qint16, "threshold": 10} for m in ["head.det.fc_before"] }, ), ModuleNameTemplate( { m: {"input": [qint16, qint16], "output": qint16} for m in int16_add_ops }, ), *sensitivity_templates, ModuleNameTemplate( {m: None for m in float32_ops}, # quant int8, fixed scale method 1 freeze=True, ),]calibration_qconfig_setter = QconfigSetter( reference_qconfig=get_qconfig( # 1. Mainly used to obtain observer # observer=(observer_v2.MSEObserver) observer=(observer_v2.HistogramObserver) ), templates=q_template, enable_optimize=True, save_dir=os.path.join( ckpt_dir, "./qconfig_setting" ), custom_qconfig_mapping=None,)
The quantization configuration here is slightly complex, and we will explain it in detail step by step.
First, the first parameter of calibration_qconfig_setterreference_qconfig defines a basic quantization configuration, using the observer observer_v2.HistogramObserver.
This is a calibration tool based on histogram statistics. It globally statistically analyzes the histogram statistics of the input and output features of each layer of the model, and then determines the quantization step size scale for each layer.
The second parameter of calibration_qconfig_setter is templates corresponding to q_template, which uses many templates to define the quantization configuration from top to bottom.
At the top is ModuleNameTemplate({"": qint8}), defining the global quantization configuration as qint8,
then the two lines MatmulDtypeTemplate and ConvDtypeTemplate define the input types for matmul operators and conv operators as qint8 respectively.
Then, many lines use ModuleNameTemplate to specially set the quantization type and fixed scale for many ops in the model, which is a configuration based on observations of the model structure and statistics.
Users can also similarly modify the quantization configuration of each layer of the model.
Note that there is a line *sensitivity_templates in q_template, which is a high-precision template configured based on sensitivity analysis results.
Although based on experience, we have already manually configured some high-precision operators, it is still difficult for quantized models to recover the accuracy of the floating-point model. To analyze the problems of model quantization, we suggest users perform sensitivity analysis on the model.
To run the model sensitivity analysis tool, you can run the following command:
The main principle of sensitivity analysis is: given data, the tool runs the floating-point model and quantized model separately, finds the samples with the largest output difference, known as bad case.
Then for the quantized model, the tool traverses each operator in the model, performs single operator quantization, sorts based on the error comparison of the floating-point model output after quantization, obtaining a list of model quantization-sensitive operators.
sensitivity_templates precisely reads this sensitivity ranking list, configuring the operators ranked at the top of sensitivity to high precision, thereby improving the quantization accuracy of the model.
After calibration, the quantization accuracy of this model has reached over 99% of floating-point training. We can achieve the accuracy target without performing additional quantization-aware training.
