Lidar Fusion Sparse-bev Perception Model
This tutorial mainly guides readers on how to use HAT to train a Lidar fusion perception model on the autonomous driving dataset Nuscenes,
covering floating-point, quantized, and fixed-point models.
Below, the configuration and training of a Lidar fusion sparse-bev perception model are introduced
using configs/lidar_bevfusion/bev_sparse_lidar_fusion_henet_tinym_nuscenes.py as an example.
bev_sparse_lidar_fusion_henet_tinym_nuscenes is a multi-modal autonomous driving perception model that takes camera and lidar inputs and outputs 3D detection boxes for dynamic elements.
Training Process
If you simply want to train the bev_sparse_lidar_fusion_henet_tinym_nuscenes model, start by reading this section. Like other tasks, all training and evaluation tasks in HAT are completed using a tools + config format. Once the original dataset is prepared, you can conveniently complete the entire training process by following the steps below.
Dataset Preparation
Take the nuscenes dataset as an example, which can be downloaded from https://www.nuscenes.org/nuscenes. For the Occupancy prediction task, you also need to download the OCC GT, which can be downloaded from https://github.com/CVPR2023-3D-Occupancy-Prediction/CVPR2023-3D-Occupancy-Prediction. It is recommended to decompress the downloaded dataset into the occ3d/gts folder within the nuscenes dataset folder. Then you can run the following commands to package lidar, images, occ gt, and other data into lmdb format:
The above two commands correspond to the conversion of training and validation datasets, respectively. After packing is complete, the file structure under the data directory should be as follows:
train_lmdb and val_lmdb are the packed training and validation datasets, which are also the datasets that the network ultimately reads.
Model Training
After the dataset is ready, you can start training the floating-point Lidar fusion multi-task perception model bev_sparse_lidar_fusion_henet_tinym_nuscenes .
First, you can use the following command to evaluate the model's computational load:
If you want to reproduce the entire model floating-point and fixed-point training process, you need to:
The above commands complete the pre-training of the camera input floating-point model, floating-point model training, and fixed-point model training, respectively. Fixed-point model training requires a well-trained floating-point model as a basis. For specific content, please read the Quantization-Aware Training section.
Exporting Fixed-Point Models
After completing the quantization training, you can start exporting fixed-point models. You can export them through the following command:
Model Verification
After the training is completed, you can obtain the trained floating-point, quantized, or fixed-point models.
Similar to the training method, we can use the same method to verify the trained models to obtain metrics for float , calibration and qat , which are the metrics for floating-point and quantized models, respectively.
Similar to training models, when the --stage parameter is set to "float" , calibration or "qat" , it can complete the verification of the trained floating-point models and quantized models, respectively.
Fixed-point model accuracy verification can also be done using the following command, but it is important to note that hbir must be exported first:
Model Inference
HAT provides the infer_hbir.py script to visualize the inference results of fixed-point models.
The following command will save data from the dataset in the ./demo folder and visualize it in the ${save_path} folder.
Simulation Board Accuracy Verification
In addition to the above model verification, we also provide a accuracy verification method that is completely consistent with the board. It can be completed in the following way:
Fixed-Point Model Checking and Compilation
The quantization training toolchain integrated in HAT is mainly prepared for the Horizon computing platform, so it is necessary to check and compile the quantized models. We provide a model checking interface in HAT, which can be used to check whether the defined quantization model can run normally on the BPU before training:
After the model training is completed, the compile_perf_hbir script can be used to compile the quantized model into an hbm file that can be run on the board,
and this tool can also estimate the running performance on the BPU:
The above is the entire process from data preparation to generating a quantizable deployable model.
Training Details
In this section, we explain some of the precautions to be taken when training the model, mainly related to some settings in the config.
Model Construction
The model definition related config is as follows:
Among them, lidar_network defines the encoder for processing lidar input, and model provides the complete model definition.
The type under model indicates the name of the defined model, and the remaining variables represent other parts of the model.
The advantage of defining the model in this way is that we can easily replace the structures we want.
Data Augmentation
Like the definition of model, the data augmentation process is implemented by defining data_loader and val_data_loader
in the config configuration file, corresponding to the processing of the training set and validation set, respectively.
For example, the definition of the training set is:
Training Strategy
To train a high accuracy model, a good training strategy is essential.
For each training task, the corresponding training strategy is also defined in the config file, which can be seen from the float_trainer variable.
The configuration of float_trainer for the lidar fusion sparse bev model is as follows, we load the single-camera input sparse bev model as a pre-training.
float_trainer defines our training method from a broad perspective, including the use of multi-card distributed training (distributed_data_parallel_trainer),
the number of epochs for model training, and the choice of optimizer.
At the same time, callbacks reflects the small strategies and operations you want to implement during model training,
including the way the learning rate changes (CosineAnnealingLrUpdater), validating model metrics during training (Validation), and saving (Checkpoint) model operations.
Of course, if you have operations you hope the model will implement during training, you can also add them in this dict manner.
Through the above introduction, you should have a clear understanding of the functions of the config file. Then, through the aforementioned training scripts, you can train a high accuracy pure floating-point detection model. Of course, training a good detection model is not our ultimate goal, it is just a pretrain for us to train fixed-point models later.
Quantization Model Training
When we have a pure floating-point model, we can start training the corresponding fixed-point model. Like the floating-point training method, we can get a pseudo-quantized model by running the following script:
It can be seen that our configuration file has not changed, only the type of stage has changed. The calibration process can provide better initial parameters for QAT quantization training.
We use the template to configure Float2Calibration and Float2QAT to convert the model into calibration and qat models, respectively. The specific calibration and QAT config are:
Different Training Strategies
As we said before, quantization training is actually a fine-tune on the basis of pure floating-point training.
Therefore, during quantization training, our initial learning rate is much smaller compared to float, and the number of training epochs is greatly reduced.
Most importantly, when defining model, our pretrained needs to be set to the address of the well-trained pure floating-point model.
After making these simple adjustments, you can start training our quantization model.
Regarding the key steps in quantization training, such as preparing floating-point models, operator replacement, inserting quantization and dequantization nodes, setting quantization parameters, and operator fusion, please read the Quantization-Aware Training section.