Lidar Fusion MultiTask Model Training
This tutorial primarily guides you on how to use HAT to train a lidar fusion multi-task perception model on the Nuscenes autonomous driving dataset,
including floating-point, quantized, and fixed-point models. The following example configs/lidar_bevfusion/bevfusion_pointpillar_henet_multisensor_multitask_nuscenes.py illustrates how to configure and train a lidar fusion multi-task perception model.
bevfusion_pointpillar_henet_multisensor_multitask_nuscenes is a multimodal, multi-task autonomous driving perception model. It takes inputs from two modalities, camera and lidar, and outputs dynamic object 3D bounding boxes and 3D occupancy grids.
Training Process
If you want to quickly train the bevfusion_pointpillar_henet_multisensor_multitask_nuscenes model, you can start by reading this section.
Like other tasks, for all training and evaluation tasks, HAT uniformly uses the tools + config approach to complete them. After preparing the original dataset, you can easily follow the steps below to complete the entire training process.
Dataset Preparation
Here, using the NuScenes dataset as an example, you can download the dataset from https://www.nuscenes.org/nuscenes. For the occupancy prediction task, you'll also need to download the OCC ground truth (GT) from https://github.com/CVPR2023-3D-Occupancy-Prediction/CVPR2023-3D-Occupancy-Prediction.
It's recommended to unzip the downloaded datasets into the occ3d/gts folder within the NuScenes dataset directory.
Then, you can run the following commands to package lidar, images, and OCC GT data into LMDB format:
These two commands correspond to converting the training dataset and validation dataset, respectively. After packaging, the file structure in the data directory should look like this:
train_lmdb and val_lmdb are the packaged training and validation datasets, which are also the final datasets read by the network.
Model Training
Once the dataset is ready, you can start training the floating-point lidarMultiTask network. Before the network training starts, you can test the computation and the number of parameters of the network by using the following command.
If you simply want to start such a training task, just run the following command:
The commands above separately train the pretrained float-point model, floating-point model and the fixed-point (quantized) model. The training of the fixed-point model needs to be based on the previously trained floating-point model. For more details, please refer to the Quantized Awareness Training section.
Exporting the Quantized Model
After completing quantization training, you can start exporting the quantized model. You can use the following command to export it:
Model Validation
After training is complete, you will obtain floating-point, quantized, or fixed-point models.
Similar to the training process, you can validate the trained models using the same method to obtain metrics for float , calibration and qat , corresponding to the floating-point and quantized metrics, respectively.
Similar to when training the model, when the parameter after --stage is "float" , "calibration" or "qat" ,
you can validate the trained floating-point and quantized models, respectively.
You can also validate the accuracy of the fixed-point model using the command below, but it’s important to note that you must first export the HBIR:
Model Inference and Results Visualization
HAT provides the infer_hbir.py script to visualize the inference results of the quantized model:
Fixed-point Model Checking and Compilation
The quantization training toolchain integrated in HAT is mainly prepared for the Horizon computing platform, so it is necessary to check and compile the quantized models. We provide a model checking interface in HAT, which can be used to check whether the defined quantization model can run normally on the BPU before training:
After model training is complete, the compile_perf_hbir script can compile the quantized model into an hbm file suitable for board deployment.
This tool also estimates the model's performance on the BPU:
This outlines the entire process from data preparation to generating a deployable quantized model.
Training Details
In this note, we explain some things that need to be considered for model training, mainly including settings related to config.
Model Construction
In this context, the type under model indicates the name of the defined model, while the remaining variables represent other components of the model. The advantage of defining the model this way is that it allows us to easily swap out structures as needed.
The bevfusion_pointpillar_henet_multisensor_multitask_nuscenes model mainly consists of three parts:
lidar network, camera_network, and bev_decoders.
The bev_decoders contain two modules: BEVFormerDetDecoder and BevformerOccDetDecoder, which output 3D detection boxes and 3D occupancy grid predictions, respectively.
Data Augmentation
Similar to the model definition, the data augmentation process is implemented by defining two dicts in the config file:
data_loader and val_data_loader, which correspond to the processing workflows for the training and validation datasets, respectively. For example, the training set definition is as follows:
Training Strategy
A good training strategy is essential for training a high-accuracy model. For each training task, the corresponding training strategy is defined in the config file, as indicated by the float_trainer variable.
For this multi-network fusion model, we suggest training the lidar input model first, followed by the camera input model, and finally training the fusion model with both lidar + camera inputs to achieve better results.
The lidar input model can refer to the training of centerpoint.
The camera input model bevformer_henet_camera_multitask_nuscenes could be trained with configs/lidar_bevfusion/bevformer_henet_camera_multitask_nuscenes_pretrain.py,
and the command is:
The configuration for the float_trainer is as follows. You can see that we have loaded the pre-trained models for both the camera input and the lidar input separately.
The float_trainer defines our training approach from a broad perspective, including using multi-card distributed training (distributed_data_parallel_trainer), the number of epochs for model training, and the choice of optimizer.
Meanwhile, callbacks reflect the smaller strategies and operations you wish to implement during training. This includes learning rate adjustment methods (e.g., CosineAnnealingLrUpdater), validating model metrics during training (Validation), and saving model checkpoints (Checkpoint).
If you have additional operations you'd like to implement during training, you can also add them in this dict format.
From the above introduction, you should have a clear understanding of the config file's functionality. Using the training scripts mentioned earlier, you can train a high accuracy floating-point detection model. Of course, training a good detection model is not our final goal, it serves as a pretrain for training the fixed-point model later.
Quantized Model Training
Once we have the floating-point model, we can start training the corresponding fixed-point model. Similar to floating-point training, you can obtain a pseudo-quantized model by running the following scripts, which will use calibration and qat to achieve the target:
As you can see, our config file remains unchanged, only the stage type has been altered. The calibration process can provide better initialization parameters for QAT quantization training.
We configure Float2Calibration and Float2QAT to convert the model into calibration and QAT models, respectively. The specific configurations for calibration and QAT are as follows:
Different Training Strategies
As previously mentioned, the quantization training is actually the finetuning based on the pure floating-point training.
Therefore, in the quantization training, set the initial learning rate is to one-tenth of the floating-point training,
and the number of epochs of the training will greatly decrease as well.
The most important thing is that when defining model, we need to set pretrained to the address of the trained pure floating-point model.
After these simple adjustments, we can start training our quantized model.