The BEV reference algorithm is developed based on Horizon's self-developed deep learning framework. The training configuration is located under the HAT/configs/bev/ path. The following uses HAT/configs/bev/bevformer_tiny_resnet50_detection_nuscenes.py as an example to explain how to configure and train the BEV reference algorithm.
Here is an example of the nuscense dataset, which can be downloaded from https://www.nuscenes.org/nuscenes .
Also, in order to improve the speed of training, we have done a packing of the original jpg format dataset to convert it to lmdb format. Just run the following script and it will successfully convert the dataset.
The above two commands correspond to transforming the training dataset and the validation dataset, respectively.
After the packing is completed, the file structure in the data directory should look as follows.
The train_lmdb and val_lmdb are the training and validation datasets after packaging, and are the datasets that the network will eventually read. metas is the map information needed for the segmentation model.
These three commands correspond to the following three training stages:
Floating-point Model Training (float):
This is the first step of training, where the model is trained with floating-point precision to generate a high-accuracy floating-point model.
This model will serve as the foundation for subsequent calibration and quantization-aware training.
Calibration Model Training (calibration):
In this stage, the model weights are not updated. Instead, real samples are fed into the model to collect the input and output ranges for each layer, generating quantization parameters (e.g., scale and zero-point).
The purpose of calibration is to prepare for the subsequent Quantization-Aware Training (QAT).
Quantization-Aware Training (qat):
This is the final stage of training, where the model is fine-tuned based on the quantization parameters.
QAT simulates the quantization process (e.g., using low-precision computations) to ensure the model performs well during actual deployment.
After completing the training, we get the trained floating-point, quantized, or fixed-point model. Similar to the training method, we can use the same method to complete metrics validation on the trained
model and get the metrics of Float, Calibration, and Quantized, which are floating-point, quantized, and fully fixed-point metrics, respectively.
Similar to the model training, we can use --stage followed by "float", "calibration", to validate the trained floating-point model, and quantized model, respectively.
The following command can be used to verify the accuracy of a fixed-point model, but it should be noted that hbir must be exported first:
The quantization training toolchain integrated in HAT is mainly prepared for the Horizon computing platform, so it is necessary to check and compile the quantized models.
We provide a model checking interface in HAT, which can be used to check whether the defined quantization model can run normally on the BPU before training:
After the model is trained, you can use the compile_perf_hbir script to compile the quantized model into an HBM file that supports on-board running. The tool can also predict the performance on the BPU.
Among them, the type under model represents the name of the defined model, while the remaining variables represent other components of the model. The advantage of defining the model in this way is that we can easily replace the structure we want. For example, if we want to train a model with a backbone of ResNet50, we only need to replace the backbone under model.
Similar to the definition of model, the data augmentation process is implemented by defining the data_loader and val_data_loader dictionaries in the config configuration file, which correspond to the processing flows of the training set and validation set, respectively. Taking data_loader as an example:
The type here directly uses PyTorch's built-in interface torch.utils.data.DataLoader, which is used to combine images into batches of size batch_size. The only variable that might require attention here is dataset. CocoFromLMDB indicates that images are read from the LMDB dataset, with the path being the one mentioned in the first part of the dataset preparation. Under transforms, there is a series of data augmentation operations. In val_data_loader, except for image flipping (RandomFlip), the other data transformations are consistent with those in data_loader. You can also insert new dictionaries into transforms to implement your desired data augmentation operations.
To train a high-accuracy model, a good training strategy is essential. For each training task, the corresponding training strategy is also defined in the config file, as can be seen from the float_trainer variable.
The float_trainer defines our training method from a high-level perspective, including multi-GPU distributed training (distributed_data_parallel_trainer), the number of training epochs, and the choice of optimizer.
At the same time, the callbacks section reflects the small strategies used during the training process and the operations you want to implement, such as learning rate adjustment (WarmupStepLrUpdater), validation of model metrics during training (Validation), and saving the model (Checkpoint). Of course, if you have specific operations you want to implement during training, you can add them in this dict format.
Note
If you need to reproduce the accuracy, it is best not to modify the training strategy in the config file. Otherwise, unexpected training results may occur.
From the above introduction, you should now have a clearer understanding of the functionality of the config file. Then, using the training scripts mentioned earlier, you can train a high-accuracy pure floating-point detection model. However, training a good detection model is not our ultimate goal; it serves as a pre-trained model for training fixed-point models later.
Once we have a pure floating-point model, we can start training the corresponding fixed-point model. Similar to the floating-point training method, we can obtain a pseudo-quantized model by running the following scripts:
It can be seen that our configuration file has not changed, only the stage type has been modified. Bevformer needs to go through two steps, Calibration and Quantization-Aware Training (QAT), to achieve the quantization goal.
First, we perform model calibration (calibration). This step does not update the model weights but uses real samples to collect the input and output ranges of each layer, obtaining the quantization scale for each layer.
The calibration_trainer is defined as follows:
When training a quantized model, it is necessary to set up the model_convert_pipeline. This pipeline first loads the floating-point weights of the model and then uses Float2Calibration to convert the model from a floating-point model to a calibrated model.
The calibration_qconfig_setter defines the quantization configuration of the model. For the basic configuration method, readers can refer to the Building QConfig section.
The calibration_qconfig_setter for bevformer is defined as follows:
Here is a suggested revision: For the above code example, we will break it down step by step for you.
QconfigSetter.get_fuse_mode sets the default BN fuse mode to BNAddReLU, which fuses the BN, Add, and ReLU operators together.
The code constructs two key dictionaries:
int16_qconfig specifies qint16 output for layers sensitive to numerical precision in view_transformer and bev_decoders.
None_qconfig_template sets layers like positional encoding to None, forcing them to remain in floating-point computation.
q_templates defines the priority of configuration application. First, the global default is set to qint8. Then, the Int16 and None configurations are applied, with freeze=True. This effectively locks the configuration of these critical layers, preventing them from being overridden by subsequent rules. Finally, the remaining Conv and Matmul operators are handled using the general templates MatmulDtypeTemplate and ConvDtypeTemplate.
Before the final initialization of the Setter, the code also uses mix_qconfig to individually assign MixObserver to all Int16 layers, achieving a mixed-precision strategy of "weights in Int8 + activations in Int16."
Quantization training is essentially a fine-tuning process based on pure floating-point training. Therefore, during quantization training, the initial learning rate is set to one-twentieth of the floating-point training rate, and the number of training epochs is significantly reduced.
It is worth noting that qat_qconfig_setter is similar to calibration_qconfig_setter, but the observer is changed to MinMaxObserver, which is much faster than MSEObserver. We recommend using MinMaxObserver for QAT.
When defining the model, we first convert the model into a QAT model, then load the calibration model saved in the previous step, and start training.
For more comprehensive knowledge about quantization training, such as preparing floating-point models, operator replacement, inserting quantization and dequantization nodes, setting quantization parameters, and operator fusion, please refer to the Quantization-Aware Training (QAT) section.
