This tutorial primarily guides you on training a QCNet model from scratch on the Argoverse 2 dataset, including floating-point, quantized, and fixed-point models.
QCNet is a trajectory prediction model, which can be referenced in the paperQuery-Centric Trajectory Prediction.
If you want to quickly train the QCNet model, start by reading this section. Similar to other tasks, all training and evaluation tasks in HAT are accomplished using a tools + config approach. After preparing the raw dataset, you can conveniently complete the entire training process by following the steps below.
The first step before training the model is to prepare the dataset. You can download it from the Argoverse 2 Dataset.
Download the following: Training, Validation, and Test.
After downloading, extract and organize the folders as follows:
data |-- argoverse-2 |-- train |-- val |-- test
To speed up the training process, we packed the dataset information files into lmdb format. You can convert the dataset by running the following script:
Next, you can start training. Training can be done using the following scripts. Before training, ensure that the dataset path in the configuration has been switched to the packed dataset path.
These commands respectively complete the training of the floating-point model and the fixed-point model. Training the fixed-point model requires a pre-trained floating-point model. For more details, please read the section on Quantized Awareness Training.
The export path for the fixed-point model is ${ckpt_dir}.
Due to the relative spatiotemporal encoding used in QCNet, which introduces spatiotemporal invariance, we can reuse the features of historical frame encodings during model deployment. We offer a cold start model deployment solution: encoding frames sequentially, where the cold start is based on QAT training of the floating-point model from the hot start.
After completing the training, we can obtain the trained floating-point and quantized models.
Similar to the training method, we can use the same approach to evaluate the metrics of the trained models.
This will yield metrics for Float, Calibration, and Qat. The former provides the metrics for the floating-point model,
while the latter two provide the metrics for the models obtained through quantization calibration and quantized awareness training, respectively.
The quantization training toolchain integrated in HAT is mainly prepared for Horizon's computing platform. Therefore, checking and compiling the quantized model is necessary.
We provide an interface for model checking in HAT, allowing you to define a quantized model and then check if it can run normally on BPU:
After training the model, you can compile the quantized model into an hbm file that can be run on board using the compile_perf_hbir script. This tool can also estimate the performance on BPU:
The QCNet network structure can be referenced from the paper.
We define and modify the model conveniently by defining a dict variable named model in the config file.
Similar to the definition of model, the data_loader and val_data_loader for the training and validation stages are defined in the config configuration file using the data_loader and val_data_loader dictionaries, respectively, which correspond to the processing pipelines for the training and validation datasets.
Since QCNet does not include complex data augmentation (transforms=None), collate_fn defines how individual data items are grouped into batches, including alignment operations.
First, let's introduce the strategy for training the floating-point model of QCNet on the Argoverse2 dataset.
We use AdamW as the optimizer with settings lr=5e-4 and weight_decay=1e-4. However, we do not apply weight decay to bias , nn.Embedding , and normalization layers.
Therefore, using custom_param_optimizer here easily achieves this setup. We employ a Cosine learning rate schedule, set the warmup length to 1 epoch, and train the model for a total of 64 epochs.
Below is a complete configuration example for the float_trainer in the config file:
For key steps in quantization training, such as preparing the floating-point model, operator replacement, inserting quantization and dequantization nodes, setting quantization parameters, and operator fusion,
please refer to the Quantized Awareness Training (QAT) section. Here we focus on defining and using the quantized model.
The quantization training strategy for the QCNet example model can be found in the configs/traj_pred/qcnet_oe_argoverse2.py file,
mainly divided into the quantization calibration calibration and quantized awareness training qat phases.
Here, float2calibration and float2qat respectively define the conversion process from floating-point to calibration model and from floating-point to quantized awareness training model. The cali_qconfig_setter and qat_qconfig_setter provide the qconfig settings for the calibration model and the qat model respectively.
For more details on setting and debugging qconfig, such as default qconfig settings and quantization-sensitive operator settings, please refer to the Quantized Awareness Training - Qconfig Details section.
Then you should rename saved "calibration-checkpoint-last.pth.tar“ to "calibration-checkpoint-best-defaultQconfig.pth.tar".
And calling the sensitivity analysis tool to compare the similarity between the float model and the calibration model layer by layer with the command:
