#FAQ

#Unable to import hat in Docker container?

If you get a GPU RuntimeError when running a sample in GPU Docker, or a ModuleNotFoundError: No module named 'hat', Segmentation fault (core dumped), or something like that when loading a hat, you can go in the following two directions troubleshooting:

1. Start the GPU Docker container using the run_docker.sh script provided in the OE package.
2. Execute the following script in the container to check if torch and cuda are working. If CUDA cannot be called, you can use nvidia-smi to check if the driver version meets the requirement. Because the OE package has upgraded the torch environment to 1.1.30+cu116 since v1.1.60, the corresponding driver requirements can be found in: Environment Deployment - Docker Container Deployment.

import torch
import time
print(torch.__version__)
print(torch.cuda.is_available())
a = torch.randn(10000, 1000)
b = torch.randn(1000, 2000)
t0 = time.time()
c = torch.matmul(a, b)
t1 = time.time()
print(a.device, t1 - t0, c.norm(2))
device = torch.device('cuda')
a = a.to(device)
b = b.to(device)
t0 = time.time()
c = torch.matmul(a, b)
t2 = time.time()
print(a.device, t2 - t0, c.norm(2))
t0 = time.time()
c = torch.matmul(a, b)
t2 = time.time()
print(a.device, t2 - t0, c.norm(2))

#How to find the specific implementation of the module in the config file?

The corresponding implementation of the module is the value corresponding to type, which can be interpreted as class, and the class will be instantiated when building the model, and the path of the class can be obtained in the following two ways:

1. The implementation of each module is stored in hat/models/module name/, or you can check in init.py to see if the class is included in that path. For example, if the dict["type"] of neck is "FastSCNNNNeck" in the bev_mt_ipm model's config file, then the FastSCNNNNeck implementation is in hat/models/necks/fast_scnn.py. In addition, the module definitions for a particular task will be under task_modules (head, postprocessing, decoder). For example, if the head configuration of the detection branch is "CenterPoint3dHead", which is part of the task_modules include, it will actually appear as hat/models/task_modules/centerpoint3d/head.py.
2. Use the grep command to find it, for example:

grep -rn "class $type_value$" $HAT_PATH$

#Why the config prints correctly but launch trainer fails?

For example, the following error requires you to change device_ids in config to the currently available GPU resources, if the error contains "CUDA out of memory", you need to change batch_size_per_gpu to the batch size that the current resources can support.

Traceback (most recent call last):
File "/root/.local/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 59, in _wrap
    fn(i, *args)
File "/usr/local/lib/python3.6/site-packages/hat/engine/ddp_trainer.py", line 394, in _main_func
    torch.cuda.set_device(local_rank % num_devices)
File "/root/.local/lib/python3.6/site-packages/torch/cuda/__init__.py", line 311, in set_device
    torch._C._cuda_setDevice(device)
RuntimeError: CUDA error: invalid device ordinal
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

#Why the configuration of the device on the data run does not take effect?

Please check whether to_cuda succeeds in hat/engine/processors/processor.py, it may not because the data is not of the tensor type and the device is not called.

#Why the configuration of the trainer's runtime device does not take effect?

The device and device_id configurations are prioritized: if the trainer is invoked using the default launch of train.py, only the device-ids will work, and the device configuration will not work. If the trainer (predictor, loop base, etc.) is called directly, the device can work directly.

#How to set the validation/printing frequency?

In the config file there is a parameter for the callback frequency, the authentication frequency can be modified in the following way, and all other callback related frequencies are set in the form of "xxx_frep":

val_metric_updater = dict(
    type="MetricUpdater",
    metrics=[val_nuscenes_metric, val_miou_metric],
    metric_update_func=update_val_metric,
    step_log_freq=10000,
    epoch_log_freq=1,    # Configure here
    log_prefix="Validation " + task_name,
)

#Why the plt.show image is not displayed?

This may be due to the fact that there is no graphical interface in the current Linux system, you can add plt.savefig('. /test.png',dpi=300) in the corresponding code of each dataset under the path of hat/visualize, save the file locally and check it again.

#How to new dataset?

dataloader internal class call flow chart is shown below (take Coco dataset as an example), in which all classes and interfaces are called in config, data loading is realized by calling dataloader, dataloader loaded dataset is returned by Coco class, Coco class reads lmdb format packaged data, which is realized by calling packer interface. Packed data in lmdb format, which is realized by calling packer interface, implementation flow: packer calls CocoDetectionPacker class to pack the data returned by CocoDetection (image, label...) into the set format (lmdb, mxrecord).

To implement a dataset requires an override of the classes involved above, overriding __getitem()__ based on the dataset's format (which can also be adapted to the format of the original dataset).

#How to resume an unexpectedly interrupted training session?

It is possible to recover unexpectedly interrupted training, or to recover only the optimizer for fine-tuning, by configuring the resume_optimizer and resume_epoch_or_step fields in {stage}_trainer of config. For example, add the following fields to trainer in the config file:

# Restore the status of optimizer and LR
resume_optimizer=True,
# Restore epoch and step
resume_epoch_or_step=True,
model_convert_pipeline=dict(
    type="ModelConvertPipeline",
    converters=[
        dict(
            type="LoadCheckpoint",
            checkpoint_path=os.path.join(
                # Load model parameters
                ckpt_dir, "float-checkpoint-best.pth.tar"
            ),
        ),
    ],
)

#How to visualize the model structure?

Supports visualization via the hb_model_info tool -v parameter.

#How to change the method of calibration?

The calibration_trainer field in config can be modified to add a new qconfig_params dictionary as referenced below:

calibration_trainer = dict(
    type="Calibrator",
    model=model ,
    model_convert_pipeline=dict(
        type="ModelConvertPipeline",
        qat_mode="fuse_bn",
        qconfig_params=dict(
            activation_calibration_observer="percentile",
            activation_calibration_qkwargs=dict(
                percentile=99.975,
            ),
        ),
    converters=[
        dict(
            type="Loadcheckpoint",
            checkpoint_path=os.path.join(
                ckpt_dir, "float-checkpoint-best.pth.tar'
            ),
        ),
        dict(type="Float2Calibration", convert_mode=convert_mode),
    ],
),
......

#Whether the DETR model supports deformable conv?

This operator is mainly used in deformable structures in deformable-DETR. Horizon is not yet supported, but the gridsample operator, which is already supported, can be used instead.

#Why is Lss less accuray than IPM in the BEV model?

The input resolution of the Lss is 256x704, which is lower than the 512x960 of the IPM, so the accuracy is even lower.

#What are the constraints on the Lss inputs in the BEV model?

The Lss does dimension folding before grid_sample, so there are operator compilation constraints on h, w of input_feature, currently: H,W ∈ [1, 1024] and H*W ≤ 720*1024.

#How to solve the time consuming mul(depth,feature) of Lss in BEV model?

The featuremap H and W can be converted to [128,128] by the grid_sample operator before doing the mul calculation.

#How is voxelpooling implemented for Lss in BEV model? How many points are selected?

In order not to lose the point cloud features located in the same voxel, we will sample each voxel 10 times and sum each point cloud feature to get a 128x128x64 BEV feature map, corresponding to the following code:

num_points=10
for i in range(self.num_points):    # 10 samples per voxel
    homo_feat = self.grid_sample(
        feat,
        self.quant_stub(points[i]),
    )
    homo_dfeat = self.dgrid_sample(
        dfeat,
        self.dquant_stub(points[i + self.num_points]),
    )
    homo_feat = self.floatFs.mul(homo_feat, homo_dfeat)

trans_feat = homo_feats[0]
for f in homo_feats[1:]:
    trans_feat = self.floatFs.add(trans_feat, f)    # Point cloud feature summation

#How are reference points selected for Lss of the BEV model?

The generation of points in _gen_reference_point sets invalid points outside the feature range to the larger value. In order not to take invalid points, the first 10 points with smaller values are aggregated using topk (k=10 for faster training).

#How does the Lss for the BEV model handle the case where the gridsample input is large?

The Lss model is prone to exceed the 720*1024 BPU operator constraint limit for H*W because it collapses 3 dimensions into 1 before the gridsample operator. At this point it is recommended that before dimension folding (i.e. dfeat = dfeat.view(B, 1, -1, H * W)), the dimensions that may be overrun are split, the gridsample is computed separately, and the results are finally superimposed.

#Why is the BEV model's gkt accuracy metric lower? What are the advantages?

The main reason why the accuracy of Transformer-based gkt is lower than that of IPM is that the amount of data in the open source dataset is not enough, and we try to add some business data to get a higher accuracy than that of IPM. Moreover, the gkt model is more robust and can solve the accuracy effect caused by camera offset.

#Whether the transfomer of gkt is global in the BEV model?

It's not global, it's a kernel (3x3) feature that is selected for attention, so it's a kernel-transformer.

#Whether the BEV model supports the public version of bevformer?

J6 already supports bevformer optimized version.

#Whether the accuracy is constrained between BEV model detection and segmentation tasks?

There will be an impact because the accuracy of the detection task in multitasking will be degraded, which needs to be balanced by the training strategy.

#What are the components of latency in the PointPillars model?

The current latency of Pointpillars consists of two parts: the first part is the pre-processing (voxelization), i.e., the time from the point cloud input to the head, the second part is the post-processing time.

#How does the amount of point cloud affect the performance of the PointPillars model?

The higher the number of valid point clouds, the longer the pre-processing time, as specified in the Voxelization cfg parameter set:

# Voxelization cfg
pc_range = [0, -39.68, -3, 69.12, 39.68, 1]    # Valid point cloud range
voxel_size = [0.16, 0.16, 4.0]                 # voxel size
max_points_in_voxel = 100                      # Maximum number of points in a voxel
max_voxels_num = 12000                         # Number of voxel

#What operations are included in the preprocessing of a PointPillars model?

The pre-processing is pillarization (voxel), the corresponding stage is voxeliza tation, and the flow is shown schematically below:

#How many types of task detection does the PointPillars model support? How can it be extended?

Currently the PointPillars_Kitti_Car model only supports the car class of detection, but we also provide a multi-category detection model centerpoint_pointpillar_nuscenes.

#How is the input data for PointPillars' board-side hbm model generated? How is it preprocessed?

The data preprocessing includes reshape and padding (padding to (1,1,150000,4) where 150000 is the maximum amount of point cloud in the bin file), the reference code is shown below:

#padding bin
padding_np=(np.ones(150000*4)*-100).astype(np.float32).reshape(1,1,-1,4)

for f in os.listdir(list_dir):
    ori_bin=np.fromfile(os.path.join(list_dir,f), dtype=np.float32).reshape(1,1,-1,4)
    con_bin=np.concatenate((ori_bin,padding_np),axis=2)
    pad_bin=con_bin[:,:,:150000,:]
    if not os.path.exists(save_path):
        os.mkdir(save_path)

    with open(os.path.join(save_path,f),"w") as sf:
        print("the pad file is saved in:",os.path.join(save_path,f))
        pad_bin.tofile(sf)

#Why does the board side perf data for the PointPillars model not match the official metrics?

When using the hrt_model_exec perf tool on the board side to evaluate performance, please specify a real input_file, otherwise the tool will use randomly generated point cloud data, which may result in inaccurate perf metrics.

#Whether the PTQ program support conversion to Lidar models in the PointPillars model?

It is supported on the functional link to export ONNX models from the training framework for PTQ conversion checking. However, from the established experience, the Lidar point cloud model is more risky in terms of the accuracy of quantization by going to the PTQ scheme, mainly due to the following reasons: the point cloud is relatively sparse, and the data distribution situation is not friendly to the quantization.

#Can centerpoint and pointpillars be used interchangeably?

The two models can be used interchangeably, and only the relevant configurations involved in the dataset need to be modified correspondingly, e.g., point cloud range, prediction category, and anchor, target, etc. for post-processing.