#Common Algorithm Model Samples

#Sample Location

The common algorithmic model samples are located at 03_classification/, 04_detection/ and 07_segmentation/ folders in the horizon_model_convert_sample path.

#Prepare Datasets

#Dataset Download Address

The dataset can be downloaded from the following address.

#Dataset Reference Structure

To facilitate your subsequent steps, after the dataset has been downloaded, you need to process the evaluation dataset according to the structure suggested by Horizon below.

imagenet/ 
 ├── calibration_data 
 │   ├── ILSVRC2012_val_00000001.JPEG 
 │   ├── ... 
 │   └── ILSVRC2012_val_00000100.JPEG 
 ├── ILSVRC2017_val.txt 
 ├── val 
 │   ├── ILSVRC2012_val_00000001.JPEG 
 │   ├── ... 
 │   └── ILSVRC2012_val_00050000.JPEG 
 └── val.txt

coco/
 ├── calibration_data
 │   ├── COCO_val2014_000000181007.jpg
 │   ├── ...
 │   └── COCO_val2014_000000181739.jpg
 └── coco_val2017
     ├── annotations
     │   ├── instances_train2017.json
     │   └── instances_val2017.json
     └── images
         ├── 000000000139.jpg
         ├── 000000000285.jpg
         ├── ...
         ├── 000000581615.jpg
         └── 000000581781.jpg 

 VOCdevkit/ 
 └── VOC2012 
     ├── Annotations 
     │   ├── 2007_000027.xml 
     │   ├── ... 
     │   └── 2012_004331.xml 
     ├── ImageSets 
     │   ├── Action 
     │   │   ├── jumping_train.txt 
     │   │   ├── jumping_trainval.txt 
     │   │   ├── jumping_val.txt 
     │   │   ├── ... 
     │   │   ├── val.txt 
     │   │   ├── walking_train.txt 
     │   │   ├── walking_trainval.txt 
     │   │   └── walking_val.txt 
     │   ├── Layout 
     │   │   ├── train.txt 
     │   │   ├── trainval.txt 
     │   │   └── val.txt 
     │   ├── Main 
     │   │   ├── aeroplane_train.txt 
     │   │   ├── aeroplane_trainval.txt 
     │   │   ├── aeroplane_val.txt 
     │   │   ├── ... 
     │   │   ├── train.txt 
     │   │   ├── train_val.txt 
     │   │   ├── trainval.txt 
     │   │   ├── tvmonitor_train.txt 
     │   │   ├── tvmonitor_trainval.txt 
     │   │   ├── tvmonitor_val.txt 
     │   │   └── val.txt 
     │   └── Segmentation 
     │       ├── train.txt 
     │       ├── trainval.txt 
     │       └── val.txt 
     ├── JPEGImages 
     │   ├── 2007_000027.jpg 
     │   ├── ... 
     │   └── 2012_004331.jpg 
     ├── SegmentationClass 
     │   ├── 2007_000032.png 
     │   ├── ... 
     │   └── 2011_003271.png 
     ├── SegmentationObject 
     │   ├── 2007_000032.png 
     │   ├── ... 
     │   └── 2011_003271.png 
     └── train.txt 

 cityscapes/ 
 ├── cityscapes_calibration_data 
 │   ├── aachen_000000_000019_leftImg8bit.png 
 │   ├── ... 
 │   └── aachen_000099_000019_leftImg8bit.png 
 ├── gtFine 
 │   ├── test 
 │   │   ├── berlin 
 │   │   ├── ... 
 │   │   └── munich 
 │   ├── train 
 │   │   ├── aachen 
 │   │   ├── ... 
 │   │   └── zurich 
 │   └── val 
 │       ├── frankfurt 
 │       ├── lindau 
 │       └── munster 
 ├── leftImg8bit 
 │   ├── test 
 │   │   ├── berlin 
 │   │   ├── ... 
 │   │   └── munich 
 │   ├── train 
 │   │   ├── aachen 
 │   │   ├── ... 
 │   │   └── zurich 
 │   └── val 
 │       ├── frankfurt 
 │       ├── lindau 
 │       └── munster 
 ├── license.txt 
 └── README 

#Prepare Models

When using the model conversion sample package, please prepare the corresponding floating-point model first.

Sources and modifications (if any) of the original models, please refer to below subsections.

#MobileNetv1

1. model source: https://github.com/shicai/MobileNet-Caffe

2. md5sum code:

   md5sum	File
   3fd6889ec48bda46451d67274144e2a8	mobilenet.caffemodel
   8922f90f629d428fecf866e798ac7c08	mobilenet_deploy.prototxt

#MobileNetv2

1. model source: https://github.com/shicai/MobileNet-Caffe

2. md5sum code:

   md5sum	File
   54aab8425ea068d472e8e4015f22360c	mobilenet_v2.caffemodel
   13101ee86ab6d217d5fd6ed46f7a4faa	mobilenet_v2_deploy.prototxt

#ResNet50

1. model source: https://pytorch.org/vision/main/models/generated/torchvision.models.resnet50.html

2. md5sum code:

   md5sum	File
   ebaeb70c972d1a3e6eb06c0e1541eeb4	resnet50.onnx

#GoogleNet

1. model source: https://github.com/HorizonRobotics-Platform/ModelZoo/tree/master/GoogleNet

2. md5sum code:

   md5sum	File
   f107ae6806ea1016afbc718210b7a617	googlenet.onnx

#EfficientNet_Lite0/1/2/3/4

1. model source: obtain the TAR package from https://github.com/tensorflow/tpu/tree/master/models/official/efficientnet/lite

2. md5sum of the converted ONNX models in Horizon model_zoo:

   md5sum	File
   001a329bd367fbec22b415c7a33d7bdb	efficientnet_lite0_fp32.onnx
   1205e95aea66650c71292bde236d55a9	efficientnet_lite1_fp32.onnx
   474741c15494b79a89fe51d89e0c43c7	efficientnet_lite2_fp32.onnx
   550455b41848d333f8359279c89a6bae	efficientnet_lite3_fp32.onnx
   bde7fe57eadb4a30ef76f68da622dcd5	efficientnet_lite4_fp32.onnx

3. Download and get .tflite from the TAR package, and then convert it to an ONNX model using the tflite2onnx tool (https://pypi.org/project/tflite2onnx/).

Note that model layouts may vary by tflite2onnx version. If the input layout of the converted ONNX model is NHWC, when building, the configure input_layout_train of the EfficientNet_Lite0/1/2/3/4 should be NHWC.

#Vargconvnet

1. model source: https://github.com/HorizonRobotics-Platform/ModelZoo/tree/master/VargConvNet

2. md5sum code:

   md5sum	File
   e21b8db17916f9046253bbe0bb8de3ef	vargconvnet.onnx

#Efficientnasnet_m

1. model source: https://github.com/HorizonRobotics-Platform/ModelZoo/tree/master/EfficientnasNet

2. md5sum code:

   md5sum	File
   fc36c052c6f034c0b64a6197b91b0c62	efficientnasnet-m.onnx

#Efficientnasnet_s

1. model source: https://github.com/HorizonRobotics-Platform/ModelZoo/tree/master/EfficientnasNet

2. md5sum code:

   md5sum	File
   e2744bd748f4265f4488676835a6ca24	efficientnasnet-s.onnx

#ResNet18

1. model source: https://pytorch.org/vision/main/models/generated/torchvision.models.resnet18.html

2. md5sum code:

   md5sum	File
   62de4ff68317c65ab4bb6a451e719e6d	resnet18.onnx

#YOLOv2_Darknet19

1. To get YOLOv2_Darknet19, first download the .cfg and .weight files of YOLOv2 608x608 from YOLO's official website (https://pjreddie.com/darknet/yolo/) and convert it to a Caffe model using the darknet2caffe conversion tool (https://github.com/xingyanan/darknet2caffe).

2.md5sum code:

#YOLOv3_Darknet53

1. YOLOv3_Darknet53 model:

URL: https://github.com/ChenYingpeng/caffe-yolov3/ .

The caffemodel file can be downloaded from the Baidu Cloud URL provided in the README.md file in GitHub, in which you should add an NCHW2NHWC Permute operation to the output node.

2. md5sum code:

   md5sum	File
   935af6e1530af5c0017b3674adce95e9	yolov3_transposed.prototxt
   9a0f09c850656913ec27a6da06d9f9cc	yolov3.caffemodel

#YOLOv5x

1. YOLOv5x model: Download the corresponding pt file from https://github.com/ultralytics/yolov5/releases/tag/v2.0

2. md5sum code:

   md5sum	File
   2e296b5e31bf1e1b6b8ea4bf36153ea5	yolov5l.pt
   16150e35f707a2f07e7528b89c032308	yolov5m.pt
   42c681cf466c549ff5ecfe86bcc491a0	yolov5s.pt
   069a6baa2a741dec8a2d44a9083b6d6e	yolov5x.pt

• To better adapt to post-processing code, before exporting the ONNX model, we modified the code at Github as follows (for more code details see: https://github.com/ultralytics/yolov5/blob/v2.0/models/yolo.py):

def forward(self, x):
    # x = x.copy()  # for profiling
    z = []  # inference output
    self.training |= self.export
    for i in range(self.nl):
        x[i] = self.m[i](x[i])  # conv
        bs, _, ny, nx = x[i].shape  # x(bs,255,20,20) to x(bs,3,20,20,85)
        #  x[i] = x[i].view(bs, self.na, self.no, ny, nx).permute(0, 1, 3, 4, 2).contiguous()
        x[i] = x[i].permute(0, 2, 3, 1).contiguous()

The bottom left image displays the visualization of a certain output node before modifying the model, while the bottom right image displays the visualization of the corresponding output node after the modification.

• After download, perform the pt to ONNX file conversion using the script https://github.com/ultralytics/yolov5/blob/v2.0/models/export.py .

#SSD_MobileNetv1

1. SSD_MobileNetv1 model: Obtain the Caffe model from URL: https://github.com/chuanqi305/MobileNet-SSD

2. md5sum code:

   md5sum	File
   bbcb3b6a0afe1ec89e1288096b5b8c66	mobilenet_iter_73000.caffemodel
   3c230e4415195a50c6248be80c49882d	MobileNetSSD_deploy.prototxt

#Efficientdetd0

1. model source: https://github.com/HorizonRobotics-Platform/ModelZoo/tree/master/EfficientDet

2. md5sum code:

   md5sum	File
   ec4129c4b300cd04f1e8f71e0fe54ca5	efficientdet_nhwc.onnx

#CenterNet_Resnet101

1. model source: https://github.com/HorizonRobotics-Platform/ModelZoo/tree/master/Centernet

2. md5sum code:

   md5sum	File
   db195ff784792f475e573c5126401d2a	centernet_resnet101_coco_modify.onnx

#Fcos_efficientnetb0

1. model source: https://github.com/HorizonRobotics-Platform/ModelZoo/blob/master/Fcos_Efficientnetb0

2. md5sum code:

   md5sum	File
   9f9a1fe8508e2bd068e70146eb559b4f	fcos_efficientnetb0.onnx

#Yolov4

1. model source: https://github.com/HorizonRobotics-Platform/ModelZoo/tree/master/YoloV4

2. md5sum code:

   md5sum	File
   aaa3c3e5e4c4c1d4830b6501b1720e4d	yolov4_efficientnetb0.onnx

#YOLOv3_VargDarknet

1. YOLOv3_VargDarknet model

URL: https://github.com/HorizonRobotics-Platform/ModelZoo/tree/master/Yolov3_VargDarknet . The model file can be downloaded from the Baidu Cloud URL provided in the README.md file in GitHub.

2. md5sum code:

   md5sum	File
   fd4e46bc7c9798b51778d3aa09c5053a	yolov3_vargdarknet53.onnx

#Fcos_resnet50

1. model source: https://github.com/HorizonRobotics-Platform/ModelZoo/tree/master/Fcos_Resnet50

2. md5sum code:

   md5sum	File
   0218942777615fac2f54cefdac4fbfa7	fcos_resnet50.onnx

#Fcos_resnext101

1. model source: https://github.com/HorizonRobotics-Platform/ModelZoo/tree/master/Fcos_Resnext101

2. md5sum code:

   md5sum	File
   4b80efd22448021721ac5a860909c59f	fcos_resnext101.onnx

#Unet_mobilenet

1. model source: https://github.com/HorizonRobotics-Platform/ModelZoo/tree/master/MobilenetUnet

2. md5sum code:

   md5sum	File
   21c6c645ebca92befbebc8c39d385c1e	tf_unet_trained.onnx

#DeeplabV3plus_efficientnetb0

1. model source: https://github.com/HorizonRobotics-Platform/ModelZoo/tree/master/DeeplabV3Plus

2. md5sum code:

   md5sum	File
   cf3a683f31b4b0ebe090647729f869d9	deeplabv3plus_efficientnetb0.onnx

#Fastscnn_efficientnetb0

1. model source: https://github.com/HorizonRobotics-Platform/ModelZoo/tree/master/FastSCNN

2. md5sum code:

   md5sum	File
   c1ace8f08a9c7b9c91509fa68327d0c8	fastscnn_efficientnetb0.onnx

#Deeplabv3plus_dilation1248

1. model source: https://github.com/HorizonRobotics-Platform/ModelZoo/tree/master/DeeplabV3Plus

2. md5sum code:

   md5sum	File
   ad002e572cbb49e1e99d893aac69f3e3	deeplabv3_cityscapes_dila1248_permute.onnx

#Deeplabv3plus_efficientnetm1

1. model source: https://github.com/HorizonRobotics-Platform/ModelZoo/tree/master/DeeplabV3Plus

2. md5sum code:

   md5sum	File
   0a1dfd01e173c68630d9e8dc1a6036fe	deeplabv3plus_efficientnetm1.onnx

#Deeplabv3plus_efficientnetm2

1. model source: https://github.com/HorizonRobotics-Platform/ModelZoo/tree/master/DeeplabV3Plus

2. md5sum code:

   md5sum	File
   c11a2673c4b3cf6e5d7bf1a051925d38	deeplabv3plus_efficientnetm2.onnx

#Demo Of Algorithm Model Samples

Taking the Resnet50 model as an example, this section illustrates the steps of floating-point to fixed-point model conversion by using the scripts in 03_classification/03_resnet50/.

#Docker Container Prep

First complete the docker installation and configuration and enter the docker container accoridng to Docker Container Deployment .

#Obtain The Original Model And Calibration Dataset

Execute 00_init.sh in the 03_classification/03_resnet50/ folder to obtain the model and calibration dataset needed for the current sample.

# 1. Enter the folder the demo script resides
cd samples/ai_toolchain/horizon_model_convert_sample/03_classification/03_resnet50/
# 2. Execute the script to obtain the original model and calibration dataset
sh 00_init.sh

#Check If the Model is Executable

Run the script as follows:

# Execute model checking
sh 01_check.sh

#Prepare Calibration Dataset

Run the 02_preprocess.sh script in the same directory, as follows:

# Convert the images in 01_common/calibration_data/imagenet
# to: ./calibration_data_bgr
sh 02_preprocess.sh

#Building Models

Run the 03_build.sh script in the same directory, as follows:

sh 03_build.sh

The output of the above script is as follows:

ls model_output | cat
resnet50_224x224_nv12.hbm
resnet50_224x224_nv12_calibrated_model.onnx
resnet50_224x224_nv12_ptq_model.onnx
resnet50_224x224_nv12_optimized_float_model.onnx
resnet50_224x224_nv12_original_float_model.onnx
resnet50_224x224_nv12_quantized_model.bc
resnet50_224x224_nv12.html
resnet50_224x224_nv12.json 
resnet50_224x224_nv12_quant_info.json 
resnet50_224x224_nv12_advice.json
resnet50_224x224_nv12_node_info.csv
hb_compile.log

#Single Image Inference

Run the 04_inference.sh script to inference a single image, as follows:

sh 04_inference.sh

#Accuracy Evaluation

Run the 05_evaluate.sh script to evaluate the accuracy, as follows:

export PARALLEL_PROCESS_NUM=${parallel_process_num}
sh 05_evaluate.sh

#FAQ

#Why is the reproduced accuracy slightly different from the indicators in the documentation?

There are two possible reasons:

1. There may be minor differences in calculation methods when in different server environments, which can cause small data fluctuations in the accuracy of the compiled fixed-point ONNX models in different server environments when compared with the documentation.
2. The different versions of third-party libraries such as OpenCV and NumPy, which may produce different results after pre-processing, and this may also lead to slight data fluctuations in reproduced accuracy when compared with the documentation.

No need to worry much about this situation, the records provided in the documentation is only for reference, and it is ok that your reproduced accuracy is slightly different from those in documents.

#Why is the fixed-point model accuracy not aligned with the on-board accuracy of the hbm file in the ai_benchmark sample?

In the standard delivery, when adding the sample, we have already aligned the fixed-point model accuracy with the hbm on-board accuracy in the ai_benchmark sample.

If you find any unaligned accuracy, we recommend you first checking the model input for consistency.

When executing the fixed-point model evaluation script, you use the dataset of image type, while for the hbm model used on board, you use the binary dataset converted by the hb_eval_preprocess tool.

Based on this, if the dataset you used on-board is not generated by using the above methods, we recommend that you first use our data preprocessing tool (i.e., hb_eval_preprocess) to regenerate the dataset needed for on-board running on the same server that you run the fixed-point model accuracy and rerun the on-board accuracy to ensure the model inputs are consistent.