#The hrt_model_exec Tool Introduction

#Tool Introduction

hrt_model_exec is a model execution tool that can evaluate the inference performance of the model and get the model information directly on the development board.

On one hand, it allows you to get a realistic understanding of the model's real performance; On the other hand, it also helps you to learn the speed limit that the model can achieve, which is useful information in application tuning.

hrt_model_exec tool source code is located in the samples/ucp_tutorial/tools/hrt_model_exec path of the horizon_j6_open_explorer publication. The structure is as follows:

├── include               # Header file
├── src                   # Source code
├── build.sh              # Compile script
├── build_x86.sh          # Compile to produce x86 tools
├── build_aarch64.sh      # Compile to produce aarch64 tools
├── CMakeLists.txt        
├── README.md
├── script                       
│   └── run_hrt_model_exec.sh    # The original aarch64 tool running script, which will be downloaded to the output_shared_J6_aarch64/script directory after compilation.
└── script_x86                   
    └── run_hrt_model_exec.sh    # The original x86 tool running script, which will be downloaded to the output_shared_J6_x86/script_x86 directory after compilation.

hrt_model_exec provides three types of functions including model inference infer, model performance analysis perf and viewing model information model_info, respectively, as shown in the following table:

The tool can view the tool's dnn prediction library version number with the -v or --version commands.

hrt_model_exec -v
hrt_model_exec --version

#Parameters Description

After setting the profile_path parameter and the tool runs normally, profiler.log and profiler.csv files will be generated. The files include the following parameters:

• ucp_version：UCP and HBRT version.

• perf_result：Record perf results.

• running_condition：Operating environment information.

• model_latency： Model node time consumption statistics.

• processor_latency：Model processor time consumption statistics.

• task_latency：Model task time-consuming statistics.

#Usage Instructions

This tool provides three types of functions: model information acquisition, single-frame inference function, and multi-frame performance evaluation.

Run hrt_model_exec, hrt_model_exec -h, or hrt_model_exec --help for tool usage details.

#model_info

#Overview

This parameter is used to get the model information, supporting both QAT and PTQ models. This parameter is used together with model_file to get detailed information about the model, including model input and output information hbDNNTensorProperties.

If model_name is not specified, all the models in the model are outputted. If model_name is specified, only the information of the corresponding model is outputted.

#Example

1. Single Model

hrt_model_exec model_info --model_file=xxx.hbm

../aarch64/bin/hrt_model_exec model_info --model_file=resnet50_224x224_nv12.hbm

Load model to DDR cost 1965.57ms.
This model file has 1 model:
[resnet50_224x224_nv12]
---------------------------------------------------------------------
[model name]: resnet50_224x224_nv12

input[0]:
name: input_y
valid shape: (1,224,224,1,)
aligned byte size: -1
tensor type: HB_DNN_TENSOR_TYPE_U8
quanti type: NONE
stride: (-1,-1,1,1,)

input[1]:
name: input_uv
valid shape: (1,112,112,2,)
aligned byte size: -1
tensor type: HB_DNN_TENSOR_TYPE_U8
quanti type: NONE
stride: (-1,-1,2,1,)

output[0]:
name: output
valid shape: (1,1000,)
aligned byte size: 4096
tensor type: HB_DNN_TENSOR_TYPE_F32
quanti type: NONE
stride: (4000,4,)

---------------------------------------------------------------------

2. Multi-model (output all model information)

hrt_model_exec model_info --model_file=xxx.hbm,xxx.hbm  

3. Multi-model - pack model (output specified model information)

hrt_model_exec model_info --model_file=xxx.hbm --model_name=xx

#infer

#Overview

This parameter is used for model inference, where the input images are defined by user and one frame is inferred. This parameter should be used together with input_file to specify the input image path, and the tool resizes the image according to the model information and organizes the model input information.

The program runs a single frame of data in a single thread and outputs the time of the model execution.

#Sample

1. Single Model

hrt_model_exec infer --model_file=xxx.hbm --input_file=xxx.bin

../aarch64/bin/hrt_model_exec infer --model_file=resnet50_224x224_nv12.hbm --input_file=zebra_cls.jpeg,zebra_cls.jpeg --input_img_properties=Y,UV --input_stride=50176,224,1,1;25088,224,2,1
Load model to DDR cost 1965.03ms.
[I][35143][06-28][10:39:51:373][file_util.cpp:527][hrt_model_exec][HRT_MODEL_EXEC] The input valid shape is (1,224,224,1), and the image [zebra_cls.jpeg] will be scaled to 224x224
[I][35143][06-28][10:39:51:377][file_util.cpp:527][hrt_model_exec][HRT_MODEL_EXEC] The input valid shape is (1,112,112,2), and the image [zebra_cls.jpeg] will be scaled to 224x224

---------------------Frame 0 begin---------------------
Infer time: 1.464 ms
---------------------Frame 0 end---------------------

2. Multi-model

hrt_model_exec infer --model_file=xxx.hbm,xxx.hbm --model_name=xx --input_file=xxx.jpg

#Optional Parameters

You can use the following parameters based on your scenario:

core_id、input_img_properties、input_valid_shape 、input_stride 、frame_count、dump_intermediate、enable_dump、dump_precision、dequantize_process、remove_padding_process、dump_format、dump_txt_axis、enable_cls_post_process、dump_path.

For parameter usage instructions, please refer to the Parameters Description section.

#perf

#Overview

This parameter is used to test the model performance.

In this mode, you does not need to input data, and the program automatically constructs the input tensor according to the model, and the tensor data are random numbers.

By default, the program runs 200 frames of data in a single thread. When perf_time is specified, frame_count is disabled, and the program will run for the specified period of time and then exit.

Outputs the latency and the frame rate of the model. The program prints the performance information every 200 frames: max, min, and average values of latency. If < 200 frames, prints once before the programs ends.

The program finally outputs the running-related data, including number of program threads, number of frames, total model inference time, average latency of model inference, and frame rate.

#Sample

1. Single Model

hrt_model_exec perf --model_file=xxx.hbm

../aarch64/bin/hrt_model_exec perf --model_file=resnet50_224x224_nv12.hbm --input_stride=50176,224,1,1;25088,224,2,1 --frame_count=200 --thread_num=8
[BPU][[BPU_MONITOR]][INFO]BPULib verison(2, 0, 1)[]!
Load model to DDR cost 1965.34ms.
Frame count: 200,  Thread Average: 5.262160 ms,  thread max latency: 5.498000 ms,  thread min latency: 1.636000 ms,  FPS: 1467.383789

Running condition:
  Thread number is: 8
  Frame count   is: 200
  Program run time: 136.440000 ms
Perf result:
  Frame totally latency is: 1052.432007 ms
  Average    latency    is: 5.262160 ms
  Frame      rate       is: 1465.845793 FPS

2. Multi-model

hrt_model_exec perf --model_file=xxx.hbm,xxx.hbm --model_name=xxx

#Optional Parameters

You can use the following parameters based on your scenario:

core_id、input_file 、input_img_properties、input_valid_shape、input_stride、frame_count、dump_intermediate、 perf_time、 thread_num、profile_path.

For parameter usage instructions, please refer to the Parameters Description section.

#Multi-input Model Description

The tool infer supports inference for multiple input models, supporting image input, binary file input, and text file input, with input data separated by commas. The model input information can be viewed via model_info.

Example:

hrt_model_exec infer --model_file=xxx.hbm --input_file=xxx.jpg,input.txt

#Dynamic input instructions

If the model input is dynamic, you need to use the input_valid_shape and input_stride parameters to complete the dynamic information according to the actual input situation. You can choose to specify the parameters in the following two ways:

• Only give the validShape or stride information of the dynamic input.

• Give the validShape or stride information of all inputs. The information of non-dynamic inputs must be consistent with the model information.

Taking the model in the Dynamic Input Introduction section as an example, you can run the model with the following command:

# Only given dynamically input information 
hrt_model_exec infer --model_file=xxx.hbm --input_file="input_y.bin,input_uv.bin,input_roi.bin" --input_valid_shape="1,220,220,1;1,110,110,2" --input_stride="49280,224,1,1;24640,224,2,1"

# Given all the input information
hrt_model_exec infer --model_file=xxx.hbm --input_file="input_y.bin,input_uv.bin,input_roi.bin" --input_valid_shape="1,220,220,1;1,110,110,2;1,4" --input_stride="49280,224,1,1;24640,224,2,1;16,4"

#Image type input instructions

When input_file is given an image input, you need to use the input_img_properties parameter to specify which color space of the image you want to use as input to the model. Currently, only Y and UV color spaces are supported.

hrt_model_exec infer --model_file=xxx.hbm --input_file="img.jpg,img.jpg,input_roi.bin" --input_img_properties="Y,UV"

#Tool operation instructions

#Build

There is a pre-configured compilation script build.sh in the ucp_tutorial/tools/hrt_model_exec directory. The options -a x86 and -a aarch64 support two compilation modes respectively. You can use this script and specify the compilation options for compilation. In addition, the directory also contains two compilation scripts, build_aarch64.sh and build_x86.sh, which correspond to two compilation options respectively. Compiling with these two scripts is equivalent to using the build.sh script and specifying the compilation options.

# Build board-side hrt_model_exec tools
bash -ex build_aarch64.sh
# Build x86-side hrt_model_exec tools
bash -ex build_x86.sh

#Execute

After building board-side hrt_model_exec tools, the output_shared_J6_aarch64 folder will be generated. You can use this tool by copying the folder to the board environment and executing output_shared_J6_aarch64/script/run_hrt_model_exec.sh.

After building x86-side hrt_model_exec tools, the output_shared_J6_x86 folder will be generated. You can use this tool on the x86 environment and executing output_shared_J6_x86/script_x86/run_hrt_model_exec.sh.

The run_hrt_model_exec.sh script is divided into two parts: setting environment variables and getting model information and inferring the model.

# Set environment variables
# arch represents the architecture type, aarch64 or x86
arch=aarch64
bin=../$arch/bin/hrt_model_exec
lib=../$arch/lib/
export LD_LIBRARY_PATH=${lib}:${LD_LIBRARY_PATH}

# Get model information, infer the model and get model performance
${bin} model_info --model_file=xxx.hbm

${bin} infer --model_file=xxx.hbm --input_file=xxx.bin

${bin} perf --model_file=xxx.hbm --frame_count=200

#FAQ

#How are Latency and FPS data calculated?

Latency refers to the average time spent by a single-process inference model. It focuses on the average time it takes to infer one frame when resources are sufficient. This is reflected in the statistics of single-core and single-thread running on the board. The pseudo code of the statistical method is as follows:

// Load model and prepare input and output tensor
  ...

  // Loop run inference and get latency
  {
    int32_t const loop_num{1000};
    start = std::chrono::steady_clock::now();
    for(int32_t i = 0; i < loop_num; i++){
        hbUCPSchedParam sched_param{};
        HB_UCP_INITIALIZE_SCHED_PARAM(&sched_param);
        // create task
        hbDNNInferV2(&task_handle, output_tensor, input_tensor, dnn_handle);
        // submit task
        hbUCPSubmitTask(task_handle, &sched_param);
        // wait task done
        hbUCPWaitTaskDone(task_handle, 0);
        // release task handle
        hbUCPReleaseTask(task_handle);
        task_handle = nullptr;
    }
    end = std::chrono::steady_clock::now();
    latency = (end - start) / loop_num;
  }

  // release tensor and model
  ...

FPS refers to the average number of frames per second of model inference performed by multiple processes at the same time, which focuses on the throughput of the model when fully utilizing the resources.

In on-board running situation, it is represented as single-core and multi-threaded. The statistical method is to perform the model inference by initiating multiple threads at the same time and calculate the total number of frames of the inference in average 1 second.

#Why is the FPS estimated by Latency inconsistent with the FPS measured by the tool?

Latency and FPS are different in statistical scenarios. Latency is single-process (single-core, single-thread) inference, and FPS is multi-process (single-core, multi-thread) inference, so the calculation is different. If the number of processes (threads) is set to 1 when counting the FPS, then the FPS estimated by Latency is consistent with the measured one.

#How to deal with out of memory during Perf?

When prompted to be out of memory, you can solve it in the following ways:

• Reduce the value of thread_num to reduce parallelism and reduce memory usage.

• Optimize the model to reduce the model's memory usage.