#Board Evaluation

Board evaluation includes two categories: performance evaluation and accuracy evaluation.

Performance evaluation is used to assess the inference performance of the model. During evaluation, the primary focus is on two metrics: single-frame inference latency (Latency) and inference throughput (FPS).

• Latency refers to the time required for a model to complete a single inference. It mainly focuses on the system’s response speed to an individual request. The latency measurement starts when input data preparation is completed and ends when the inference result is obtained, and is measured in milliseconds (ms). If inference is performed using the asynchronous UCP interface, latency statistics are measured from the call to hbDNNInferV2 until the return of hbUCPWaitTaskDone.
• FPS represents the number of inference frames the system can process per unit time, reflecting the overall throughput of the system, and is measured in frames per second.

Accuracy evaluation is used to assess the correctness and consistency of model inference outputs in the deployment environment.

We provide the hrt_model_exec tool and the hbm_infer tool to facilitate model inference performance and accuracy evaluation.

#Evaluation Using hrt_model_exec Tool

hrt_model_exec is a model execution tool that can be used directly on the development board to obtain model information, perform single-frame inference, and evaluate model inference performance. The source code for the tool is located at samples/ucp_tutorial/tools/hrt_model_exec. The tool’s environment setup and three example usage scenarios are as follows:

# Set the environment variables, where arch represents the architecture type.
arch=aarch64
bin=../${arch}/bin/hrt_model_exec
lib=../${arch}/lib/
export LD_LIBRARY_PATH=${lib}:${LD_LIBRARY_PATH}

# Obtain Model Information
${bin} model_info --model_file=xxx.hbm

# Single-Frame Model Inference (Inference result files are available for accuracy evaluation)
${bin} infer --model_file=xxx.hbm --input_file=xxx.bin --enable_dump=true

# Model Performance Evaluation – Latency (Single-Threaded)
${bin} perf --model_file=xxx.hbm --thread_num 1 --frame_count=1000

# Model Performance Evaluation – FPS (Multi-Threaded, thread count adjustable based on actual conditions)
${bin} perf --model_file=xxx.hbm --thread_num 8 --frame_count=1000

For a more detailed introduction to the hrt_model_exec tool, please refer to Model Inference Tools - The hrt_model_exec Tool.

#Evaluation Using hbm_infer Tool

To improve the efficiency of accuracy evaluation, the toolchain provides the hbm_infer tool, which allows you to perform model pre- and post-processing on the X86 side using Python code, while the actual inference is executed by the board-side server. It is better suited for evaluation on large datasets. A basic single-process usage example is as follows:

# Create Session
sess = HbmRpcSession(host="xx.xx.xx.xx", local_hbm_path="xx.hbm", with_profile=True)

# Print Model Info
sess.show_input_output_info()

# Prepare Input Data
input_data = {'img': torch.ones((1, 3, 224, 224), dtype=torch.int8)}

# Perform Inference and Return Results
output_data = sess(input_data)

# Use the inference results for post-processing or accuracy evaluation.

# If with_profile=True is set, timing statistics for each stage of frame inference can be obtained, which can be used to evaluate single-frame latency.
print(sess.get_profile())

# Close Server
sess.close_server()

For a more detailed introduction to multi-process evaluation, model chaining, and transmission optimization using the hbm_infer tool, please refer to Model Inference Tools - The hbm_infer Tool.