#Board Evaluation

This section covers the evaluation stage of the model deployment link. The prerequisites for this section are that the inference application already runs on the board and the hbm model is deployed and produces single-frame results. The outputs of this section are the Latency / FPS performance metrics and the accuracy evaluation result.

Board evaluation includes two categories: performance evaluation and accuracy evaluation.

Performance evaluation is used to assess the inference performance of the model. During evaluation, the primary focus is on two metrics: single-frame inference latency (Latency) and inference throughput (FPS).

• Latency refers to the time required for a model to complete a single inference. It mainly focuses on the system’s response speed to an individual request. The latency measurement starts when input data preparation is completed and ends when the inference result is obtained, and is measured in milliseconds (ms). If inference is performed using the asynchronous UCP interface, latency statistics are measured from the call to hbDNNInferV2 until the return of hbUCPWaitTaskDone.
• FPS represents the number of inference frames the system can process per unit time, reflecting the overall throughput of the system, and is measured in frames per second.

Accuracy evaluation is used to assess the correctness and consistency of model inference outputs in the deployment environment.

The reference model for accuracy evaluation is the fixed-point model generated during the quantization stage. The on-board inference results should be consistent with the outputs of this model. The reference artifact is quantized_model.bc (or quantized_removed_model.bc if the input/output nodes were removed during quantization).

If the on-board result does not match the reference artifact, first check the input data preparation and the alignment handling rather than suspecting the model itself. For the investigation method, refer to the ONNX Model Deployment Consistency Analysis or PyTorch Model Deployment Consistency Analysis section.

We provide the hrt_model_exec tool and the hbm_infer tool to facilitate model inference performance and accuracy evaluation.

#Choosing Between the Tools

hrt_model_exec and hbm_infer differ in where they run and which scenarios they suit. Use the table below to choose:

If you only need to confirm that the model produces a correct single-frame result on the board, or you need Latency / FPS metrics, hrt_model_exec is enough. If you need to verify accuracy over a dataset, hbm_infer is the better choice.

#Evaluation Using hrt_model_exec Tool

hrt_model_exec is a model execution tool that can be used directly on the development board to obtain model information, perform single-frame inference, and evaluate model inference performance. The source code for the tool is located at samples/ucp_tutorial/tools/hrt_model_exec. The tool’s environment setup and three example usage scenarios are as follows:

# Set the environment variables, where arch represents the architecture type.
arch=aarch64
bin=../${arch}/bin/hrt_model_exec
lib=../${arch}/lib/
export LD_LIBRARY_PATH=${lib}:${LD_LIBRARY_PATH}

# Obtain Model Information
${bin} model_info --model_file=xxx.hbm

# Single-Frame Model Inference (Inference result files are available for accuracy evaluation)
${bin} infer --model_file=xxx.hbm --input_file=xxx.bin --enable_dump=true

# Model Performance Evaluation – Latency (Single-Threaded)
${bin} perf --model_file=xxx.hbm --thread_num 1 --frame_count=1000

# Model Performance Evaluation – FPS (Multi-Threaded, thread count adjustable based on actual conditions)
${bin} perf --model_file=xxx.hbm --thread_num 8 --frame_count=1000

For a more detailed introduction to the hrt_model_exec tool, please refer to Model Inference Tools - The hrt_model_exec Tool.

#Evaluation Using hbm_infer Tool

To improve the efficiency of accuracy evaluation, the toolchain provides the hbm_infer tool, which allows you to perform model pre- and post-processing on the X86 side using Python code, while the actual inference is executed by the board-side server. It is better suited for evaluation on large datasets. A basic single-process usage example is as follows:

# Create Session
sess = HbmRpcSession(host="xx.xx.xx.xx", local_hbm_path="xx.hbm", with_profile=True)

# Print Model Info
sess.show_input_output_info()

# Prepare Input Data
input_data = {'img': torch.ones((1, 3, 224, 224), dtype=torch.int8)}

# Perform Inference and Return Results
output_data = sess(input_data)

# Use the inference results for post-processing or accuracy evaluation.

# If with_profile=True is set, timing statistics for each stage of frame inference can be obtained, which can be used to evaluate single-frame latency.
print(sess.get_profile())

# Close Server
sess.close_server()

For a more detailed introduction to multi-process evaluation, model chaining, and transmission optimization using the hbm_infer tool, please refer to Model Inference Tools - The hbm_infer Tool.

#Where to Look Next

• Performance metrics fall short of expectations: refer to the Model Performance Optimization section.

• The on-board accuracy does not match the reference artifact: refer to the ONNX Model Deployment Consistency Analysis or PyTorch Model Deployment Consistency Analysis section.

• You need to confirm hardware IP utilization and memory usage: refer to the Board Resources Evaluation section.