#UCP Trace Instructions

UCP Trace provides the ability to in-depth analysis of the scheduling logic of UCP applications by embedding trace recording on the critical path executed by UCP. When performance anomalies occur, it can quickly locate the time point of the anomaly by analyzing UCP trace.

UCP trace provides two trace backend options: Perfetto Trace and Chrome Trace. You can choose between them by setting an environment variable to meet your sepecific performance tracking needs.

• Perfetto trace can retrieve ucp recorded traces, as well as system status, ftrace information, etc.
• Chrome trace can only retrieve ucp recoreded traces and is mainly used to analyze UCP's scheduling logic.

The UCP trace tool and configuration files are located in the samples/ucp_tutorial/tools directory path, with a directory structures as follows:

	tools/
	└── trace                          # trace tools
		├── catch_trace.sh               # catch script for chrome trace
		├── configs                      # Reference configuration files
		│   ├── ucp_bpu_trace.cfg        # perfetto configuration file for bpu trace
		│   ├── ucp_in_process.cfg       # perfetto configuration file for in_process mode
		│   ├── ucp_in_process.json      # ucp configuration file for in_process mode
		│   ├── ucp_system.cfg           # perfetto configuration file for system mode
		│   └── ucp_system.json          # ucp configuration file for system mode

#Environmental Variable

#UCP Trace Records

UCP add trace records in the application API and internal critical scheduling paths, including task trace records and operator records.

#Task Trace Records

#Operator Trace Records

#Perfetto Trace

#Overview

Perfetto is a system analysis tool developed and open-sourced by Google, which can collect performance data from different data sources and provides the Perfetto UI for data visualization and analysis. For more details on Perfetto, please refer to the Perfetto offical document.

#Configuration File

#UCP Trace Configuration File

You can configure UCP to use the Perfetto by specifying it through the environment variable HB_UCP_PERFETTO_CONFIG_PATH.

#Parameters Description

#Example 1: in_process mode

{
    "backend": "in_process",
    "trace_config": "ucp_in_process.cfg"
}

#Sample 2: system mode

{
    "backend": "system"
}

#Perfetto Configuration File

For detailed information about perfetto configuration files, please refer to Perfetto TraceConfig Reference. UCP provides reference configuration files ucp_in_process.cfg and ucp_system.cfg, which can be modified based on application scenario.

#Example: ucp_in_process.cfg

# Enable periodic flushing of the trace buffer into the output file.
write_into_file: true

# Output file path
output_path: "ucp.pftrace"

# Sampling duration: 10s
duration_ms: 10000

# Writes the userspace buffer into the file every 2.5 seconds.
file_write_period_ms: 2500

buffers {
  # buffer size
  size_kb: 65535
  # DISCARD: no new sampling data will be stored when the storage is full.
  # RING_BUFFER: old sampling data will be discarded and new data will be stored when the storage is full.
  fill_policy: RING_BUFFER
}

# UCP data source
data_sources: {
    config {
        name: "track_event"
        track_event_config {
           enabled_categories: "dnn"
        }
    }
}

#Example

#in_process mode

In the in_process mode, only trace within UCP process can be captured, and it is not necessary to start the background process of perfetto.

#Operating Procedure

1. Configure environment variables.

# Specify the ucp perfetto configuration path.
export HB_UCP_PERFETTO_CONFIG_PATH=ucp_in_process.json

# Enable perfetto.
export HB_UCP_ENABLE_PERFETTO=true

2. Running the UCP application, using hrt_model_exec as an example.

Due to the specified file path is a relative path, the trace configuration file and scripts need to be placed in the same level directory as the running program. Also, you need to make sure that you configure the environment variables and run the program in the same shell environment.

./hrt_model_exec perf                      \
    --model_file resnet50_224x224_nv12.hbm \
    --frame_count 1000                     \
    --thread_num 8

3. The generated trace is saved in the output file ucp.pftrace specified by the perfetto command, and you can use Perfetto UI to open it.

4. Clicking on the task in the timeline will show the complete scheduling process from creation to release of the task.

5. The common operations of Perfetto UI are as follows, for more detailed operation Instructions, please refer to the help interface.

#system mode

In system mode, UCP trace is just one of the data sources, so it it necessary to run the corresponding commands for tracebox to complete the capture of trace.

#Operating Procedure

1. Running the perfetto background process.

# Start trace service.
# Start once, no need to start it again when it is already started.
tracebox traced --background

# Start data capture service.
# Start once, no need to start it again when it is already started.
tracebox traced_probes --background --reset-ftrace

2. Trigger data capture.

# -c: Specify perfetto configuration file
# -o: Specify output path of trace data.
tracebox perfetto --txt -c ucp_system.cfg -o ucp.pftrace

3. Open a new terminal and configure the UCP environment variables.

# Specify the ucp perfetto configuration path.
export HB_UCP_PERFETTO_CONFIG_PATH=ucp_system.json

# Enable perfetto.
export HB_UCP_ENABLE_PERFETTO=true

4. Running the UCP application, using hrt_model_exec as an example.

To be able to capture complete data, it is necessary to ensure that the perfetto process does not exit before the hrt_model_exec execution is complete.

./hrt_model_exec perf                      \
    --model_file resnet50_224x224_nv12.hbm \
    --frame_count 1000                     \
    --thread_num 8

5. The generated trace is saved in ucp.pftrace, and you can use Perfetto UI to open it.

#BPU Trace

In system mode, BPU trace capture is supported. Simply add BPU trace data source to the perfetto configuration file. The ucp_bpu_trace.cfg file has already defaultly included the BPU trace data source. The specific configuration items are shown below.

data_sources: {
    config {
        name: "linux.sys_stats"
        sys_stats_config {
            bputrace_period_ms: 500
        }
    }
}

The bpu_trace_period_ms is used to set the period for reading the BPU trace. You can adjust this parameter according to your actual usage scenario. When the BPU load is high, you can appropriately shorted the reading period to avoid the problem of trace data being overwritten due to mismatched read and write speeds.

To demonstrate the BPU trace during the inference process of multiple models, an example of a multi-process application is provided here. Except for the different running programs being launched, the rest of the steps are the same as in the previous section.

./hrt_model_exec perf                        \
    --model_file resnet50_224x224_nv12.hbm   \
    --frame_count 50                         \
    --thread_num 1 &                         \

./hrt_model_exec perf                        \
    --model_file googlenet_224x224_nv12.hbm  \
    --frame_count 50                         \
    --thread_num 1                           \

Visualization of BPU trace requires the use of the hbperfetto tool, which is custom-developed by Horizon Robotics. You can obtain this tool by contacting the Horizon Robotics system software technical support personnel. The effect of opening a trace file using hbperfetto is shown in the image below.

The scheduling of different model inference tasks presented in BPU trace is shown in the following figure.

hbperfetto supports the association of UCP trace and BPU trace. The following diagram illustrates the complete process from the creating, submission, scheduling and execution, to the task's completion and eventual releases.

Additionally, you can query the raw data of BPU trace based on SQL.

select * from bpu_trace

#Long Trace Configuration

By default, Perfetto stores trace data in an in-memory buffer until the trace session ends, at which point the data in the buffer is dumped to a file. If the trace data exceeds the buffer capacity, there is no guarantee of data integrity.

Perfetto supports periodically writing buffer data to a file, which can be achieved by adding the following fields to the trace configuration file.

#Introduction to Common Data Sources

#Chrome Trace

Chrome trace only supports capturing UCP trace, and does not support capturing data sources. For capturing multiple data sources, please use Perfetto trace. The characteristic of Chrome trace is simplicity and ease of use, using text logs to record traces without depending on any extra thrid-party libraries or tools. If you are only interested in the scheduling logic of UCP, you can use Chrome trace to capture it.

#Example

1. Configure environment variables.

# Disable perfetto
export HB_UCP_ENABLE_PERFETTO=false

# Set the log level of UCP trace to 0.
export HB_UCP_TRACE_LOG_LEVEL=0

# Set the file path for UCP log saving.
export HB_UCP_LOG_PATH=ucp_log.txt

2. Running the UCP application, using hrt_model_exec as an example.

./hrt_model_exec perf                      \
    --model_file resnet50_224x224_nv12.hbm \
    --thread_num 8

3. Execute trace capture script.

After capturing the trace logs, run the catch_trace.sh provided in the UCP distribution package to convert the raw trace logs into a json-formatted trace file.

# -i: Specify input trace log file
# -o: Specify output json-formatted trace file.
./catch_trace.sh -i ucp_log.txt -o ucp_trace_task.json

# Visualize trace as task view by defalut, but you can switch to thread view as well.
# -m: Specify convert mode，task: task view (default)，thread: thread view.
./catch_trace.sh -m thread -i ucp_log.txt -o ucp_trace_thread.json

4. Open ucp_trace_task.json and ucp_trace_thread.json using Perfetto UI or Chrome UI(chrome://tracing/).

Open ucp_trace_task.json by Chrome UI:

Open ucp_trace_thread.json by Perfetto UI: