#UCP Trace Instructions

UCP Trace provides the ability to in-depth analysis of the scheduling logic of UCP applications by embedding trace recording on the critical path executed by UCP. When performance anomalies occur, it can quickly locate the time point of the anomaly by analyzing UCP trace.

UCP trace provides two trace backend options: Perfetto Trace and Chrome Trace. You can choose between them by setting an environment variable to meet your specific performance tracking needs.

• Perfetto trace can retrieve ucp recorded traces, as well as system status, ftrace information, etc.

• Chrome trace can only retrieve ucp recorded traces and is mainly used to analyze UCP's scheduling logic.

#Basic Usage and Results

#Perfetto Trace

Perfetto is a performance analysis tool developed and open-sourced by Google. It can collect performance data from different data sources and provides Perfetto UI for visualization and analysis. For more details about Perfetto, refer to the official Perfetto documentation.

#in_process Mode Example

In in_process mode, only trace information inside the UCP process can be captured, and no Perfetto background process is required. The steps are as follows:

1. Configure environment variables.

   # Specify the UCP Perfetto config path.
   export HB_UCP_PERFETTO_CONFIG_PATH=ucp_in_process.json
   
   # Enable Perfetto.
   export HB_UCP_ENABLE_PERFETTO=true

2. Run the UCP application. hrt_model_exec is used here as an example.

./hrt_model_exec perf                      \
    --model_file resnet50_224x224_nv12.hbm \
    --frame_count 1000                     \
    --thread_num 8

3. The generated trace is saved as ucp.pftrace, which you can open with Perfetto UI.

   

4. Click a created task on the timeline to view the complete scheduling flow from task creation to task release.

   

5. Common Perfetto UI operations are listed below. For more details, refer to the help page.

   Operation	Description
   w or ctrl + mouse wheel up	Zoom in
   s or ctrl + mouse wheel down	Zoom out
   a or drag the top timeline left	Move left
   d or drag the top timeline right	Move right
   ?	Open help page

#System Mode Example

In system mode, UCP Trace is only one of the data sources, so you need to run the corresponding tracebox commands to complete trace capture. The steps are as follows:

1. Start the Perfetto background processes.

   # Start the trace service.
   # This only needs to be started once. If it is already running, you do not need to start it again.
   tracebox traced --background
   
   # Start the data capture service.
   # This only needs to be started once. If it is already running, you do not need to start it again.
   tracebox traced_probes --background --reset-ftrace

2. Trigger data capture.

   # -c: specifies the Perfetto config file.
   # -o: specifies the output trace data file path.
   tracebox perfetto --txt -c ucp_system.cfg -o ucp.pftrace

3. Open another terminal and configure UCP environment variables.

   # If HB_UCP_PERFETTO_CONFIG_PATH is not specified, system mode is used by default.
   # If it does not take effect, the current version may not support automatic selection,
   # and you need to explicitly specify the system config file.
   export HB_UCP_PERFETTO_CONFIG_PATH=ucp_system.json
   
   # Enable Perfetto.
   export HB_UCP_ENABLE_PERFETTO=true

4. Run the UCP application. hrt_model_exec is used here as an example.

./hrt_model_exec perf                      \
    --model_file resnet50_224x224_nv12.hbm \
    --frame_count 1000                     \
    --thread_num 8

5. The generated trace is saved to the output file ucp.pftrace specified by the perfetto command, and you can open it with Perfetto UI.

   

#BPU Trace Example

To demonstrate BPU trace during multi-model inference, a multi-process application example is provided here. Except for the different programs being launched, the remaining steps are the same as in the previous section.

./hrt_model_exec perf                        \
    --model_file resnet50_224x224_nv12.hbm   \
    --frame_count 50                         \
    --thread_num 1 &                         \

./hrt_model_exec perf                        \
    --model_file googlenet_224x224_nv12.hbm  \
    --frame_count 50                         \
    --thread_num 1

BPU Trace visualization requires the Horizon-customized hbperfetto tool. You can obtain this tool by contacting Horizon system software technical support. The following figure shows the effect after opening the trace file with hbperfetto:

The following figure shows the scheduling status of different model inference tasks presented in BPU Trace:

hbperfetto supports correlation between UCP Trace and BPU Trace. The following figure shows the complete flow from creation, submission, and scheduled execution of a UCP model inference task to task completion and final release.

In addition, you can query the raw BPU trace data through SQL.

select * from bpu_trace

#Chrome Trace

Chrome Trace supports capturing only UCP Trace and does not support multi-source capture. If you need to capture multiple data sources, use Perfetto Trace. Chrome Trace is simple and easy to use. It records traces in text logs and does not rely on additional third-party libraries or tools. If you only care about UCP scheduling logic, you can use Chrome Trace for capture.

#Usage Example

1. Configure environment variables.

   # Disable Perfetto.
   export HB_UCP_ENABLE_PERFETTO=false
   
   # Set the UCP Trace log level to 0.
   export HB_UCP_TRACE_LOG_LEVEL=0
   
   # Set the path for saving UCP logs.
   export HB_UCP_LOG_PATH=ucp_log.txt

2. Run the UCP application. hrt_model_exec is used here as an example.

   ./hrt_model_exec perf                      \
       --model_file resnet50_224x224_nv12.hbm \
       --frame_count 1000                     \
       --thread_num 8

3. Run the trace capture script.

   After trace log capture is complete, run catch_trace.sh provided in the UCP release package to convert the raw trace log into a JSON-format trace file.

   # -i: specifies the input trace log file.
   # -o: specifies the output JSON trace log file.
   ./catch_trace.sh -i ucp_log.txt -o ucp_trace_task.json
   
   # By default, the trace is visualized from the task perspective.
   # You can also switch to the thread perspective.
   # -m: specifies the conversion mode. task: task perspective (default), thread: thread perspective.
   ./catch_trace.sh -m thread -i ucp_log.txt -o ucp_trace_thread.json

4. Open ucp_trace_task.json and ucp_trace_thread.json with Perfetto UI.

   Perfetto UI opening ucp_trace_thread.json:

   

#Configuration and Tooling

#Tool and Configuration File Locations

UCP Trace tools and configuration files are located in samples/ucp_tutorial/tools. The directory structure is as follows:

  tools/
  └── trace                          # trace tools
    ├── catch_trace.sh               # chrome trace capture script
    ├── configs                      # reference configuration files
    │   ├── ucp_bpu_trace.cfg        # Perfetto config file for enabling BPU trace
    │   ├── ucp_dsp_trace.cfg        # Perfetto config file for enabling DSP trace
    │   ├── ucp_in_process.cfg       # Perfetto config file for in_process mode
    │   ├── ucp_in_process.json      # UCP config file for in_process mode
    │   ├── ucp_system.cfg           # Perfetto config file for system mode
    │   └── ucp_system.json          # UCP config file for system mode

#Environment Variables

#Configuration File Parameters

#UCP Trace Configuration File Parameters

#Long Trace Configuration File Parameters

By default, Perfetto stores trace data in an in-memory buffer and dumps the buffered data to a file only when the trace session ends. If the volume of trace data exceeds the buffer capacity, data integrity cannot be guaranteed.

Perfetto supports periodically writing buffered data to a file. You only need to add the following fields to the trace configuration file.

#Configuration File Templates

#UCP Trace Configuration File Template

Used to configure how UCP uses Perfetto. You can specify it through the environment variable HB_UCP_PERFETTO_CONFIG_PATH.

in_process Mode

{
    "backend": "in_process",
    "trace_config": "ucp_in_process.cfg"
}

System Mode

{
    "backend": "system"
}

#Perfetto Configuration File Template

For details about Perfetto configuration files, refer to the Perfetto TraceConfig Reference. UCP provides reference configuration files ucp_in_process.cfg and ucp_system.cfg, which you can modify according to your application scenario.

# Enable periodic flushing of the trace buffer into the output file.
write_into_file: true

# Output file path
output_path: "ucp.pftrace"

# Sampling duration: 10s
duration_ms: 10000

# Writes the userspace buffer into the file every 2.5 seconds.
file_write_period_ms: 2500

buffers {
  # buffer size
  size_kb: 65535
  # DISCARD: no new sampling data will be stored when the storage is full.
  # RING_BUFFER: old sampling data will be discarded and new data will be stored when the storage is full.
  fill_policy: RING_BUFFER
}

# UCP data source
data_sources: {
    config {
        name: "track_event"
        track_event_config {
           enabled_categories: "dnn"
        }
    }
}

#BPU Trace Configuration File Template

In system mode, BPU trace capture is supported. You only need to add the BPU trace data source to the Perfetto configuration file. The ucp_bpu_trace.cfg file already includes the BPU trace data source by default, and the relevant configuration is shown below:

data_sources: {
    config {
        name: "linux.sys_stats"
        sys_stats_config {
            bputrace_period_ms: 500
        }
    }
}

The bputrace_period_ms is used to set the interval for reading BPU Trace. You can adjust this parameter according to the actual usage scenario. When the BPU load is high, you can appropriately shorten the interval to avoid trace data being overwritten due to mismatched read and write speeds.

#DSP Trace Configuration File Template

In system mode, DSP trace capture is supported. You only need to add the DSP trace data source to the Perfetto configuration file. The ucp_dsp_trace.cfg file already includes the DSP trace data source by default, and the valid configuration is shown below:

data_sources {
  config {
    name: "linux.sys_stats"
    sys_stats_config {
      dsptrace_period_ms: 500
    }
  }
}

dsptrace_period_ms is used to set the interval for reading DSP Trace. You can adjust this parameter according to the actual usage scenario. When the DSP load is high, you can appropriately shorten the interval to avoid trace data being overwritten due to mismatched read and write speeds.

# stop vdsp fw
echo stop > /sys/class/remoteproc/remoteproc1/state

# stop or start dsp trace
echo 0 > /sys/devices/virtual/misc/vdsp0/vdsp_ctrl/trace_switch
echo 1 > /sys/devices/virtual/misc/vdsp0/vdsp_ctrl/trace_switch

# check dsp trace status
cat /sys/devices/virtual/misc/vdsp0/vdsp_ctrl/trace_switch

#UCP Trace Record Points

UCP embeds trace records in application APIs and internal key scheduling paths, including task trace records and operator trace records.

#Task Trace Record Points

#Operator Trace Record Points

#Common Data Sources