#FAQ

#General Issues

#What is the reason for returning HB_UCP_INVALID_ARGUMENT error code after task creation or submission?

Possible problems can be determined from the UCP error logs, which may exist as follows:

• Operator Constraint Problem: Most acceleration operators should satisfy usage constraints when they are created, otherwise they will return error code.

• If you encounter a log that prints op $1 of task has no proper backend, user expect $2, indicating that there is no proper backend available for execution, where $1 indicates the type of the task and $2 is the backend parameter when the task is submitted in binary form, and the configuration needs to be done according to the number of cores that each J6 series backend can support.

#How to understand the physical and virtual addresses of hbUCPSysMem?

In the J6 processor architecture, all hardware share the DDR memory, and a physically contiguous section of memory can be requested through the hbUCPMallocCached and hbUCPMalloc interfaczes.

The return values of these functions are wrapped in the hbUCPSysMem data structure, and the phyAddr and virAddr fields correspond to the physical and virtual addresses of its memory space, respectively.

As this memory space is contiguous, both physical and virtual addresses can be represented, read, and written by the first address. However, in practice, it is recommended to use virtual addresses in preference in non-essential scenarios.

#How to understand cacheable and non-cacheable hbmem?

UCP's memory management interface provides hbUCPMallocCached and hbUCPMalloc to allocate DDR read/write memory, which is physically contiguous and can be accessed and used by bpu/dsp and other ip accesses. Where hbUCPMallocCached represents the allocation of memory for the cacheable attribute and is accompanied by the hbUCPMemFlush function to refresh the Cache.

The cache mechanism is determined by the memory architecture of the platform, as shown in the following figure. The cache between CPU and memory is used as a data cache, however, there is no cache between the BPU/DSP/JPU/VPU(Video Process Unit)/PYRAMID/STITCH/GDC backend hardware and main memory. Therefore, the misuse of the cache can cause problems in data reading/writing accuracy and efficiency.

• When the CPU has finished writing data, it needs to actively flush the data in the cache to the memory, otherwise other hardware accessing the same memory space may read the old data from before.

• The CPU also needs to actively INVALIDATE the data in the Cache before accessing it when the other backend hardware has finished writing the data, otherwise the CPU will preferentially read the old data previously cached.

• In the continuous inference process of the model, those that need to be read by the cpu, such as the model output, it is recommended to apply for memory with cacheable to accelerate the efficiency of the CPU to read and write repeatedly, while those that don't need to be read, and are only written, such as the model input, can apply for non-cacheable memory.

#Can exit be used in exception handling on the application side?

Due to the existence of a resource management singleton within UCP, calling exit/std::exit on the application side (including in overridden signal handlers) will trigger the singleton's destructor. If there are still threads accessing the singleton's resources at that time, it can easily lead to exceptions. Therefore, UCP recommends using _exit/std::quick_exit. If you must use exit/std::exit, please ensure that no threads are accessing any UCP resources when exiting.

#How to ensure log integrity when application exits abnormally?

To ensure that critical log messages are not lost due to abnormal process termination, it is recommended that the application calls the hbUCPFlushLog interface before abnormal exit to flush the buffered logs and ensure that all log messages are written to disk. This is particularly important for debugging anomalies and tracing system failures.

#Model Inference

#What are the possible causes of the timeout on the hbUCPWaitTaskDone interface for model reasoning?

• The model itself takes a long time to execute, and an insufficient timeout set by the asynchronous wait interface, or a long queuing time for the task due to a high load on the current computational resources, may trigger an interface timeout.

• Memory leaks exist. Slow allocation of memory in case of insufficient system memory may lead to inference timeout.

• CPU load is too high. The scheduling thread can't get CPU, at this time, even if the task is completed, it can't be synchronized to the user interface in time, which leads to the reasoning timeout situation.

#Reasons why model inference gets stuck

Model problem: The underlying running error caused by the model command reason, the error is not reported, resulting in hang. At this point, the bpu task situation can be viewed by cat /sys/devices/system/bpu/bpu0/task_running, as shown below:

If s_time is not null, it means that the task has started normally, while p_time is null, it means that it has not returned normally, which can be assumed that the BPU task hangs, and can be solved by contacting sr or the compiler team.

#What are the ROI input model constraints?

You can refer to the ROI Introduction and Constraints for the introduction of ROI constraints.

#Troubleshooting model inference result consistency issues

1. Please ensure that the outputs used for consistency comparison have had padding removed.

2. If the model has dynamic-shaped outputs, please ensure that the hbDNNGetTaskOutputTensorProperties API is called to obtain the actual output shape.

3. Please ensure that Flush operations are correctly performed on both input and output memory.

4. If the consistency issue is resolved after disabling UCP ARM optimized operators, then the issue lies with the optimized operators. Method to disable:

export _HB_NN_DISABLE_UDE_EXTENSION_=true

#Model inference performance optimization recommendations

1. Try to eliminate CPU operators in the model as much as possible.

2. It is recommended to allocate inference memory in a unified manner, use it in a distributed way, and enable memory reuse.

3. On the premise of implementing input/output memory reuse for the model, enable the Map LRU Cache strategy to mitigate BPU Map overhead. Method to enable:

export HB_NN_ENABLE_MEM_LRU_CACHE=true

4. Adjust thread priorities. It is recommended to increase the priority of scheduling threads. If only model inference is used, it is recommended to disable unused backends such as DSP and ISP to reduce the number of threads. Reference example:

# Set the UCP scheduling thread priority to the highest level
export HB_UCP_SCHEDULE_PRIORITY=99
# Disable unused backends
export HB_UCP_ENABLE_DSP_BACKEND_CORE_NUM=0
export HB_UCP_ENABLE_ISP_BACKEND_CORE_NUM=0

5. Specify the BPU core ID during inference to reduce the number of Memory Map operations on multi-core hardware.

#Custom OP Development

#DSP Related

#What are the DSP models selected for the J6?

The J6 uses Cadence's Tensilica Vision Q8 DSP IP (hereinafter referred to as Q8), a digital signal processor dedicated to vision/image processing, with the number of DSP IPs varying slightly depending on the development board model. More information can be found: DSP Development Documentation or Cadence Official Documentation。

#How do to get the official Cadence documentation?

Installation steps can be referred to: DSP Development Documentation section; After installing the Cadence Development Kit, you can view some of the documentation within the development package. To obtain the complete documentation package, please contact Horizon technical support staff.

#What computational accuracy does the DSP support?

Support int8/int16/int32 integer calculations, as well as float32 and double floating point calculations.

#How to view the log output on the DSP side?

In the X86 simulation environment, you can view the log output by modifying the log printing level environment variable HB_DSP_VDSP_LOG_LEVEL when executing the sample script, and the method of setting the log level is the same as that of the UCP, and the modification method, please refer to the contents of the script;

In the development board, you can view the logs on the DSP side by monitoring the log file. The specific method is as follows:

1. Add the environment variable export HB_DSP_WRITE_VDSP_LOG_TO_ARM=true to enable DSP log output.

export HB_DSP_VDSP_LOG_LEVEL=3
export HB_DSP_WRITE_DSP_LOG_TO_ARM=true
export HB_DSP_ENABLE_CONFIG_VDSP=true

2. Modify the DSP log printing level environment variable HB_DSP_VDSP_LOG_LEVEL.

3. Use the following command to enable file monitoring.

hrut_remoteproc_log -b /sys/class/remoteproc/remoteproc1/log

#Can the DSP custom operator samples be run directly?

No. Using custom operators requires recompiling the DSP image before registering the operator implementation into the DSP calling framework. Therefore, the DSP image need to be compile first, then copy the image to the development board, and then run the example. For the compilation and running steps, please refer to: Custom Operator Samples section.

However, for code development or compiling new images, the DSP development software Xplorer and the corresponding License file are required, which can be obtained by contacting Horizon technical support.

#What are the commonly used DSP debugging commands?

1. Obtain the status of the vdsp core, which can be either "offline" or "online".

# get vdsp core0 state
root@hobot:~# cat /sys/class/remoteproc/remoteproc1/state
offline
# get vdsp core1 state in J6P
root@hobot:~# cat /sys/class/remoteproc/remoteproc2/state
offline

2. Obtain the vdsp image name and version. The version contains information such as the compilation version, compilation date, hash_id, etc., which can be used for version tracing.

# get vdsp core0 name
root@hobot:~# cat /sys/class/remoteproc/remoteproc1/name
soc:remoteproc_vdsp0
# get vdsp core0 version
root@hobot:~# cat /sys/class/remoteproc/remoteproc1/version
version = VDSP0_V1.4.17_20251210144524_debug
compile_time = 20251210144524
git_hash_id = eccb27563e71f9f43585cadf7ddc0e9da0d1af1
# get vdsp core1 name in J6P
root@hobot:~# cat /sys/class/remoteproc/remoteproc2/name
soc:remoteproc_vdsp1
# get vdsp core1 version in J6P
root@hobot:~# cat /sys/class/remoteproc/remoteproc2/version
version = VDSP0_V1.4.17_20251210144524_debug
compile_time = 20251210144524
git_hash_id = eccb27563e71f9f43585cadf7ddc0e9da0d1af1

3. Use the dmesg command to view low-level software log information.

4. Check the latest vdsp logs written to disk.

# get dsp core0 log
root@hobot:~# cat /log/dsp0/message
# get dsp core1 log in J6P
root@hobot:~# cat /log/dsp1/message

5. DSP start/stop commands.

# echo dsp image path to firmware path
echo -n $image_path > /sys/module/firmware_class/parameters/path
# set firmware name of dsp core0
echo vdsp0 > /sys/class/remoteproc/remoteproc1/firmware
# start dsp core0
echo start > /sys/class/remoteproc/remoteproc1/state
# stop dsp core0
echo stop > /sys/class/remoteproc/remoteproc1/state
# set firmware name of dsp core1 in J6P
echo vdsp1 > /sys/class/remoteproc/remoteproc2/firmware
# start dsp core1 in J6P
echo start > /sys/class/remoteproc/remoteproc2/state
# stop dsp core1 in J6P
echo stop > /sys/class/remoteproc/remoteproc2/state

#How to Troubleshoot DSP Custom Operator Hangs?

1. Check DSP logs for any error messages.

2. Determine if the DSP is hung or has coredumped by using the dmesg command to view kernel logs.

3. Investigate ARM-side code for potential memory Map/Unmap or cache refresh issues.

4. Examine DSP-side code for cache refresh problems, using methods like global refresh or binary search to locate faults.

5. Check DSP-side code for logical or memory access errors, using dmesg to view kernel log messages.

#What Issues Can DDR Cache Desynchronization Cause in DSP Custom Operators?

1. Data inconsistency between ARM and DSP read/write operations.

2. DSP freezes or becomes unresponsive, potentially due to using unrefreshed memory.

3. DSP coredumps with error logs in kernel messages (dmesg), which can help locate specific fault addresses.