#FAQ

#General Issues

#What is the reason for returning HB_UCP_INVALID_ARGUMENT error code after task creation or submission?

Possible problems can be determined from the UCP error logs, which may exist as follows:

• Operator Constraint Problem: Most acceleration operators should satisfy usage constraints when they are created, otherwise they will return error code.

• If you encounter a log that prints op $1 of task has no proper backend, user expect $2, indicating that there is no proper backend available for execution, where $1 indicates the type of the task and $2 is the backend parameter when the task is submitted in binary form, and the configuration needs to be done according to the number of cores that each J6 series backend can support.

#How to understand the physical and virtual addresses of hbUCPSysMem?

In the J6 processor architecture, all hardware share the DDR memory, and a physically contiguous section of memory can be requested through the hbUCPMallocCached and hbUCPMalloc interfaczes.

The return values of these functions are wrapped in the hbUCPSysMem data structure, and the phyAddr and virAddr fields correspond to the physical and virtual addresses of its memory space, respectively.

As this memory space is contiguous, both physical and virtual addresses can be represented, read, and written by the first address. However, in practice, it is recommended to use virtual addresses in preference in non-essential scenarios.

#How to understand cacheable and non-cacheable hbmem?

UCP's memory management interface provides hbUCPMallocCached and hbUCPMalloc to allocate DDR read/write memory, which is physically contiguous and can be accessed and used by bpu/dsp and other ip accesses. Where hbUCPMallocCached represents the allocation of memory for the cacheable attribute and is accompanied by the hbUCPMemFlush function to refresh the Cache.

The cache mechanism is determined by the memory architecture of the platform, as shown in the following figure. The cache between CPU and memory is used as a data cache, however, there is no cache between the BPU/DSP/JPU/VPU(Video Process Unit)/PYRAMID/STITCH/GDC backend hardware and main memory. Therefore, the misuse of the cache can cause problems in data reading/writing accuracy and efficiency.

• When the CPU has finished writing data, it needs to actively flush the data in the cache to the memory, otherwise other hardware accessing the same memory space may read the old data from before.

• The CPU also needs to actively INVALIDATE the data in the Cache before accessing it when the other backend hardware has finished writing the data, otherwise the CPU will preferentially read the old data previously cached.

• In the continuous inference process of the model, those that need to be read by the cpu, such as the model output, it is recommended to apply for memory with cacheable to accelerate the efficiency of the CPU to read and write repeatedly, while those that don't need to be read, and are only written, such as the model input, can apply for non-cacheable memory.

#Can exit be used in exception handling on the application side?

Due to the existence of a resource management singleton within UCP, calling exit/std::exit on the application side (including in overridden signal handlers) will trigger the singleton's destructor. If there are still threads accessing the singleton's resources at that time, it can easily lead to exceptions. Therefore, UCP recommends using _exit/std::quick_exit. If you must use exit/std::exit, please ensure that no threads are accessing any UCP resources when exiting.

#How to ensure log integrity when application exits abnormally?

To ensure that critical log messages are not lost due to abnormal process termination, it is recommended that the application calls the hbUCPFlushLog interface before abnormal exit to flush the buffered logs and ensure that all log messages are written to disk. This is particularly important for debugging anomalies and tracing system failures.

#Model Inference

#What are the possible causes of the timeout on the hbUCPWaitTaskDone interface for model reasoning?

• The model itself takes a long time to execute, and an insufficient timeout set by the asynchronous wait interface, or a long queuing time for the task due to a high load on the current computational resources, may trigger an interface timeout.

• Memory leaks exist. Slow allocation of memory in case of insufficient system memory may lead to inference timeout.

• CPU load is too high. The scheduling thread can't get CPU, at this time, even if the task is completed, it can't be synchronized to the user interface in time, which leads to the reasoning timeout situation.

#Reasons why model inference gets stuck

Model problem: The underlying running error caused by the model command reason, the error is not reported, resulting in hang. At this point, the bpu task situation can be viewed by cat /sys/devices/system/bpu/bpu0/task_running, as shown below:

If s_time is not null, it means that the task has started normally, while p_time is null, it means that it has not returned normally, which can be assumed that the BPU task hangs, and can be solved by contacting sr or the compiler team.

#What are the ROI input model constraints?

You can refer to the ROI Introduction and Constraints for the introduction of ROI constraints.

#Troubleshooting model inference result consistency issues

1. Please ensure that the outputs used for consistency comparison have had padding removed.

2. If the model has dynamic-shaped outputs, please ensure that the hbDNNGetTaskOutputTensorProperties API is called to obtain the actual output shape.

3. Please ensure that Flush operations are correctly performed on both input and output memory.

4. If the consistency issue is resolved after disabling UCP ARM optimized operators, then the issue lies with the optimized operators. Method to disable:

export _HB_NN_DISABLE_UDE_EXTENSION_=true

#Model inference performance optimization recommendations

1. Try to eliminate CPU operators in the model as much as possible.

2. It is recommended to allocate inference memory in a unified manner, use it in a distributed way, and enable memory reuse.

3. On the premise of implementing input/output memory reuse for the model, enable the Map LRU Cache strategy to mitigate BPU Map overhead. Method to enable:

export HB_NN_ENABLE_MEM_LRU_CACHE=true

4. Adjust thread priorities. It is recommended to increase the priority of scheduling threads. If only model inference is used, it is recommended to disable unused backends such as DSP and ISP to reduce the number of threads. Reference example:

# Set the UCP scheduling thread priority to the highest level
export HB_UCP_SCHEDULE_PRIORITY=99
# Disable unused backends
export HB_UCP_ENABLE_DSP_BACKEND_CORE_NUM=0
export HB_UCP_ENABLE_ISP_BACKEND_CORE_NUM=0

5. Specify the BPU core ID during inference to reduce the number of Memory Map operations on multi-core hardware.

#Custom OP Development

#DSP Related

#What are the DSP models selected for the J6?

The J6 uses Cadence's Tensilica Vision Q8 DSP IP (hereinafter referred to as Q8), a digital signal processor dedicated to vision/image processing, with the number of DSP IPs varying slightly depending on the development board model. More information can be found: DSP Development Documentation or Cadence Official Documentation。

#How do to get the official Cadence documentation?

Installation steps can be referred to: DSP Development Documentation section; After installing the Cadence Development Kit, you can view some of the documentation within the development package. To obtain the complete documentation package, please contact Horizon technical support staff.

#What computational accuracy does the DSP support?

Support int8/int16/int32 integer calculations, as well as float32 and double floating point calculations.

#How to view the log output on the DSP side?

In the X86 simulation environment, you can view the log output by modifying the log printing level environment variable HB_DSP_VDSP_LOG_LEVEL when executing the sample script, and the method of setting the log level is the same as that of the UCP, and the modification method, please refer to the contents of the script;

In the development board, you can view the logs on the DSP side by monitoring the log file. The specific method is as follows:

1. Add the environment variable export HB_DSP_WRITE_VDSP_LOG_TO_ARM=true to enable DSP log output.

export HB_DSP_VDSP_LOG_LEVEL=3
export HB_DSP_WRITE_DSP_LOG_TO_ARM=true
export HB_DSP_ENABLE_CONFIG_VDSP=true

2. Modify the DSP log printing level environment variable HB_DSP_VDSP_LOG_LEVEL.

3. Use the following command to enable file monitoring.

hrut_remoteproc_log -b /sys/class/remoteproc/remoteproc1/log

#Can the DSP custom operator samples be run directly?

No. Using custom operators requires recompiling the DSP image before registering the operator implementation into the DSP calling framework. Therefore, the DSP image need to be compile first, then copy the image to the development board, and then run the example. For the compilation and running steps, please refer to: Custom Operator Samples section.

However, for code development or compiling new images, the DSP development software Xplorer and the corresponding License file are required, which can be obtained by contacting Horizon technical support.

#What are the commonly used DSP debugging commands?

1. Obtain the status of the vdsp core, which can be either "offline" or "online".

# get vdsp core0 state
root@hobot:~# cat /sys/class/remoteproc/remoteproc1/state
offline
# get vdsp core1 state in J6P
root@hobot:~# cat /sys/class/remoteproc/remoteproc2/state
offline

2. Obtain the vdsp image name and version. The version contains information such as the compilation version, compilation date, hash_id, etc., which can be used for version tracing.

# get vdsp core0 name
root@hobot:~# cat /sys/class/remoteproc/remoteproc1/name
soc:remoteproc_vdsp0
# get vdsp core0 version
root@hobot:~# cat /sys/class/remoteproc/remoteproc1/version
version = VDSP0_V1.4.17_20251210144524_debug
compile_time = 20251210144524
git_hash_id = eccb27563e71f9f43585cadf7ddc0e9da0d1af1
# get vdsp core1 name in J6P
root@hobot:~# cat /sys/class/remoteproc/remoteproc2/name
soc:remoteproc_vdsp1
# get vdsp core1 version in J6P
root@hobot:~# cat /sys/class/remoteproc/remoteproc2/version
version = VDSP0_V1.4.17_20251210144524_debug
compile_time = 20251210144524
git_hash_id = eccb27563e71f9f43585cadf7ddc0e9da0d1af1

3. Use the dmesg command to view low-level software log information.

4. Check the latest vdsp logs written to disk.

# get dsp core0 log
root@hobot:~# cat /log/dsp0/message
# get dsp core1 log in J6P
root@hobot:~# cat /log/dsp1/message

5. DSP start/stop commands.

# echo dsp image path to firmware path
echo -n $image_path > /sys/module/firmware_class/parameters/path
# set firmware name of dsp core0
echo vdsp0 > /sys/class/remoteproc/remoteproc1/firmware
# start dsp core0
echo start > /sys/class/remoteproc/remoteproc1/state
# stop dsp core0
echo stop > /sys/class/remoteproc/remoteproc1/state
# set firmware name of dsp core1 in J6P
echo vdsp1 > /sys/class/remoteproc/remoteproc2/firmware
# start dsp core1 in J6P
echo start > /sys/class/remoteproc/remoteproc2/state
# stop dsp core1 in J6P
echo stop > /sys/class/remoteproc/remoteproc2/state

#How to Troubleshoot DSP Custom Operator Hangs?

1. Check DSP logs for any error messages.

2. Determine if the DSP is hung or has coredumped by using the dmesg command to view kernel logs.

3. Investigate ARM-side code for potential memory Map/Unmap or cache refresh issues.

4. Examine DSP-side code for cache refresh problems, using methods like global refresh or binary search to locate faults.

5. Check DSP-side code for logical or memory access errors, using dmesg to view kernel log messages.

#What Issues Can DDR Cache Desynchronization Cause in DSP Custom Operators?

1. Data inconsistency between ARM and DSP read/write operations.

2. DSP freezes or becomes unresponsive, potentially due to using unrefreshed memory.

3. DSP coredumps with error logs in kernel messages (dmesg), which can help locate specific fault addresses.

#How to Further Optimize Performance of DSP VP/HPL Operators?

Currently, each submission of a VP/HPL DSP operator is treated as an independent task and needs to go through address mapping, task scheduling, RPC communication, address unmapping, and other procedures. In scenarios where a large number of operators are repeatedly invoked, the overhead of address mapping and unmapping can become significant. To optimize performance in such cases, the following methods can be used:

1. For reusing the same address for input and output, use the hbDSPAddrMap interface for address pre-mapping. Repeated mapping only increases the reference count and does not actually trigger the underlying software address mapping logic, reducing the overhead of map and unmap operations in the underlying software for each VP/HPL submission. After all operations are completed, make sure to call hbDSPAddrUnmap in time to release the mapped address.

#include "hb_dsp_addr_map.h"
void repeat_process_sample(){
  // prepare memory
  hbUCPSysMem input_mem, output_mem;
  hbUCPMallocCached(&input_mem, 1920 * 1080, 0);
  hbUCPMallocCached(&output_mem, 1920 * 1080, 0);
  // pre map
  hbDSPAddrMap(&input_mem, &input_mem);
  hbDSPAddrMap(&output_mem, &output_mem);
  // multiply process operator
  int32_t process_count = 100;
  for(int i = 0; i < process_count; i++){
    hbVPImage src{HB_VP_IMAGE_FORMAT_Y,
                HB_VP_IMAGE_TYPE_U8C1,
                1920,
                1080,
                1920,
                input_mem.virAddr,
                input_mem.phyAddr,
                0,
                0,
                0};
    hbVPImage dst{HB_VP_IMAGE_FORMAT_Y,
                HB_VP_IMAGE_TYPE_U8C1,
                1920,
                1080,
                1920,
                output_mem.virAddr,
                output_mem.phyAddr,
                0,
                0,
                0};
    hbVPInterpolationType interp = HB_VP_INTER_LINEAR;
    // run operator
    hbVPResize(nullptr, &dst, &src, interp);
    // post process logic
  }
  // post unmap
  hbDSPAddrUnmap(&input_mem);
  hbDSPAddrUnmap(&output_mem);
}

2. Fuse VP/HPL operators with other operators through custom operators to reduce the number of operator invocations, thereby reducing the overhead of map and unmap operations.

int dsp_custom_operator(void *spec, void *non, void *tm){
  // get operator spec, and split to resize and flip
  spec_resize = spec.resize;
  spec_flip = spec.flip;
  // process resize
  hb_dsp_resize(spec_resize, nullptr, tm);
  // process flip
  hb_dsp_flip(spec_flip, nullptr, tm);
  return 0;
}

3. Decompose VP/HPL operator functionality and encapsulate it into custom operators, so that the timing of address mapping and unmapping can be manually controlled within the operators, thereby reducing the overhead of map and unmap operations. For the methods and steps for developing custom operators, refer to the DSP Custom Operator Development section. Among them, the operator communication structure definitions and DSP implementation source code are located in the sample directory ucp_tutorial/deps_aarch64/ucp/plugin/dsp_plugin/hobot/hb_dsp_algo, and the operator cmd values are located in the sample directory ucp_tutorial/deps_aarch64/ucp/plugin/dsp_plugin/hobot/include. These can be modified and extended according to actual requirements.

#How to avoid DSP hang caused by acore abnormal exit?

In the current DSP framework, memory is allocated by the ARM side and mapped to the DSP side for use. If the ARM side experiences an abnormal exit (such as coredump), the allocated memory will be reclaimed by the system. If the DSP side task is executed after resource reclamation due to scheduling or other reasons, it may cause exceptions when the DSP side accesses the memory, leading to DSP hang. To avoid such situations, you can optimize through the following two methods:

1. Optimize the memory release timing on the ARM side. When the ARM side abnormally exits, delay for a period of time before releasing the memory, giving the DSP side enough time to complete the current task and exit, avoiding the DSP side from accessing already released memory. The specific delay time can be controlled via the following commands.

# get current unmap delay time, default is 1000ms
cat /sys/devices/virtual/misc/vdsp0/vdsp_ctrl/unmap_delay
# set unmap delay time to 500ms
echo '500' > /sys/devices/virtual/misc/vdsp0/vdsp_ctrl/unmap_delay

2. Use the vdsp_msg communication protocol in UCP, with the following specific enable method:

# ARM side environment variable setting, enable vdsp_msg protocol. Default is 0, which is ipcf protocol; setting to 1 or vdsp_msg enables the vdsp_msg protocol.
export HB_DSP_RPC_MODE=1

# DSP side configure corresponding startup parameters before the image starts. msg_mode=1 enables vdsp_msg protocol, the other two parameters are related to low-level software and need to be configured simultaneously.
echo 'msg_mode=1 vdsp0.msgmode=1 vdsp1.msgmode=1' > /sys/module/hobot_remoteproc/parameters/bootargs

#How to use fusa version of VDSP in J6B environment?

VDSP Q8 supports functional safety by default. VDSP V130 supports limited functional safety, including functional safety version of libxos.a and header files, libxi.a, libtile_manager.a, as well as functional safety compilation parameters for xt-clang. Active adaptation and enabling of related options is required.

The VDSP custom operator compilation example already provides the -fusa compilation parameter. When compiling VDSP V130 images, you can ensure functional safety through the following steps:

1. Adapt to the functional safety version of libxos.a and header files.
2. Refer to the -fusa compilation parameter description in the example to ensure that the functional safety compilation parameters of xt-clang are enabled for image compilation.
3. Configure functional safety related startup parameters in the startup script for VDSP images.

The following provides detailed explanation for each step:

Step 1: Adapt to the functional safety version of libxos.a and header files.

It is necessary to upgrade XOS from RJ-2025.5 (QM version, XOS_v3.06) to the FuSa version required for functional safety (XOS_FuSa_v3.02.00S). You can refer to the following steps to complete the update of libxos.a and header files, and perform build verification combined with the functional safety compilation parameters of xt-clang.

1. Obtain the FuSa version library

   1. The FuSa version library (XOS FuSa source code and compilation guide) is provided by Horizon technical support personnel. Please contact technical support to obtain it.
   2. Package information: Horizon_v3_XS1a.zip (FuSa version XOS source code and XOS compilation guide), Horizon_v3_XS1b.zip (XOS/XTOS/HAL/libiDMA/XT-CLANG user manuals).

2. XOS upgrade steps (using installation in /opt/xtensa directory as example)

   1. Backup current version xos

   cd /opt/xtensa/XtDevTools/install/tools/RJ-2025.5-linux/XtensaTools/xtensa-elf/src/
   mkdir xos_v3.06_bak
   mv xos/src xos/testsuite xos_v3.06_bak/
   2. Extract FuSa version package and copy src and testsuite

   cp -ra src /opt/xtensa/XtDevTools/install/tools/RJ-2025.5-linux/XtensaTools/xtensa-elf/src/xos/
   cp -ra testsuite /opt/xtensa/XtDevTools/install/tools/RJ-2025.5-linux/XtensaTools/xtensa-elf/src/xos/
   3. Compile XOS

   cd /opt/xtensa/XtDevTools/install/tools/RJ-2025.5-linux/XtensaTools/xtensa-elf/src/xos/src
   export PATH=$PATH:/opt/xtensa/XtDevTools/install/tools/RJ-2025.5-linux/XtensaTools/bin
   xt-make clean
   xt-make MODE=OPT all
   
   # Generated libxos.a
   ls -lh /opt/xtensa/XtDevTools/install/tools/RJ-2025.5-linux/XtensaTools/xtensa-elf/src/xos/bin/libxos.a
   
   # Generated header files
   ls -lh /opt/xtensa/XtDevTools/install/tools/RJ-2025.5-linux/XtensaTools/xtensa-elf/src/xos/include/xtensa/
   4. Backup current version libxos.a and header files

   cp /opt/xtensa/XtDevTools/install/builds/RJ-2025.5-linux/v130_real_config/xtensa-elf/arch/lib/libxos.a \
      /opt/xtensa/XtDevTools/install/builds/RJ-2025.5-linux/v130_real_config/xtensa-elf/arch/lib/libxos.a.bak
   
   cp -r /opt/xtensa/XtDevTools/install/tools/RJ-2025.5-linux/XtensaTools/xtensa-elf/include/xtensa \
      /opt/xtensa/XtDevTools/install/tools/RJ-2025.5-linux/XtensaTools/xtensa-elf/include/xtensa_bak
   5. Replace with FuSa compiled output

   # Replace libxos.a
   cp /opt/xtensa/XtDevTools/install/tools/RJ-2025.5-linux/XtensaTools/xtensa-elf/src/xos/bin/libxos.a \
      /opt/xtensa/XtDevTools/install/builds/RJ-2025.5-linux/v130_real_config/xtensa-elf/arch/lib/libxos.a
   
   # Replace header files
   cp /opt/xtensa/XtDevTools/install/tools/RJ-2025.5-linux/XtensaTools/xtensa-elf/src/xos/include/xtensa/* \
      /opt/xtensa/XtDevTools/install/tools/RJ-2025.5-linux/XtensaTools/xtensa-elf/include/xtensa/
   6. Verify Link libxos.a in the build project (e.g., LDFLAGS += -lxos). It is recommended to recompile and verify the vdsp executable running result using the newly compiled xos/libxos.a.

   //The xos.h header file contains version number definition, with the following format
   //-----------------------------------------------------------------------------
   // XOS version.
   //-----------------------------------------------------------------------------
   #define XOS_VERSION_MAJOR       3
   #define XOS_VERSION_MINOR       2
   #define XOS_VERSION_PATCH       0
   #define XOS_VERSION_STRING      "3.02.00S"      ///< XOS version string.
   
   //Add compilation macro when building the project to print version information
   #define STR(x) #x
   #define XSTR(x) STR(x)
   #pragma message "XOS Version: " XSTR(XOS_VERSION_STRING)
   #error "Only for Test Version"

Step 2: Enable xt-clang functional safety compilation parameters.

The functional safety guide requires that when compiling functional safety versions with xt-clang, -safety must be added to the compilation parameters. There are the following restrictions. The VDSP custom operator development example has already made corresponding adaptations. You can locate the -fusa compilation option to view specific adaptation details. Additionally, you should also pay attention to this during custom operator development.

1. Restriction 1: --whole-archive is not supported

Solution: Remove --whole-archive from Makefile or CMAKE, and compare the difference of compiled vdsp executable before and after removal. You need to supplement any potentially missing .o dependencies after difference analysis.

# with --whole-archive
readelf -a vdsp0 | grep FILE | sed 's/.*FILE/FILE/' | sort > safety_with_whole-archive.txt

# after removing --whole-archive
readelf -a vdsp0 | grep FILE | sed 's/.*FILE/FILE/' | sort > safety_no_whole-archive.txt

2. Restriction 2: -LNO:simd is not supported

Solution: Remove -LNO:simd. After removal, access bandwidth may decrease. You can use __vec_memcpy to replace ordinary memcpy in appropriate scenarios (Note: __vec_memcpy cannot be used in interrupt handling functions).

extern void *__vec_memcpy(char *__restrict dest, const char *__restrict src, int n);

3. Restriction 3: .extern is not supported in assembly

Solution: Remove .extern related usage in assembly.

4. Restriction 4: Thread stack start address alignment

When using xos_thread_create to create threads, ensure thread stack start address is 4-byte aligned, refer to:

uint8_t stack[XOS_STACK_MIN_SIZE] __attribute__((aligned(4)));

5. Restriction 5: Need to remove several options that conflict with -safety compilation option

-rdynamic, -rpath, -fvectorize, etc. For other potentially conflicting compilation options, please refer to the -safety related chapters in the documentation to confirm.

Step 3: Configure functional safety related startup parameters.

In the startup script for VDSP images, add configuration to enable functional safety related startup parameters. This feature periodically calls idma self-test when system load is low to monitor potential memory errors. When system load is high, the idma self-test frequency will be reduced to avoid excessive consumption of system resources. The minimum frequency is 10 times the original period.

echo 'idma_stl_enable=1' > /sys/module/hobot_remoteproc/parameters/bootargs