#Common issues

#General troubleshooting methods

If an exception occurs, please first check the error log message. If the x86-side logs cannot accurately locate the issue, please check the logs on the development board at the path ${remote_root}/${session_token}/log/s.log. ${remote_root} defaults to /map/hbm_infer; ${session_token} is the token value printed in the error log.

#SSH-related errors

If an SSH error occurs, first verify that the network connection between the x86 host and the development board is functioning properly. If the SSH port on the development board is not 22, specify the actual port using the ssh_port parameter. If the login account on the development board is not root, specify the correct value using the username parameter. If the account password is not empty, specify the correct value using the password parameter. In this case, the sshpass tool must be installed on the x86 host to allow the password to be passed during resource cleanup.

#InactivcateRpcError exception

• If the first-frame inference fails and there are no exceptions in the development board logs (no error logs or timed out), please test gRPC communication availability as follows:

1. The test cases we provide are located in the grpctest directory of the hbm_infer package. To use them, copy test_grpc_server_${os} to the development board and run it manually, where ${os} can be either linux or qnx.

# If no parameters are specified, the port will be automatically assigned.
./test_grpc_server_linux
# Output: Server is running on port: 35017

# Specify the port number.
./test_grpc_server_linux 33333
# Output: Server is running on port: 33333

2. Run on the x86 host:

# IP is the development board’s IP address, and PORT is the port number printed by the server.
python3 -m hbm_infer.grpctest.test_grpc_client IP:PORT

3. If the connection is successful, both the x86 host and the server will print successful transmission messages; otherwise, the x86 host will throw an exception.

If the issue persists, check whether the required ports on the development board are open and accessible. Open them if necessary and retry the test. If only some ports on the development board are open, you can use the grpc_port parameter of HbmRpcSession to specify the gRPC port on the development board. Note that at the same time, all different sessions connecting to the same development board must not use the same gRPC port to prevent data confusion.

• If an InactivateRpcError occurs after several frames of inference, it is recommended to check the network stability and whether the server process has any exceptions.

#Other considerations

• HbmRpcSession, HbmServer, and HbmHandle all need to explicitly call the close interface to release development board resources. It is not recommended to call close in the __del__ method of upper-level objects to avoid exceptions caused by uncertain garbage collection timing.

• It is recommended to use necessary exception handling logic to ensure proper release of development board resources. If multiple failures occur during debugging, it may result in high usage of development board processes and memory; in such cases, manually clean up the board in time.

• When using multiple model sessions, the model file names and model graph names must not be duplicated; otherwise, exceptions such as model files being overwritten or not found may occur.

• When inferring L2M models, use the tool’s remote_environment parameter to set the corresponding environment variables, and use the core_id parameter to specify the BPU core for inference.

• If you want to quickly evaluate the tool’s efficiency, you can refer to the following command for testing:

python3 -m hbm_infer.perftest.test_performance --device xx.xx.xx.xx --model_dir xx/xx --log_path perf.log --epoch 100