• English
  • 3.7.1

    • Model Inference

    The hrt_model_exec infer tool can be used for model inference, where the input data are defined by user and one frame is inferred.

    The program runs a single frame of data in a single thread and outputs the time of the model execution.

    Support Range

    The support range of the model: QAT(Quantized Awareness Training) model, PTQ (Post-Training Quantization) model.

    Usage

    Usage:
hrt_model_exec [Option...] [Parameter]

[Option]                      [instruction]
---------------------------------------------------------------------------------------------------------------
-h --help                     Display this information
-v --version                  Display this version

[Option]                      [Parameter]
---------------------------------------------------------------------------------------------------------------
--model_file                  [string]: Model file paths, separate by comma, each represents one model file path.
--model_name                  [string]: Model name.
                                        When model_file has one more model and Subcommand is infer or perf,
                                        "model_name" must be specified! 
--input_file                  [string]: Input file paths, separate by comma, each represents one input.
                                        The extension of files should be one of [jpg, JPG, jpeg, JPEG, png, PNG, bin, txt, npy]
                                        bin for binary such as image data, nv12 or yuv444 etc.
                                        txt for plain data such as image info.
--core_id                     [string]: core id, 0 for any core, 1 for core 0, 2 for core 1 and etc, default is 0.
                                        Please confirm the number of bpu cores on the board before setting up.
                                        When you need to specify multiple cores, separate them with commas, such as "1,2".
--frame_count                 [int]   : frame count for run loop, default 200, valid when perf_time is 0 in perf mode; default 1 for infer mode.
--dump_intermediate           [string]: dump intermediate layer input and output. The default is 0. Subcommand must be infer.
--enable_dump                 [bool]  : flag for dump infer input and output. The default is false. Subcommand must be infer.
--dump_precision              [int]   : Output dump precision for float32/float64 in txt file. Default is 9 decimal places.
--dequantize_process          [bool]  : dequantize the model infer output. The default is false.
                                        Subcommand must be infer, enable_dump set as true
--remove_padding_process      [bool]  : remove padding of the model infer output. The default is false.
                                        Subcommand must be infer, enable_dump set as true
--dump_format                 [string]: output dump format, only support [bin, txt, npy]. The default is bin.
                                        When dumping data in .npy format, the tool will always remove padding; the remove_padding_process parameter has no effect.
                                        Subcommand must be infer.
--dump_txt_axis               [int]   : The txt file of dump is expanded according to the specified axis;
                                        the default is -1, which means there is only one data per line;
                                        Subcommand must be infer, dump_format must be txt.
                                        range:[0, tensor_dimension].
--enable_cls_post_process     [bool]  : flag for classfication post process, only for ptq model now.
                                        Subcommand must be infer.
--dump_path                   [string]: dump file path, --enable_dump or --dump_intermediate will dump model nodes inputs and outputs files.
--input_img_properties        [string]: Specify the color space of the image type input. Each image needs to specify the color space, separated by commas.
                                        The supported color spaces are [Y, UV].
--input_valid_shape           [string]: Complete the validshape of the model input, allowing only the dynamic part to change. Provide two ways to set:
                                        1. This only needs to be set when the validShape of the model input is dynamic. 2. Set for all inputs.
                                        Different inputs are separated by semicolons, and different dimensions are separated by commas.
                                        For example: --input_valid_shape="1,376,376,1;1,188,188,2".
--input_stride                [string]: Complete the stride of the model input, allowing only the dynamic part to change. Provide two ways to set:
                                        1. This only needs to be set when the stride of model input is dynamic. 2. Set for all inputs.
                                        Different inputs are separated by semicolons, and different dimensions are separated by commas.
                                        For example: --input_stride="144384,384,1,1;72192,384,2,1".

[Examples]
---------------------------------------------------------------------------------------------------------------
hrt_model_exec infer              
   --model_file                 
   --model_name                  
   --core_id                    
   --input_file                  
   --input_img_properties        
   --input_valid_shape            
   --input_stride               
   --frame_count                 
   --dump_intermediate            
   --enable_dump                  
   --dump_precision               
   --dequantize_process         
   --dump_path                     
   --remove_padding_process        
   --dump_format                   
   --dump_txt_axis                 
   --enable_cls_post_process

    Parameters Introduction

    ParameterData typeParameter DescriptionCorrelated Parameters
    -h, --helpNone.Display the help information.None.
    -v, --versionNone.View the version number of the DNN prediction library of the tool.None.
    inferNone.Perform model inference and get model inference results.This parameter should be used together with input_file to specify the input image path, and the tool resizes the image according to the model information and organizes the model input information.
    model_filestringModel file path, multiple paths can be separated by commas.None.
    model_namestringSpecify the name of a model.None.
    input_filestringModel input information.
    The input of the image type, it must have one of the following file name suffixes: PNG/JPG/JPEG/png/jpg/jpeg/bin/txt/npy.
    The inputs should be separated by commas ,, such as xxx.jpg,input.txt.    		This parameter needs to be used in conjunction with the subcommand inferto specify the input image path, and the tool resizes the image according to the model information and organizes the model input information.
    core_idstringSpecify the running core, 0 means arbitrary core, 1 means core0, 2 means core1 and etc. Default value is 0. When you need to specify multiple cores, separate them with commas, such as "1,2".None.
    input_img_propertiesstringThe color space information of the model image input, range [Y, UV].This parameter should be used together with input_file, each image type input in input_file needs to specify a Y/UV type, and each input color space needs to be separated by an English character comma ,, such as: Y,UV.
    input_valid_shapestringModel dynamic validShape input information.
    If the model input attribute validShape contains -1, the -1 part needs to be completed, and multiple validShape are separated by English semicolons.
    For example: --input_valid_shape="1,376,376,1;1,188,188,2".    		None.
    input_stridestringModel dynamic stride input information.
    If the model input attribute stride contains -1, the -1 part needs to be completed, and multiple strides are separated by English semicolons.
    For example:--input_stride="50176,224,1,1;25088,224,2,1".    		None.
    frame_countintThe number of running frames of the execution model.
  • When the subcommand is infer, defaults to 1.
  • When the subcommand is perf, defaults to 200.
    		• When the subcommand is perf, it takes effect when perf_time is not set.
    dump_intermediatestringdump model each layer of input and output, range [0, 3]. Default value is 0.
  • When dump_intermediate=0, the dump function is turned off by default.
  • When dump_intermediate=1, the input and output data of each node layer in the model are saved as bin, where inputs and outputs of node are stride data.
  • When dump_intermediate=2, the input and output data of each node layer in the model are saved as bin and txt, where the inputs and outputs of node are stride data.
  • When dump_intermediate=3, the input and output data of each node layer in the model are saved as bin and txt, where the inputs and outputs of node are valid data.
    		• None.
    enable_dumpboolEnables dump model input and output. Default value is false.None.
    dump_precisionintControls the number of decimal places of the float type data output in txt format. Default value is 9.None.
    dequantize_processboolInverse quantization of model output. Default value is false.Effective when the enable_dump parameter is true.
    remove_padding_processboolRemove padding of model output. Default value is false.Effective when the enable_dump parameter is true.
    dump_formatstringThe format of the dump model input and output, range [bin, txt, npy]. Default value is bin.When dumping data in .npy format, the tool will always remove padding; the remove_padding_process parameter has no effect.
    dump_txt_axisintControl line feed rules for txt format input and output.
    If output dimension = n, then parameter range: [0, n]. Defaults to -1, which means one data per row.    		None.
    enable_cls_post_processboolEnables classification post-processing. Default value is false.This parameter needs to be used in conjunction with the subcommand infer. Currently, it only supports post-processing of the ptq classification model and printing of classification results.
    dump_pathstringThe path of dump model input and output.Effective when the enable_dump or dump_intermediate parameter is set.

    Usage Example

    Model Inference

    Single Model Inference

    hrt_model_exec infer --model_file=xxx.hbm --input_file=xxx.bin

../aarch64/bin/hrt_model_exec infer --model_file=resnet50_224x224_nv12.hbm --input_file=zebra_cls.jpeg,zebra_cls.jpeg --input_img_properties=Y,UV --input_stride=50176,224,1,1;25088,224,2,1
Load model to DDR cost 1965.03ms.
[I][35143][06-28][10:39:51:373][file_util.cpp:527][hrt_model_exec][HRT_MODEL_EXEC] The input valid shape is (1,224,224,1), and the image [zebra_cls.jpeg] will be scaled to 224x224
[I][35143][06-28][10:39:51:377][file_util.cpp:527][hrt_model_exec][HRT_MODEL_EXEC] The input valid shape is (1,112,112,2), and the image [zebra_cls.jpeg] will be scaled to 224x224

---------------------Frame 0 begin---------------------
Infer time: 1.464 ms
---------------------Frame 0 end---------------------

    Multiple Models Inference

    hrt_model_exec infer --model_file=xxx.hbm,xxx.hbm --model_name=xx --input_file=xxx.jpg

    Model Input Instructions

    Multi-input Model Description

    The tool infer supports inference for multiple input models, supporting image input, binary file input, text file input, and NumPy array file input, with input data separated by commas. The model input information can be viewed via model_info.

    hrt_model_exec infer --model_file=xxx.hbm --input_file=xxx.jpg,input.txt

    Dynamic input instructions

    If the model input is dynamic, you need to use the input_valid_shape and input_stride parameters to complete the dynamic information according to the actual input situation. You can choose to specify the parameters in the following two ways:

    • Only give the validShape or stride information of the dynamic input.

    • Give the validShape or stride information of all inputs. The information of non-dynamic inputs must be consistent with the model information.

    Note

    The tool will automatically complete the dynamic input information to the maximum extent, which is convenient for you to perform performance evaluation more easily. You can choose whether to let the tool automatically complete it according to the actual situation.

    • If the input stride is dynamic and validShape is fixed, you can not specify the input_stride parameter. The tool will automatically complete it according to the minimum alignment rule and print the alignment information.

    • If both validShape and stride are dynamic:

      • If the input is a picture type, you need to specify the color space information of the picture input_img_properties, and the other parameters can be left unspecified. The tool will fill in the input_valid_shape and input_stride information according to the size of the picture.

      • If the input is a type other than a picture or is not specified, you need to set the input_valid_shape information, and the input_stride information will be automatically filled in and printed.

    Taking the model in the Dynamic Input Introduction section as an example, you can run the model with the following command:

    # Only given dynamically input information 
hrt_model_exec infer --model_file=xxx.hbm --input_file="input_y.bin,input_uv.bin,input_roi.bin" --input_valid_shape="1,220,220,1;1,110,110,2" --input_stride="49280,224,1,1;24640,224,2,1"

# Given all the input information
hrt_model_exec infer --model_file=xxx.hbm --input_file="input_y.bin,input_uv.bin,input_roi.bin" --input_valid_shape="1,220,220,1;1,110,110,2;1,4" --input_stride="49280,224,1,1;24640,224,2,1;16,4"

    Image type input instructions

    When input_file is given an image input, you need to use the input_img_properties parameter to specify which color space of the image you want to use as input to the model. Currently, only Y and UV color spaces are supported.

    hrt_model_exec infer --model_file=xxx.hbm --input_file="img.jpg,img.jpg,input_roi.bin" --input_img_properties="Y,UV"