• English
  • 3.0.31

    • The hb_verifier Tool

    hb_verifier is a consistency verification tool that supports performing cosine similarity comparisons between onnx models, between onnx model and hbir model, between hbir model and hbir model, and output consistency comparisons between bc model and Hbm model.

    The cosine similarity indicates the consistency between quantized models at different stages. As the cosine similarity gets closer to 1, it indicates that the outputs of the two quantized models being compared are closer to each other.

    Consistency comparison prints output consistency information for the comparison models, including output name, consistency, number of inconsistent elements, maximum absolute error, and maximum relative error.

    Range of Support

    • Fbc represents the floating-point hbir model.

    • Qbc represents the fixed-point hbir model.

    Comparison ModelsComparison ScenariosWhether the Model is from hb_compileInput Data Requirement
    Onnx vs OnnxCosine similarity comparisonNot interestedOne original data
    Onnx vs FbcCosine similarity comparisonNot interestedOne original data
    Onnx vs Qbc
    		Cosine similarity comparisonYes
    		One original data
    Onnx vs QbcCosine similarity comparisonNoOne original data
    One runtime data
    Fbc vs QbcCosine similarity comparisonYes
    		One original data
    Fbc vs QbcCosine similarity comparisonNo
    		One original data
    One runtime data
    Qbc vs HbmOutput consistency comparisonYesOne runtime data
    Qbc vs HbmOutput consistency comparisonNoOne runtime data

    How to Use

    # onnx vs bc
hb_verifier -m ${model.onnx},${model.bc} \ 
            -i ${raw_input_data_1.npy}

# fbc vs qbc
hb_verifier -m ${float_model.bc},${quantized_model.bc} \ 
            -i ${raw_input_data_1.npy} \
            -i ${runtime_input_data_1.npy}

# multi input model verification for onnx vs bc
hb_verifier -m ${model.onnx},${model.bc} \ 
            -i ${raw_input_data_1.npy},${raw_input_data_2.npy}

# multi input model verification for fbc vs qbc not from hb_compile
hb_verifier -m ${model.bc},${model.bc} \ 
            -i ${raw_input_data_1.npy},${raw_input_data_2.npy}
            -i ${runtime_input_data_1.npy},${runtime_input_data_2.npy}

    Parameters

    ParameterDescription
    -h, --helpDisplay help information and exit.
    --versionDisplay version information and exit.
    -m, --model

    Specify the model input, supports ONNX model (*.onnx), HBIR model (*.bc) file, HBM model (*.hbm) file.
    The two model addresses are separated by a “,”: -m model1,model2.

    -i, --inputSpecify the data to be used for inference testing; only *.npy files are supported.
    For multi-input models, there are two ways to pass input data, separated by commas:
    • input_name1:input_data_1.npy,input_name2:input_data_2.npy, …
    • input_data_1.npy,input_data_2.npy…
    If you need to input data for two different models, please use two -i parameters.
    Attention:
    • The input data must include preprocessing operations, such as color conversion, mean, scale, etc., which should be done externally when preparing the input.
    • If there are no special requirements, the raw data can use the calibration dataset as input; runtime data needs to be consistent with the input of the quantized model.
    • The order of input data names must match the actual input order of the model.
    -c, --compare_digitsSet the numerical precision of the comparison inference result (i.e. the number of decimal places to compare the value), if not specified, the tool will compare to five decimal places by default.

    Examples of Reference Usage

    1. Cosine similarity comparisons were performed between the ONNX model and the ONNX model.

    Take the model optimization stage model optimized_float_model.onnx and the model calibration stage model calibrated_model.onnx as an example:

    hb_verifier -m googlenet_optimized_float_model.onnx,googlenet_calibrated_model.onnx -i input.npy
    1. Cosine similarity comparisons were performed between the ONNX model and the HBIR model.

    Take the model optimization stage model optimized_float_model.onnx and the model quantization stage fixed-point model quantized_model.bc as an example:

    hb_verifier -m googlenet_optimized_float_model.onnx,googlenet_quantized_model.bc -i input.npy
    1. Output consistency comparisons were performed between the HBIR model and the HBM model.

    Take the model quantization stage fixed-point model quantized_model.bc and the model compilation stage model googlenet.hbm as an example:

    hb_verifier -m googlenet_quantized_model.bc,googlenet.hbm -i runtime_input.npy

    Output Contents

    Cosine Similarity Comparison

    The cosine similarity information for the compared models will be printed within the terminal, as shown in the example below:

    +-----------------------------------+----------------------------+-------------------+
| NodeName                          | TensorName                 | ConsineSimilarity |
+-----------------------------------+----------------------------+-------------------+
| Conv_0                            | 365                        | 0.999978          |
| Relu_2                            | 367                        | 0.999976          |
| MaxPool_3                         | 368                        | 0.999901          |
| Conv_4                            | 369                        | 0.999112          |
| Relu_6                            | 371                        | 0.999212          |
| ...                               | ...                        | ...               |
| Conv_189                          | 554                        | 0.984707          |
| Relu_191                          | 556                        | 0.989554          |
| Concat_192                        | 557                        | 0.989221          |
| GlobalAveragePool_193             | 558                        | 0.995896          |
| Gemm_195                          | transposed_replaced_output | 0.995836          |
| Gemm_195_transpose_output_reshape | output                     | 0.995836          |
+-----------------------------------+----------------------------+-------------------+

    Among them:

    • NodeName represents the name of the operator.
    • TensorName represents the Tensor name of the first output of the operator.
    • ConsineSimilarity represents the calculated cosine similarity.

    Output Consistency Comparison

    +------------+-------------+---------------------+--------------+--------------+
| OutputName | Consistency | Mismatched Elements | Max Abs Diff | Max Rel Diff |
+------------+-------------+---------------------+--------------+--------------+
| output     | True        | 0/1000 (0.00%)      | 0.0          | 0.0          |
+------------+-------------+---------------------+--------------+--------------+

    Among them:

    • OutputName represents the output name.

    • Consistency represents whether the output is consistent or not.

    • Mismatched Elements represents the number and percentage of inconsistent elements.

    • Max Abs Diff represents the maximum absolute error.

    • Max Rel Diff represents the maximum relative error.