#ResNet50 MNIST Quantization Example

The OE package provides one ONNX Post-Training Quantization (PTQ) conversion example under the samples/onnx_ptq path, namely the ResNet50 MNIST example located at samples/onnx_ptq/resnet50_mnist.

This example targets a ResNet50 model trained on the MNIST handwritten digit dataset. It ships a complete set of scripts that cover model checking, calibration data preparation, quantization and compilation, single-image inference, and dataset accuracy evaluation. Except for the dataset accuracy evaluation in Step 5, which requires you to prepare the MNIST test set yourself and fill in its path in the script, all scripts run as is without modification. If your model input is non-image data such as a feature map, radar point cloud features, or spectral data, you can reference the configuration used here directly.

#What the Sample Contains

The sample is located at samples/onnx_ptq/resnet50_mnist in the OE package. The floating-point model, the calibration images, and the test images all ship with the package, so no extra download is needed. The directory contents are as follows.

resnet50_mnist/
├── 01_check.sh                     # Check the floating-point model
├── 02_preprocess.sh                # Prepare calibration data
├── 03_build.sh                     # Quantize and compile
├── 04_inference.sh                 # Single-image inference
├── 05_evaluate.sh                  # Dataset accuracy evaluation
├── resnet50_mnist_config.yaml      # Quantization and compilation configuration file
├── preprocess.py                   # Preprocessing and test set data loading
├── postprocess.py                  # Postprocessing and accuracy statistics
├── infer.py                        # Single-image inference script
├── eval.py                         # Dataset evaluation script
├── resnet50_mnist.onnx             # Floating-point ONNX model, input shape 1×1×28×28
├── calibration_data/               # 28×28 grayscale PNG images for calibration
├── test_data/                      # Grayscale PNG images for inference verification
└── README.md                       # Brief description shipped with the sample

The five numbered scripts cover the complete conversion path and are meant to be run in order from 01 to 05. The four Python files are called by the numbered scripts, and their responsibilities are as follows.

• preprocess.py, handles calibration data preprocessing, single-image inference preprocessing, and test set data loading.

• postprocess.py, handles classification postprocessing, accuracy statistics, and evaluation report generation.

• infer.py, the single-image inference script, called by 04_inference.sh.

• eval.py, the dataset evaluation script, called by 05_evaluate.sh.

#Sample Workflow

#Environment Preparation

Before running the numbered scripts, install and configure the Docker environment and enter the container by following the section Docker Container Preparation, then enter the sample directory.

cd samples/onnx_ptq/resnet50_mnist

The sample ships with its own floating-point model, calibration images, and test images, so nothing extra needs to be prepared.

#Prepare the Evaluation Dataset

The only thing you need to prepare yourself is the official MNIST test set in IDX format, which is used for the dataset accuracy evaluation in Step 5. Prepare t10k-images-idx3-ubyte and t10k-labels-idx1-ubyte, and place them in the same directory. For how to configure the path, see Step 5: Evaluate Dataset Accuracy.

Once everything is in place, run the five scripts in order from 01 to 05.

#Step 1: Check the Floating-point Model

Run 01_check.sh to perform model checking.

sh 01_check.sh

The script internally calls hb_compile to check resnet50_mnist.onnx. The example checks against the J6E platform by default, so --march is set to nash-e. If your target platform differs, change the value accordingly. For quantization strategy differences across platforms, see the section Platform Differences.

When checking passes, the terminal prints the deployment status of each operator. If checking fails, review the error message from the terminal output or from hb_compile.log generated in the current path. For more details, see the section Model Checking.

#Step 2: Prepare Calibration Data

Run 02_preprocess.sh to generate calibration data.

sh 02_preprocess.sh

The script internally calls preprocess.py, which reads calibration images from the calibration_data directory, stores the processed data into the calibration_data_float32 directory, and uses 100 images for calibration by default. These three items are controlled by the --src_dir, --dst_dir, and --cal_img_num options respectively, and can be modified as needed.

The preprocessing steps are exactly the same as the training preprocessing: the image is read as single-channel grayscale, pixel values are divided by 255 to scale them to the 0～1 range, and the result is standardized with MNIST_MEAN and MNIST_STD. These two values are defined in preprocess.py, with a mean of 0.1307 and a standard deviation of 0.3081, matching the values used during model training, and the same values are shared by calibration, single-image inference, and dataset evaluation.

Each processed image is saved as one .npy file named in the form of xxx.cali.npy, with data type float32 and shape (1, 28, 28), which matches input_shape in the configuration file with the batch dimension removed. When the script finishes, it prints a verification line containing the shape, data type, and value range of the first sample, which you can use to confirm the preprocessing result.

For general guidance on calibration data preparation, see the section Data Preparation.

#Step 3: Quantize and Compile

This step performs quantization and compilation based on resnet50_mnist_config.yaml. Before running it, confirm that the configuration file matches your model.

#Configuration File

The main parameters in resnet50_mnist_config.yaml are as follows.

model_parameters:
  onnx_model: './resnet50_mnist.onnx'
  march: "nash-e"
  working_dir: 'model_output'
  output_model_file_prefix: 'resnet50_mnist_28x28_gray'

input_parameters:
  input_name: ""
  input_type_rt: 'featuremap'
  input_type_train: 'featuremap'
  input_layout_train: 'NCHW'
  input_shape: '1x1x28x28'
  norm_type: 'no_preprocess'

calibration_parameters:
  cal_data_dir: './calibration_data_float32'
  calibration_type: 'default'

compiler_parameters:
  optimize_level: 'O2'

The input parameter group deserves the most attention, as it is where this example differs most from an image input model. Both input_type_rt and input_type_train are set to featuremap, meaning that the data format received by the on-board model and the data format of the original floating-point model are both treated as float32 feature maps, and the conversion tool does not insert a color space conversion node at the model front end. norm_type is set to no_preprocess, meaning that normalization is not performed by the conversion tool but by preprocess.py on the Python side.

In calibration_parameters, cal_data_dir points to the calibration_data_float32 directory generated in Step 2, and calibration_type is set to default. If the accuracy after quantization does not meet your requirements, try mix, kl, and max in that order.

For the value ranges and usage of the remaining parameters, see the section Specific Parameter Information. For the full template, see the section Configuration File Template.

#Run the Compilation

Once the configuration is confirmed, run 03_build.sh.

sh 03_build.sh

The conversion outputs are stored in the model_output directory specified by working_dir in the configuration file, and file names are prefixed with the output_model_file_prefix value resnet50_mnist_28x28_gray. The next two steps use two of these outputs:

• model_output/resnet50_mnist_28x28_gray_original_float_model.onnx, used to evaluate the floating-point baseline accuracy.

• model_output/resnet50_mnist_28x28_gray_quantized_model.bc, used to evaluate the accuracy of the quantized model.

For the full output list, the purpose of each output, and which one to use for consistency comparison, see the section Conversion Output.

The Calibrated Cosine and Quantized Cosine similarity values printed during conversion can serve as a reference for numerical stability after quantization, but they have no absolute direct correlation with model accuracy. Whether the quantization result is acceptable should still be judged by the dataset accuracy obtained in Step 5.

#Step 4: Verify Single-image Inference

Run 04_inference.sh to perform inference on a single test image.

# Test the single-image inference result of the quantized model
sh 04_inference.sh

# Test the single-image inference result of the floating-point model
sh 04_inference.sh origin

Without arguments, the script uses the quantized model resnet50_mnist_28x28_gray_quantized_model.bc. When the argument contains origin, it uses the floating-point model resnet50_mnist_28x28_gray_original_float_model.onnx. Both models use the NCHW input layout, and the inference image defaults to ./test_data/test_0000.png.

When inference succeeds, the terminal prints the top-1 class index, probability, and class name. Class names are defined by MNIST_CLASSES in postprocess.py and range from digit 0 to digit 9. If the printed result matches the digit in the image, the preprocessing, inference, and postprocessing chain is working, and you can move on to dataset accuracy evaluation.

#Step 5: Evaluate Dataset Accuracy

Edit 05_evaluate.sh and change MNIST_DATA_ROOT to the actual path of the directory holding the IDX test set.

MNIST_DATA_ROOT="<YOUR_MNIST_DATA_PATH>"
mnist_test_images="${MNIST_DATA_ROOT}/t10k-images-idx3-ubyte"
mnist_test_labels="${MNIST_DATA_ROOT}/t10k-labels-idx1-ubyte"

After the modification, run the evaluation.

# Evaluate the accuracy of the quantized model
sh 05_evaluate.sh

# Evaluate the accuracy of the floating-point model
sh 05_evaluate.sh origin

As with single-image inference, the script evaluates the quantized model without arguments and the floating-point model when the argument contains origin. The script calls eval.py, and MNISTIdxDataLoader in preprocess.py reads the IDX files directly and normalizes each sample, so there is no need to convert the test set into images beforehand.

During evaluation, the terminal prints the cumulative accuracy every 1000 samples, and prints the total sample count, the number of correct samples, and the final accuracy when it finishes. It also generates resnet50_mnist_eval.log in the current directory, which records the index, predicted class, and correctness of every sample, and can be used to locate specific misclassified samples.

eval.py also provides a --parallel_num option to specify the number of parallel processes, which defaults to 1. If evaluation takes too long, append this option to the python3 -u ./eval.py command in 05_evaluate.sh to raise the parallelism as appropriate.

The floating-point model in this example reaches an accuracy of 0.9968 on all 10000 MNIST test samples, which can serve as the baseline before quantization. If the accuracy of the quantized model is noticeably lower than this baseline, first confirm that the calibration data preprocessing matches the training preprocessing, then refer to the section Model Accuracy Tuning to adjust the quantization configuration.

#What to Read Next

After completing the five steps above, you have a quantized model ready for deployment. Choose your next direction as needed.

• To understand the full description of each stage and the meaning of each output, see the section ONNX Model Quantization Principle and Process.

• To evaluate the static performance and the measured on-board performance of the model, see the section Model Performance Evaluation.

• To verify the output consistency between quantized_model.bc and hbm on the dev PC, see the section Model Accuracy Evaluation.

• To deploy the model to a dev board and integrate it into an upper-layer application, see the dev board environment verification and model deployment parts of the section ONNX Model Quick Start.