#PyTorch Model Accuracy Tuning Tool

Due to the error in the floating point to fixed point process, you will inevitably encounter the problem of quantization model accuracy dropout when using quantization training tools. Typically, there are several reasons for this:

1. The original floating-point model is not favorable for quantization, such as the existence of shared ops or shared structures.

2. The QAT network structure or configuration is abnormal, such as there is a pattern without fuse in the model, and the output is not set with high accuracy.

3. Some operators are sensitive to quantization, and the quantization error of the operator accumulates layer by layer in the forward propagation process, which ultimately leads to a large error in the model output.

For the above cases, we provides the accuracy tuning tool to help you quickly locate and solve the accuracy problems, which mainly includes the following:

• Model Structure Checking: check if there are shared ops, patterns without fuse, or quantization configurations in the model that do not meet expectations.

• Use the QuantAnalysis Class to compare and analyze the two models automatically: by comparing, locate the abnormal operators or quantization-sensitive ops in the quantized model.

• Use the ModelProfiler Class and the HbirModelProfiler Class to get information about the numerical characteristics of each op in the model: the obtained information includes the maximum and minimum values of inputs and outputs, etc. The functionality of these two classes is identical, the difference is that HbirModelProfiler accepts as input only the qat hbir model. Usually you don't need to call this module manually, you can get the numerical information of both models directly from QuantAnalysis.run.

#Quickstart

When encountering quantization model accuracy dropout problems, we recommend using the accuracy tuning tool according to the following process.

1. Check if there are any unfavorable structures or abnormal configurations in the model.

2. Use QuantAnalysis module to analyze the model as follows:

   1). Find the input (i.e., the bad case) that exhibits the maximum discrepancy between the outputs of a baseline model and the model under analysis, and use this input as the input for the model.

   2). Perform quantization sensitivity analysis, the current experience is that the first n L1 sensitivities are usually the quantitative sensitivity ops (the value of n varies from model to model, so there is no automatic method to determine it, so we need to try it manually, e.g., the first 10, 20...). Set the quantization sensitive op to high accuracy quantization (e.g., int16 quantization), and redo the quantization process.

   3). Or compare the inputs and outputs of the two models layer by layer to check whether there is an abnormal quantization op such as the data range is too large or the scale is unreasonable, e.g., certain ops with physical meanings should be set to a fixed scale.

3. When you encounter deployment consistency issues, QuantAnalysis also supports the analysis of hbir models. You can refer to Deployment Consistency Analysis section for interface usage and analysis procedures.

The overall flow chart is as follows:

A whole example is as follows:

from horizon_plugin_profiler import QuantAnalysis
from horizon_plugin_profiler.utils.entities import Metric

import torch
from torch import nn
from torch.quantization import DeQuantStub, QuantStub

from horizon_plugin_pytorch import set_march, qint8
from horizon_plugin_pytorch.quantization import FakeQuantState
from horizon_plugin_pytorch.quantization import hbdk4 as hb4
from horizon_plugin_pytorch.quantization import (
    prepare,
    set_fake_quantize,
    get_qconfig,
)
from horizon_plugin_pytorch.quantization.qconfig_setter import *


class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv = nn.Conv2d(3, 3, 1)
        self.relu = nn.ReLU()
        self.quant = QuantStub()
        self.dequant = DeQuantStub()
    
    def forward(self, x):
        x = self.quant(x)
        x = self.conv(x)
        x = self.relu(x)
        x = torch.nn.functional.interpolate(
            x, scale_factor=1.3, mode="bilinear", align_corners=False
        )
        x = self.dequant(x)
        return x

data = torch.rand((1, 3, 32, 32))
float_net = Net()
float_net(data)

set_march("nash-e")


############################### Model Structure Checking ##############################
# verify that the prompted exception layer is as expected
calib_net = prepare(
    float_net,
    data,
    qconfig_setter=QconfigSetter(
        get_qconfig(),
        [
            ModuleNameTemplate({"": qint8}),
            ConvDtypeTemplate(),
            MatmulDtypeTemplate(),
        ],
    ),
)
##########################################################################

# Perform calibration. There are two cases here.

# Case 1: If you have not yet performed calibration, please do so first.
set_fake_quantize(calib_net.eval(), FakeQuantState.CALIBRATION)
calib_net(data)

# Case 2: If you have already performed calibration, you can refer to the following steps to directly load the weight. Note that you need to replace the parameters with your actual model weight path.
# calib_ckpt = torch.load("/your_calib_ckpt_path")
# calib_net.load_state_dict(calib_ckpt)

float_net.eval()
# Important!!! Set the calibration model into validation state
set_fake_quantize(calib_net.eval(), FakeQuantState.VALIDATION)

# export hbir model
qat_hbir = hb4.export(calib_net, (data,))

############################### quant analysis ############################

# 1. initialization
qa = QuantAnalysis(
    baseline_model=float_net,
    analysis_model=calib_net,
    analysis_model_type="fake_quant",
    device_ids=0,  # Single GPU only. For multi-machine/multi-GPU, refer to the Multi-machine Multi-GPU section
    out_dir="./floatvsqat",
)

# compare qat and qat hbir is also supported
# qa = QuantAnalysis(
#     baseline_model=calib_net,
#     analysis_model=qat_hbir,
#     analysis_model_type="export",
#     device_ids=0,  # Single GPU only. For multi-machine/multi-GPU, refer to the Multi-machine Multi-GPU section
#     out_dir="./qatvshbir",
# )

# 2. If there is a known badcase, the badcase input can be set directly through this interface.
qa.set_bad_case(data)

# in practice, it is recommended to use auto_find_bad_case to search for bad cases across the dataloader
# setting the num_steps parameter to control the search range is also supported
# qa.auto_find_bad_case(your_dataloader, num_steps=100)

# 3. run two model
qa.run()

# 4. compare the two model layer-by-layer. Verify that the abnormal layer indicated by the advice column in the outputs is as expected
qa.compare_per_layer()

# 5. calculate sensitivity nodes. You can set the nodes with the highest qint8-qint16 gain to high accuracy to improve the quantization model accuracy
qa.sensitivity(use_sensitivity_v2=True)

# 6. Optional. Automatically search for the ops that need to be promoted to int16 based on the sensitivity results
# The key of outputs_config is the actual output name of the model. This example is a single-output model, whose name is ""
qa.tune_bad_case(
    outputs_config={
        "": {
            "metric": Metric.ATOL,
            "threshold": 0.2,
            "max_tuned_stubs": 10,
        },
    },
)

#Recommended Workflow Based on a Loss Metric

The quickstart above uses the default Metric.ATOL, which measures the absolute error on a single tensor. For multi-output networks, there is usually no direct correspondence between "the ATOL of a certain head output is large" and "the task accuracy drops".

Therefore, we recommend that you first register a custom loss metric that is consistent with your final task objective, and debug around this loss. In this way you can locate problems directly around the task objective, instead of only looking at the local error on a certain output.

#Two Common Ways to Write a Loss Metric

1. Compute the loss directly with the real GT

Let the model output carry the real GT as well, then extract the GT from output1 inside the metric, and compute the loss against the corresponding quantized prediction in output2.

import torch.nn.functional as F
from horizon_plugin_profiler.utils.entities import Metric


@Metric.register("ce_loss_with_gt", False, "all")
def ce_loss_with_gt(output1, output2):
    gt_label = output1["label"]
    qat_logits = output2["logits"]
    return F.cross_entropy(qat_logits, gt_label)

2. Build a pseudo GT from the floating-point model output first, then compute the loss

Build a pseudo GT from the floating-point prediction in output1, then compute the loss against the quantized prediction in output2.

import torch.nn.functional as F
from horizon_plugin_profiler.utils.entities import Metric


@Metric.register("distill_loss", False, "all")
def pseudo_gt_ce_loss(output1, output2):
    float_logits = output1["logits"]
    qat_logits = output2["logits"]

    pseudo_label = float_logits.argmax(dim=-1)
    return F.cross_entropy(qat_logits, pseudo_label)

How to choose:

• If you have a reliable GT and care more about task accuracy (accuracy tuning), use the "real GT loss".

• If you do not have a GT, or your current goal is to locate the error introduced by quantization (consistency analysis or accuracy tuning), use the "loss with a pseudo GT built from the floating-point output".

#Recommended Workflow

1. Register a custom loss metric consistent with the task objective through Metric.register.

2. Initialize QuantAnalysis and add this loss metric to metric_list.

3. Search for badcases automatically with auto_find_bad_case(..., topk=k). k can be set larger according to your computing resources.

4. Call run(metric=loss_metric) to collect layer-by-layer information of the badcase with the worst loss.

5. Call compare_per_layer to check the layer-by-layer error.

6. Call sensitivity(metric=loss_metric, use_sensitivity_v2=True) to further locate the sensitive ops with the largest impact on the final loss.

7. If you want to further verify the gain of promoting the accuracy on a certain case, you can call tune_bad_case to verify the accuracy gain after the sensitivity v2 analysis is finished.

import torch.nn.functional as F
from horizon_plugin_profiler import QuantAnalysis
from horizon_plugin_profiler.utils.entities import Metric


@Metric.register("loss", False, "all")
def loss(output1, output2):
    return F.l1_loss(output1, output2, reduction="mean")

qa = QuantAnalysis(
    baseline_model=float_model,
    analysis_model=qat_model,
    analysis_model_type="fake_quant",
    device_ids=0,
    out_dir="./horizon_quant_analysis",
    metric_list=[Metric.loss],
)

qa.auto_find_bad_case(data_loader, num_steps=100, topk=20)
qa.run(metric=Metric.loss)
qa.compare_per_layer()
qa.sensitivity(
    metric=Metric.loss,
    use_sensitivity_v2=True,
)
qa.tune_bad_case(
    outputs_config={
        "all": {
            "metric": Metric.loss,
            "threshold": 0.01,
            "max_tuned_stubs": 5,
        },
    }
)

If there is no suitable custom loss for your current task yet, you can also start the analysis with the default Metric.ATOL. But for multi-output networks, it is still recommended to register a loss metric consistent with the final task first.

#Multi-machine Multi-GPU

QuantAnalysis tool now supports multiple machines and multiple GPUs. If you already have a training environment built with torch.nn.parallel.DistributedDataParallel, you can simply replace the train function with the quant_analysis function in the example.

import os
import sys
import time
import torch
import torch.distributed as dist
import torch.nn as nn
import torch.optim as optim
import torch.multiprocessing as mp
from torch.nn.parallel import DistributedDataParallel as DDP
from torch.utils.data import Dataset, DataLoader
from torch.utils.data.distributed import DistributedSampler

from horizon_plugin_pytorch.quantization import (
    QuantStub,
    prepare,
    QconfigSetter,
    set_fake_quantize,
    FakeQuantState,
    get_qconfig,
    observer_v2,
)
from horizon_plugin_pytorch.quantization.qconfig_setter import ModuleNameTemplate
from horizon_plugin_pytorch.march import set_march
from horizon_plugin_pytorch.dtype import qint8
from torch.ao.quantization import DeQuantStub
from horizon_plugin_profiler import QuantAnalysis
from horizon_plugin_profiler.utils.entities import Metric

# Example dataset. Please use the evaluation dataset when debugging
class ExampleDataset(Dataset):
    def __init__(self, size=10000, input_size=32):
        self.size = size
        self.data = torch.randn(size, 3, input_size, input_size)

    def __len__(self):
        return self.size

    def __getitem__(self, idx):
        return self.data[idx]


# Model definition
class ConvNet(nn.Module):
    def __init__(self, num_classes=10):
        super(ConvNet, self).__init__()
        self.layer1 = nn.Sequential(
            nn.Conv2d(3, 16, kernel_size=3, stride=1, padding=1),
            nn.BatchNorm2d(16),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2),
        )
        self.layer2 = nn.Sequential(
            nn.Conv2d(16, 32, kernel_size=3, stride=1, padding=1),
            nn.BatchNorm2d(32),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2),
        )
        self.fc = nn.Linear(8 * 8 * 32, num_classes)
        self.quant = QuantStub()
        self.dequant = DeQuantStub()

    def forward(self, x):
        x = self.quant(x)
        x = self.layer1(x)
        x = self.layer2(x)
        x = x.reshape(x.size(0), -1)
        x = self.fc(x)
        x = self.dequant(x)
        return x


def setup(rank, world_size, master_addr, master_port):
    """Initialize the distributed environment"""
    os.environ["MASTER_ADDR"] = master_addr
    os.environ["MASTER_PORT"] = str(master_port)

    # Initialize process group
    dist.init_process_group(backend="nccl", rank=rank, world_size=world_size)
    torch.cuda.set_device(rank)  # set current GPU


def cleanup():
    """Cleaning up the distributed training environment"""
    dist.destroy_process_group()


def quant_analysis(rank, world_size, config):
    setup(rank, world_size, config["master_addr"], config["master_port"])

    # Set the target hardware platform code, pay attention to the march distinction here nash-e/m/p
    set_march("nash-e")
    float_net = ConvNet()

    # prepare, must use DistributedSampler
    dataset = ExampleDataset(size=10000)
    sampler = DistributedSampler(
        dataset,
        num_replicas=world_size,
        rank=rank,
        shuffle=False,
    )
    dataloader = DataLoader(
        dataset,
        batch_size=1,  # It is recommended to set batch = 1 when debugging
        sampler=sampler,
        num_workers=config["num_workers"],
        pin_memory=True,
    )

    # Load the trained float model checkpoint. Note that the parameter here needs to be replaced with your actual model checkpoint path.
    float_ckpt = torch.load("/your_float_ckpt_path")
    float_net.load_state_dict(float_ckpt)

    # Convert floating point model to calib model
    example_inputs = torch.rand((1, 3, 32, 32))
    calib_net = prepare(
        float_net,
        example_inputs,
        QconfigSetter(
            reference_qconfig=get_qconfig(observer=observer_v2.MSEObserver),
            templates=[
                ModuleNameTemplate({"": qint8}),
            ],
        ),
    )

    # Load the calibration checkpoint. Note that the parameter here needs to be replaced with your actual model checkpoint path.
    calib_ckpt = torch.load("/your_calib_ckpt_path")
    calib_net.load_state_dict(calib_ckpt)

    float_net.eval()
    # Important!!! Set the calibration model into validation state
    set_fake_quantize(calib_net.eval(), FakeQuantState.VALIDATION)

    # debug
    qa = QuantAnalysis(
        baseline_model=float_net,
        analysis_model=calib_net,
        analysis_model_type="fake_quant",
        device_ids=rank,  # Set device_ids to the current rank
        out_dir=config["output_dir"],
    )
    qa.auto_find_bad_case(
        data_generator=dataloader,
        num_steps=100,
        topk=10,
    )
    qa.run(metric=Metric.ATOL)
    qa.compare_per_layer()
    qa.sensitivity(metric=Metric.ATOL, use_sensitivity_v2=True)

    # Cleaning up the distributed training environment
    cleanup()


def main():
    import argparse

    parser = argparse.ArgumentParser(description="Multi-Node QuantAnalysis")
    parser.add_argument(
        "--nnodes", type=int, default=1, help="number of nodes in the cluster"
    )
    parser.add_argument(
        "--nproc_per_node",
        type=int,
        default=torch.cuda.device_count(),
        help="number of GPUs per node",
    )
    parser.add_argument(
        "--node_rank",
        type=int,
        default=0,
        help="rank of this node in the cluster (0-indexed)",
    )
    parser.add_argument(
        "--master_addr",
        type=str,
        default="localhost",
        help="IP address of the master node (node with rank 0)",
    )
    parser.add_argument(
        "--master_port",
        type=int,
        default=29500,
        help="port on master node for communication",
    )
    parser.add_argument(
        "--num_workers",
        type=int,
        default=4,
        help="number of data loader workers per GPU",
    )
    parser.add_argument(
        "--output_dir",
        type=str,
        default="./quant_analysis_output",
        help="directory to save debug outputs",
    )

    args = parser.parse_args()

    os.makedirs(args.output_dir, exist_ok=True)

    config = {
        "master_addr": args.master_addr,
        "master_port": args.master_port,
        "num_workers": args.num_workers,
        "output_dir": args.output_dir,
    }

    # Calculating the global world size
    world_size = args.nnodes * args.nproc_per_node

    # Print configuration information
    if args.node_rank == 0:
        print(f"Starting multi-node QuantAnalysis with:")
        print(f"  Nodes: {args.nnodes}")
        print(f"  GPUs per node: {args.nproc_per_node}")
        print(f"  Total GPUs: {world_size}")
        print(f"  Master address: {args.master_addr}:{args.master_port}")

    # Start distributed debugging
    mp.spawn(
        quant_analysis, args=(world_size, config), nprocs=args.nproc_per_node, join=True
    )


if __name__ == "__main__":
    main()

Launch command:

python quant_analysis_ddp.py \
  --nnodes <number_of_nodes> \
  --nproc_per_node <GPUs_per_node> \
  --node_rank <current_node_rank> \
  --master_addr <master_node_IP> \
  --master_port 29500

#API Reference

#Model Structure Checking

# from horizon_plugin_pytorch.utils.check_model import check_qat_model

def check_qat_model(
    model: torch.nn.Module,
    example_inputs: Any,
    save_results: bool = False,
    out_dir: Optional[str] = None,
):

Check if there are structures in the calibration/qat model that are not favorable for quantization and if the quantization qconfig configuration is as expected.

Parameters

• model: model to be checked.

• example_inputs: model inputs.

• save_results: whether to save the check results to a txt file. Default is False.

• out_dir: save path of the result file 'model_check_result.txt'. Default is empty, save to .model_check_results/.

Output

• screen output: abnormal layers that check out.

• model_check_result.txt: generated when save_results = True. It consists of 5 main parts:

  1). Unfused pattern.

  2). The calling times of each module. Normally each op is called only 1 time, 0 means it is not called, more than 1 time means it is shared many times. If it is not called or shared multiple times, an exception prompt will be displayed.

  3). The qconfig configuration for each op output.

  4). The qconfig configuration for each op weight (if any).

  5). Exception qconfig hints (if any).

Fusable modules are listed below:
name    type
------  -----------------------------------------------------
conv    <class 'horizon_plugin_pytorch.nn.qat.conv2d.Conv2d'>
relu    <class 'horizon_plugin_pytorch.nn.qat.relu.ReLU'>

All modules in the model run exactly once.

Each layer out qconfig:
+---------------+-----------------------------------------------------------+---------------+---------------+----------------+-----------------------------+
| Module Name   | Module Type                                               | Input dtype   | out dtype     | ch_axis        | observer                    |
|---------------+-----------------------------------------------------------+---------------+---------------+----------------+-----------------------------|
| quant         | <class 'horizon_plugin_pytorch.nn.qat.stubs.QuantStub'>   | torch.float32 | qint8         | -1             | MovingAverageMinMaxObserver |
| conv          | <class 'horizon_plugin_pytorch.nn.qat.conv2d.Conv2d'>     | qint8         | qint8         | -1             | MovingAverageMinMaxObserver |
| relu          | <class 'horizon_plugin_pytorch.nn.qat.relu.ReLU'>         | qint8         | qint8         | qconfig = None |                             |
| dequant       | <class 'horizon_plugin_pytorch.nn.qat.stubs.DeQuantStub'> | qint8         | torch.float32 | qconfig = None |                             |
+---------------+-----------------------------------------------------------+---------------+---------------+----------------+-----------------------------+

Weight qconfig:
+---------------+-------------------------------------------------------+----------------+-----------+---------------------------------------+
| Module Name   | Module Type                                           | weight dtype   |   ch_axis | observer                              |
|---------------+-------------------------------------------------------+----------------+-----------+---------------------------------------|
| conv          | <class 'horizon_plugin_pytorch.nn.qat.conv2d.Conv2d'> | qint8          |         0 | MovingAveragePerChannelMinMaxObserver |
+---------------+-------------------------------------------------------+----------------+-----------+---------------------------------------+

#QuantAnalysis Class

QuantAnalysis class can automatically find the bad case with the largest output of two models, and use it as input to compare the output of two models layer by layer. In addition, QuantAnalysis class also provides the function of calculating the sensitivity, you can try to set the node with the topk sensitivity ranking with high accuracy, such as int16 quantization, to improve the accuracy of quanitized model.

class QuantAnalysis(object):
    def __init__(
        self,
        baseline_model: Union[torch.nn.Module, HbirModule],
        analysis_model: Union[torch.nn.Module, HbirModule],
        analysis_model_type: str,
        device_ids: Optional[int] = None,
        post_process: Optional[Callable] = None,
        out_dir: Optional[str] = None,
        metric_list: Optional[List[_Metric]] = None,
    )

Parameters

• baseline_model: baseline model (high accuracy).

• analysis_model: model to be analyzed ( accuracy dropping points).

• analysis_model_type: model to be analyzed. Support input:

  • fake_quant: support two cases:

    • the model to be analyzed can be a calibration model with dropped accuracy, in which case the baseline model can be either the original floating-point model or a accuracy-compliant calibration model in a mixed int8/int16 configuration.

    • the model to be analyzed can be a fixed-point HBIR model with dropped accuracy, and the benchmark model is the corresponding pseudo-quantized HBIR model.

  • pre_export: the model to be analyzed is a pseudo-quantized model with lookup table (LUT) for fixed-point conversion, and the benchmark model is the corresponding original pseudo-quantized model.

  • export: the model to be analyzed is a pseudo-quantized HBIR model, and the benchmark model is the corresponding pseudo-quantized model with LUT for fixed-point conversion.

  • convert: the model to be analyzed is a fixed-point HBIR model, and the benchmark model is the corresponding pseudo-quantized model with LUT for fixed-point conversion.

• device_ids: GPU device index for running the analysis. Single GPU only. For multi-machine/multi-GPU, refer to the Multi-machine Multi-GPU section.

• post_process: post process function which performs on model output.

• out_dir: specify the output directory for the comparison results. Default is ./horizon_quant_analysis.

• metric_list: the list of metrics involved in the comparison. Default is [Metric.L1, Metric.COSINE, Metric.ATOL]. Available metrics are described in Metric Description.

#Metric Description

The metrics used in metric_list, run, sensitivity and other interfaces all come from horizon_plugin_profiler.utils.entities.Metric. The currently built-in metrics are as follows:

If the built-in metrics cannot meet your needs, you can also register a custom metric through Metric.register. The registered metric can be used in all functionalities such as finding badcases and computing sensitivity.

from horizon_plugin_profiler.utils.entities import Metric

@Metric.register(name, ascending, scope)
def custom_metric(output1, output2):
    ...

Parameters

• name: metric name, used for display and indexing. It must not duplicate an existing metric name. After registration it can be accessed through Metric.{name} or Metric.get_metric(name).

• ascending: the sort direction of the metric. True means the larger the value, the better the accuracy (such as COSINE and SQNR), False means the larger the value, the worse the accuracy (such as L1 and ATOL).

• scope: defines whether the metric applies to a single tensor or to the whole model output structure.

  • "pertensor": computed separately for each tensor output. The tool expands the leaf tensors in the model output and records the outputs with names such as 0, 1 and key.

  • "all": computes a single overall score for the whole model output structure. It is suitable for scenarios where the output structure is complex and you want to measure the overall error with a single total score, the most common one being a loss. In this case the output name is fixed to all.

output1 and output2 come from baseline_model and analysis_model respectively, but their granularity is determined by scope:

• When scope="all", output1 / output2 are the complete output structures of both sides.

• When scope="pertensor", the tool first expands the outputs into tensor leaves and then calls the custom function one by one, so output1 / output2 are just a single tensor.

import torch.nn.functional as F
from horizon_plugin_profiler import QuantAnalysis
from horizon_plugin_profiler.utils.entities import Metric

@Metric.register("loss", False, "all")
def loss(output1, output2):
    return F.l1_loss(output1, output2, reduction="mean")

qa = QuantAnalysis(
    baseline_model=float_net,
    analysis_model=calib_net,
    analysis_model_type="fake_quant",
    metric_list=[Metric.loss],
)

The methods in this class are as follows.

#auto_find_bad_case

def auto_find_bad_case(
    self,
    data_generator: Iterable,
    num_steps: int = sys.maxsize,
    cached_attrs: Optional[Tuple[str, ...]] = None,
    dump_in_run: bool = False,
    topk: int = 1,
):

Automatically find the badcase that causes the worst output for the two models.

Parameters

• data_generator: dataloader or a custom iterator that produces one piece of data per iteration.

• num_steps: number of iteration steps.

• cached_attrs: cached attrs to use as input. Usually used in sequence model. For instance, some results of the first frame must be treated as input when running the second frame, Default None.

• dump_in_run: whether dump badcase during the running process. When True, badcases are written to disk while being searched, which is slower but has no OOM risk. When False, they are kept in memory first and dumped together at the end, which is faster but may consume more memory. It is forced to True in multi-machine multi-GPU environments, and defaults to False in single-machine single-GPU environments.

• topk: how many badcases to keep for each output and each metric. Default is 1. It is recommended to set topk > 1 to keep multiple badcases and cross-validate them with the subsequent sensitivity(use_sensitivity_v2=True). For many models the problem is not concentrated on a single worst sample, and only looking at top-1 easily locks your view onto a certain local anomaly, which makes the conclusion of the subsequent sensitivity analysis unstable.

Output

• badcase_report.html: Display the badcase report as an HTML page, including:

  • A table showing the topk badcase index and corresponding metric value for each output under each metric.

  • Statistical information (average value, worst value) of all cases for each output under each metric.

  • Statistical information (average value, worst value) of the topk badcase for each output under each metric.

  • Distribution histogram and percentile information (e.g., 99%, 99.9%, 99.99%) of badcase error for each output under each metric.

• all_badcase_info.pt: Saved BadCaseReport object, containing complete badcase information, which can be loaded via torch.load.

• badcase directory: This directory contains the inputs of each index shown in badcase_report.html, such as 0.pt, 16.pt, etc.

Return Value

A BadCaseReport object, which is also saved in self.badcase_report.

#set_bad_case

def set_bad_case(
    self,
    data: Any,
    baseline_model_cached_attr: Optional[Dict] = None,
    analysis_model_cached_attr: Optional[Dict] = None,
):

Set the badcase manually.

Parameters

• data: badcase input.

• baseline_model_cached_attr: baseline model cached attr.

• analysis_model_cached_attr: analysis model cached attr.

#load_bad_case

def load_bad_case(self, path: Optional[str] = None)

Load a saved badcase report.

Parameters

• path: default None, which loads all_badcase_info.pt directly from the out_dir directory specified during initialization. You can also pass in a local directory, or an http/https link (such as an artifact path on a cluster). In that case the tool first copies/downloads all_badcase_info.pt, the badcase directory and badcase_report.html under that directory to out_dir, and then loads them from out_dir. In multi-machine multi-GPU scenarios only rank 0 is responsible for the copy/download, and the other ranks wait at the barrier and then load from local.

#set_model_profiler_dir

def set_model_profiler_dir(
    self,
    baseline_model_profiler_path: str,
    analysis_model_profiler_path: str,
):

Specify the path to save the output of model_profiler manually.

In some cases, the ModelProfiler is defined and run before QuantAnalysis is initialized, in which case you can directly specify the path to the existing ModelProfiler, skipping the run step of QuantAnalysis and comparing the output of the two models directly.

Parameters

• baseline_model_profiler_path: profiler path for the baseline model.

• analysis_model_profiler_path: profiler path for the model to be analyzed.

#run

def run(
    self,
    device: Optional[Union[torch.device, str, int, List[int]]] = None,
    index: Optional[int] = None,
    run_baseline_model: bool = True,
    run_analysis_model: bool = True,
    metric: Optional[_Metric] = None,
    run_on_cpu: bool = False,
    out_name: Optional[str] = None,
)

Run the two models and save the results for each layer in the model separately.

Parameters

• device: deprecated. Specify the GPU through device_ids during initialization of QuantAnalysis.

• index: use which index input as example input. Once specified, the badcase corresponding to this index is used directly and no longer selected by metric. It cannot be specified together with metric.

• run_baseline_model: whether run the baseline model. Default True.

• run_analysis_model: whether run the analysis model. Default True.

• metric: which metric's selected badcase is used as input. Default None, which is equivalent to Metric.ATOL.

• run_on_cpu: whether to force the models to run on CPU. Default False. When analysis_model_type is export or convert, the tool automatically uses CPU, because hbir models align better with torch CPU results and may differ considerably from torch CUDA, so the per-op error obtained from CPU inference is more accurate.

• out_name: only select the badcase on the specified output. Default None, which selects the worst badcase among all outputs. It only takes effect when index is not specified.

#Analyzing Statistical Information

After the run() interface of QuantAnalysis is called, both of baseline_model and analysis_model will perform inference on the badcase, and the input / output of all layers will be saved in the op_infos folder of output path. By loading the opinfo file using torch.load, you can access the input and output of an operator. Then, with matplotlib, you can visualize the corresponding value distribution.

import torch
import matplotlib.pyplot as plt

# Load opinfo
opinfo = torch.load("xxx.opinfo", weights_only=False)  # opinfo contains arbitrary Python objects, not just weights

input = opinfo.input
output = opinfo.output

# Plot value distribution
plt.hist(output[0].as_subclass(torch.Tensor).flatten().cpu().numpy(), bins=30, color='blue')
plt.title("Output Distribution")
plt.xlabel("Value")
plt.ylabel("Frequency")
plt.grid(True)
plt.savefig("output.png")

When analyzing the error of a single operator, you can build a model which only contains one operator. Feed the inputs from opinfo into the model and compare the difference between the model output and the opinfo output by visualizing with scatter plots or histograms.

#compare_per_layer

def compare_per_layer(
    self,
    prefixes: Tuple[str, ...] = None,
    types: Tuple[Type, ...] = None,
):

Compare the results of each layer in the two models.

Parameters

• prefixes: the prefix of ops.

• types: the types of ops.

Output

• compare_per_layer_report_0.html: Display the layer-by-layer comparison results as an HTML page, including trend charts of metric values for each layer and detailed data tables.

• compare_per_layer_out_0.txt: show the specific information of each layer in the model in the form of a table, including various metrics, data ranges, quantized dtype, etc.. Each column from left to right represents:

  • Index: op index.

  • mod_name: name of the op, if the op is of module type, the prefix name of the module in the model will be shown, if it is of function type, it will not be shown.

  • base_op_type: type of the op in the base model, may be module type or function name.

  • analy_op_type: type of the op in the model to be analyzed, could be module type or function name.

  • Shape: shape of the op.

  • dtype: data type output of the op.

  • qscale: quantized scale output of the op.

  • L1: L1 distance of the op output in the two models.

  • COSINE: cosine similarity of the op output in the two models.

  • ATOL: absolute error of the op output in the two models.

  • max_qscale_diff: the max N scale diff of the op output in the two models.

  • base_model_min: minimum value of the op output in the baseline model.

  • analy_model_min: minimum value of the op output in the model to be analyzed.

  • base_model_max: maximum value of the op output in the baseline model.

  • analy_model_max: maximum value of the op output in the model to be analyzed.

  • base_model_mean: average of the op output in the baseline model.

  • analy_model_mean: average of the op output in the model to be analyzed.

  • same_in_qatbc_qdiff: the quantization error (in scale units) of the op at the same position in the qat bc model. Only available in export/convert mode.

  • same_in_intbc_qdiff: the quantization error (in scale units) of the op at the same position in the fixed-point bc model. Only available in export/convert mode.

  • advice: diagnostic advice for the op, including data range too large, inf or NaN contained, output without high accuracy, clamp error, etc.

    +----+------------+--------------------------------------------------------------------+--------------------------------------------------------------------+----------------------------+---------------+-----------+-----------+-----------+-----------+-------------------+------------------+-------------------+------------------+-------------------+-------------------+--------------------+-----------------------+-----------------------+-----------+
    |    | mod_name   | base_op_type                                                       | analy_op_type                                                      | shape                      | dtype         |    qscale |    COSINE |        L1 |      ATOL |   max_qscale_diff |   base_model_min |   analy_model_min |   base_model_max |   analy_model_max |   base_model_mean |   analy_model_mean | same_in_qatbc_qdiff   | same_in_intbc_qdiff   | advice    |
    |----+------------+--------------------------------------------------------------------+--------------------------------------------------------------------+----------------------------+---------------+-----------+-----------+-----------+-----------+-------------------+------------------+-------------------+------------------+-------------------+-------------------+--------------------+-----------------------+-----------------------+-----------|
    |  0 | quant      | torch.ao.quantization.stubs.QuantStub                              | horizon_plugin_pytorch.nn.qat.stubs.QuantStub                      | torch.Size([1, 3, 32, 32]) | qint8         | 0.0078404 | 0.9999922 | 0.0019772 | 0.0039202 |         0.5000016 |        0.0002798 |         0.0000000 |        0.9996471 |         0.9957269 |         0.4986397 |          0.4986700 |                       |                       |           |
    |  1 | conv       | torch.nn.modules.conv.Conv2d                                       | horizon_plugin_pytorch.nn.qat.conv2d.Conv2d                        | torch.Size([1, 3, 32, 32]) | qint8         | 0.0056401 | 0.9999791 | 0.0020876 | 0.0092935 |         1.6477517 |       -0.7193903 |        -0.7162931 |        0.5436335 |         0.5414499 |        -0.0423445 |         -0.0413149 |                       |                       |           |
    |  2 | relu       | torch.nn.modules.activation.ReLU                                   | torch.nn.modules.activation.ReLU                                   | torch.Size([1, 3, 32, 32]) | qint8         | 0.0056401 | 0.9999741 | 0.0009586 | 0.0088447 |         1.5681799 |        0.0000000 |         0.0000000 |        0.5436335 |         0.5414499 |         0.1555644 |          0.1557564 |                       |                       |           |
    |  3 |            | horizon_plugin_pytorch.nn.interpolate.autocasted_interpolate_outer | horizon_plugin_pytorch.nn.interpolate.autocasted_interpolate_outer | torch.Size([1, 3, 41, 41]) | qint8         | 0.0056401 | 0.9924216 | 0.0160291 | 0.2094204 |        37.1305954 |        0.0000000 |         0.0000000 |        0.5149657 |         0.5301697 |         0.1550578 |          0.1559310 |                       |                       |           |
    |  4 | dequant    | torch.ao.quantization.stubs.DeQuantStub                            | horizon_plugin_pytorch.nn.qat.stubs.DeQuantStub                    | torch.Size([1, 3, 41, 41]) | torch.float32 |           | 0.9924216 | 0.0160291 | 0.2094204 |                   |        0.0000000 |         0.0000000 |        0.5149657 |         0.5301697 |         0.1550578 |          0.1559310 |                       |                       |           |
    +----+------------+--------------------------------------------------------------------+--------------------------------------------------------------------+----------------------------+---------------+-----------+-----------+-----------+-----------+-------------------+------------------+-------------------+------------------+-------------------+--------------------+--------------------+-----------------------+-----------------------+-----------+

• compare_per_layer_out_0.csv: show the specific information of each layer in csv format. The content is exactly the same as the txt file, and the csv file format is convenient for you to open and analyze by excel and other software.

#sensitivity

def sensitivity(
    self,
    metric: Union[str, _Metric] = Metric.ATOL,
    prefixes: Optional[Tuple[str, ...]] = None,
    types: Optional[Tuple[Type, ...]] = None,
    output_names: Optional[Tuple[str, ...]] = None,
    use_sensitivity_v2: bool = False,
    sensitivity_method: str = "ablation",
):

Sensitivity ordering of individual nodes in the model. Applies to the float conversion to calibration accuracy dropout problem.

Parameters

• metric: metric for similarity ordering, default is Metric.ATOL. Available values are described in Metric Description.

• prefixes: the prefix of ops to run sensitivity. Default None, run all ops.

• types: the types of ops to run sensitivity. Default None, run all ops. When both prefixes and types are specified, only ops that satisfy both conditions participate in the computation.

• output_names: only run sensitivity of specified outputs. Note that you need to use the output names shown in badcase_report.html. Default None, run sensitivity of all outputs.

• use_sensitivity_v2: whether to use the v2 sensitivity, which is recommended to be set to True. Once enabled, besides the single-node quantization sensitivity, the round / clip errors are additionally split out, and the sensitivity of the node under qint8 and qint16 as well as the difference between them are given, so that you can directly judge how much gain promoting this node to int16 can bring. It only supports analysis_model_type="fake_quant", so it cannot be used in consistency analysis scenarios.

• sensitivity_method: the sensitivity computation method. It is only available when use_sensitivity_v2=True and analysis_model_type="fake_quant".

  • "ablation": the default method, which changes the state of the fake stubs one by one and runs one forward each time to observe the output error. It is slower, but generally does not run out of GPU memory.

  • "grad_x_error": estimates the sensitivity with gradient x error, which only requires one forward plus one backward and is very fast, but has a higher requirement on GPU memory. This method requires the scope of metric to be "all", and the metric must return a scalar tensor that can be backpropagated. If a custom metric returns a non-tensor, a non-scalar tensor or a non-differentiable tensor, an error is raised at runtime.

Output

The result files are generated separately for each output and each badcase, named output_{out_name}_case_{index}_sensitive_ops.txt / .csv / .pt, where out_name is the output name and index is the index of this badcase in data_generator.

Besides the result of each individual badcase, two summary tables and one HTML report are also generated:

• output_{out_name}_case_mean_sensitive_ops.* and output_{out_name}_case_worst_sensitive_ops.*: when topk > 1, these are the summary tables obtained by taking the mean value and the worst value over the multiple badcase results of that output. Compared with the result of a single badcase, the summary tables are usually more stable and are more suitable as the basis for configuring mixed precision.

• sensitivity_v2_report.html: Display the sensitivity results as an HTML page, including the ranked bar charts of each metric and detailed data tables. The page also contains the mean and worst sensitivity tables, and the columns can be sorted interactively. Through the trend after sorting, you can decide from which op in the ranking to start setting qint8 to qint16.

It is recommended to check the csv first, so that you can analyze the accuracy problem by combining the sensitivity of different cases, the mean sensitivity and the worst sensitivity together.

The ops in the file are organized in order of quantization sensitivity from highest to lowest. The main columns mean:

• op_name: op name.

• op_type: op type.

• fake_stub_type: Type of calculating quantization sensitivities, including:

  • output: quantization sensitivity to quantize only the output of this op.

  • weight: quantization sensitivity of the weight of the op only.

  • input-{n}：quantization sensitivity of the nst input of the op only.

• metric columns: expanded into multiple columns in the form of {quantization method}_{metric}, where metric is the name of the metric you passed in, such as qint8_ATOL, qint16_ATOL, qint8-qint16_ATOL, round_ATOL, clip_ATOL and fp16_ATOL. The value range and sort direction of each metric are described in Metric Description. Among them:

  • round / clip: the sensitivity when only the rounding error or only the clipping error is kept, respectively, used to judge which kind of error the node error mainly comes from.

  • qint8 / qint16 / fp16: the sensitivity of the node under the corresponding dtype.

  • qint8-qint16: the difference between the two. The larger the value, the larger the gain of promoting this node from int8 to int16. The results are sorted by this column by default.

• dtype: the quant dtype of this op output. Usually qint8/qint16/torch.float16.

• flops: the flops of this op and its proportion in the whole model.

Return Value

A dict whose key is the output name and whose value is a list of SensitivityTable objects, one table per badcase. SensitivityTable can be re-sorted through sort_by_column.

An example of the result table is as follows (some metric columns are omitted):

op_name           fake_stub_type  dtype  flops         clip_ATOL    round_ATOL   qint8_ATOL   qint16_ATOL  qint8-qint16_ATOL
----------------  --------------  -----  ------------  -----------  -----------  -----------  -----------  -----------------
conv              output          qint8  5184(98.18%)  0.045126150  0.066827238  0.066827238  0.000211666  0.066615572
conv              weight          qint8  5184(98.18%)  0.000000358  0.011085004  0.011085004  0.000082999  0.011002004
quant             output          qint8  0(0%)         0.000243107  0.008408447  0.008408447  0.000040362  0.008368085
_generated_add_1  output          qint8  0(0%)         0.276775519  0.004773577  0.276775519  0.271611373  0.005164146

You can configure the top n ops in the qint8-qint16 column as int16 to try to improve the quantized model accuracy.

#tune_bad_case

def tune_bad_case(
    self,
    outputs_config: Dict[str, Any],
    skip_prefix: Optional[Dict[str, List[str]]] = None,
):

Based on the sensitivity v2 results, simulate promoting qint8 to qint16 in sensitivity order on the current badcase, and observe the improvement of the cumulative error. It solves the problem that "after getting the sensitivity table, you still have to build an experiment yourself to see the real gain".

The specific procedure is: read the SensitivityTable saved in output_{out_name}_case_{case}_sensitive_ops.pt, sort it by qint8-qint16_{metric}, and then try to temporarily change the candidate qint8 to qint16 one by one:

1. If this promotion improves the metric of the final output, the qint16 configuration is kept and the cumulative gain is recorded;

2. If there is no gain, the original qint8 dtype and the original scale are restored;

3. The search stops after threshold or max_tuned_stubs is reached.

Parameters

• outputs_config: the search parameters configured per output, whose key is the output name. It is the empty string "" for a single-output model, usually "0" and "1" for a multi-output model, and "all" when a metric with scope="all" is used. The configuration of each output contains:

  • metric: the metric that measures the badcase.

  • threshold: the search stops after the metric reaches this threshold.

  • max_tuned_stubs: at most how many nodes are adjusted.

• skip_prefix: skip the ops with the specified prefixes, whose key is the op name prefix and whose value is the list of fake_stub_type to be skipped, such as {"backbone": ["output", "weight"]}.

from horizon_plugin_profiler.utils.entities import Metric

# The v2 sensitivity must be run first
qa.sensitivity(metric=Metric.ATOL, use_sensitivity_v2=True)

qa.tune_bad_case(
    outputs_config={
        "0": {
            "metric": Metric.ATOL,
            "threshold": 0.2,
            "max_tuned_stubs": 10,
        },
    },
    skip_prefix={"backbone": ["output", "weight"]},
)

Output

• output_{out_name}_tuned_case_{index}_sensitive_ops.txt / .csv / .pt: a tuned_cumulative_{metric} column is added on top of the sensitivity table, recording the metric change of the model output after each adjustment. The rows with a value are exactly the nodes that do bring an improvement after being promoted to int16.

#clean

def clean(self)

Clears intermediate results. Only files such as comparison results are retained.

#ModelProfiler Class

Statistic about the inputs and outputs of each layer of operators in the forward process of the model.

# from horizon_plugin_profiler import ModelProfiler

class ModelProfiler(object):
    def __init__(
        self,
        model: torch.nn.Module,
        out_dir: str,
    )

Parameters

• model: model that requires statistics.

• out_dir: path where the associated file is saved.

with ModelProfiler(net, "./profiler_dir") as p:
    net(data)

p.get_info_manager().table()
p.get_info_manager().tensorboard()

The methods in this class are as follows.

#get_info_manager

def get_info_manager(self)

Get the structure that manages the information for each op.

Return Value

The structure OpRunningInfoManager manages the information stored for each op. Two of the important interfaces are as follows.

#table

class OpRunningInfoManager:
    def table(
        self,
        out_dir: str = None,
        prefixes: Tuple[str, ...] = None,
        types: Tuple[Type, ...] = None,
        with_stack: bool = False,
    )

Show individual model statistics in a table. Store to the statistic.txt file.

Parameters

• out_dir: storage path of statistic.txt file, default None, store to self.out_dir.

• prefixes: prefixes of ops in the model to be counted. Default is all ops.

• types: types of the ops in the model to be counted, defaults to all ops.

• with_stack: if or not show the position of each op in the code.

Output

statistic.txt file, each column from left to right reads.

• Index: op index.

• Op Name: op type, module class name or function name.

• Mod Name: if it is module class, the prefix name of the module in the model; if it is function type, the prefix name of the module where the function is located.

• Attr: input/output/weight/bias.

• Dtype: data type of the tensor.

• Scale: scale of the tensor.

• Min: minimum value of the current tensor.

• Max: maximum value of the current tensor.

• Mean: average value of the current tensor.

• Var: variance of the values in the current tensor.

• Shape: tensor shape.

+---------+--------------------------------------------------------------------+------------+--------+---------------+-----------+------------+-----------+-----------+-----------+----------------------------+
| Index   | Op Name                                                            | Mod Name   | Attr   | Dtype         | Scale     | Min        | Max       | Mean      | Var       | Shape                      |
|---------+--------------------------------------------------------------------+------------+--------+---------------+-----------+------------+-----------+-----------+-----------+----------------------------|
| 0       | horizon_plugin_pytorch.nn.qat.stubs.QuantStub                      | quant      | input  | torch.float32 |           | 0.0000797  | 0.9998382 | 0.4980516 | 0.0823590 | torch.Size([1, 3, 32, 32]) |
| 0       | horizon_plugin_pytorch.nn.qat.stubs.QuantStub                      | quant      | output | qint8         | 0.0078419 | 0.0000000  | 0.9959173 | 0.4980148 | 0.0823754 | torch.Size([1, 3, 32, 32]) |
| 1       | horizon_plugin_pytorch.nn.qat.conv2d.Conv2d                        | conv       | input  | qint8         | 0.0078419 | 0.0000000  | 0.9959173 | 0.4980148 | 0.0823754 | torch.Size([1, 3, 32, 32]) |
| 1       | horizon_plugin_pytorch.nn.qat.conv2d.Conv2d                        | conv       | weight | torch.float32 |           | -0.5051875 | 0.5390092 | 0.0225839 | 0.1342911 | torch.Size([3, 3, 1, 1])   |
| 1       | horizon_plugin_pytorch.nn.qat.conv2d.Conv2d                        | conv       | bias   | torch.float32 |           | 0.1814680  | 0.4194079 | 0.2941962 | 0.0142707 | torch.Size([3])            |
| 1       | horizon_plugin_pytorch.nn.qat.conv2d.Conv2d                        | conv       | output | torch.float32 |           | -0.3565997 | 1.1367939 | 0.3289838 | 0.0682679 | torch.Size([1, 3, 32, 32]) |
| 2       | horizon_plugin_pytorch.nn.qat.relu.ReLU                            | relu       | input  | torch.float32 |           | -0.3565997 | 1.1367939 | 0.3289838 | 0.0682679 | torch.Size([1, 3, 32, 32]) |
| 2       | horizon_plugin_pytorch.nn.qat.relu.ReLU                            | relu       | output | qint8         | 0.0089490 | 0.0000000  | 1.1365263 | 0.3404330 | 0.0589806 | torch.Size([1, 3, 32, 32]) |
| 3       | horizon_plugin_pytorch.nn.interpolate.autocasted_interpolate_outer |            | input  | qint8         | 0.0089490 | 0.0000000  | 1.1365263 | 0.3404330 | 0.0589806 | torch.Size([1, 3, 32, 32]) |
| 3       | horizon_plugin_pytorch.nn.interpolate.autocasted_interpolate_outer |            | output | qint8         | 0.0089490 | 0.0000000  | 1.1365263 | 0.3395261 | 0.0588461 | torch.Size([1, 3, 40, 40]) |
| 4       | horizon_plugin_pytorch.nn.qat.stubs.DeQuantStub                    | dequant    | input  | qint8         | 0.0089490 | 0.0000000  | 1.1365263 | 0.3395261 | 0.0588461 | torch.Size([1, 3, 40, 40]) |
| 4       | horizon_plugin_pytorch.nn.qat.stubs.DeQuantStub                    | dequant    | output | torch.float32 |           | 0.0000000  | 1.1365263 | 0.3395261 | 0.0588461 | torch.Size([1, 3, 40, 40]) |
+---------+--------------------------------------------------------------------+------------+--------+---------------+-----------+------------+-----------+-----------+-----------+----------------------------+

#tensorboard

class OpRunningInfoManager:
    def tensorboard(
        self,
        out_dir: str = None,
        prefixes: Tuple[str, ...] = None,
        types: Tuple[Type, ...] = None,
        force_per_channel: bool = False,
    ):

Show the input and output histograms for each layer in the tensorboard.

Parameters

• out_dir: directory where tensorboard related files are kept. Default is self.out_dir/tensorboard.

• prefixes: prefixes of the ops in the model to be counted, default is all.

• types: types of the ops in the model to be counted, default is all.

• force_per_channel: if or not to display the histogram in per_channel quantization.

Output

The tensorboard file, opened with the following screenshot.

#HbirModelProfiler Class

The functionality and usage of this class is identical to the ModelProfiler class. Please refer to ModelProfiler Class for usage.

with HbirModelProfiler(qat_hbir, "./hbir_dir") as p:
    qat_hbir[0](data)

p.get_info_manager().table()
p.get_info_manager().tensorboard()