Quantization and Dequantization Introduction

In order to help you better understand the quantize and dequantize pre-processing provided by the DSP sample package, this article analyzes the CPU and DSP reference implementations based on the algorithm principles, and performs consistency checks and performance evaluations on the CPU and DSP reference implementations.

Principle Introduction

Quantization and Dequantization

For the introduction, formulas and reference implementation of quantization and dequantization, please refer to the introduction of model quantization in the Key Concepts section.

CPU Reference Implementation

Quantification

static float32_t _round(float32_t input) {
  std::fesetround(FE_TONEAREST);
  float32_t result = nearbyintf(input);
  return result;
}

/**
 * quantize reference implementation
 * @param[out] dst: dst data
 * @param[in] src: src size
 * @param[in] scale: scale value
 * @param[in] zero_point: zero_point value
 * @param[in] size: data size
 * @return 0 if success, return defined error code otherwise
 */
static void quantize_ref(int8_t *dst, const float32_t *src, float32_t scale,
                         float32_t zero_point, float32_t min, float32_t max,
                         int32_t size) {
  for (int32_t i = 0; i < size; ++i) {
    float32_t value = _round(src[i] / scale + zero_point);
    value = std::min(std::max(value, min), max);
    dst[i] = static_cast<int8_t>(value);
  }
}

Dequantization

/**
 * dequantize reference implementation
 * @param[out] dst: dst data
 * @param[in] src: src size
 * @param[in] scale: scale value
 * @param[in] zero_point: zero_point value
 * @param[in] type: data type
 * @param[in] size: data size
 * @return 0 if success, return defined error code otherwise
 */
static void dequantize_ref(float32_t *dst, void *src, float32_t scale,
                           int32_t zero_point, int32_t size, uint8_t type) {
  if (type == HB_DSP_TENSOR_TYPE_S32) {
    const int32_t *ptr = static_cast<int32_t *>(src);
    for (int32_t i = 0; i < size; ++i) {
      int32_t data_diff{ptr[i] - zero_point};
      dst[i] = static_cast<float32_t>(data_diff) * scale;
    }
  } else if (type == HB_DSP_TENSOR_TYPE_S8) {
    for (int32_t i = 0; i < size; ++i) {
      const int8_t *ptr = static_cast<int8_t *>(src);
      int32_t data_diff{(int32_t)(ptr[i]) - zero_point};
      dst[i] = static_cast<float32_t>(data_diff) * scale;
    }
  } else {
    LOGE("Unsupported data type: {}", static_cast<int32_t>(type));
  }
}

DSP Optimization Acceleration

DSP Acceleration Ideas

Vectorized

Quantize/dequantize both calculate data continuously, which is very suitable for SIMD vector optimization.

Block strategy

J6 DSP has two 256kB TCMs, and the actual available space is about 210KB. Each tile is as large as possible without exceeding the total available space. Quantize mainly considers input and output, and the ratio is 4:1. Dequantize also mainly considers input and output, and the ratio is 1:4;
At the same time, because quantize is a linear calculation, and the output is smaller than the output data size, you can consider using inplace (using the same tile for input and output to further increase the available tile).

DSP Optimization Implementation

quantize SIMD computing.

#define QUANTIZE_PROCESS                                                       \
  IVP_LAN_2XF32_IP(input1, al_px, INPUT);                                      \
  IVP_LAN_2XF32_IP(input2, al_px, INPUT);                                      \
  /* Compute: input * (1/scale) + offset + 0.5f */                             \
  xb_vecN_2xf32 hfvec1 = v_zero_point_0;                                       \
  xb_vecN_2xf32 hfvec2 = v_zero_point_1;                                       \
  IVP_MULAN_2XF32(hfvec1, input1, v_scale_0);                                  \
  IVP_MULAN_2XF32(hfvec2, input2, v_scale_1);                                  \
  /* Round and Clamp the vectors to U8/S8 range */                             \
  xb_vecN_2x32v hvec1 = IVP_TRUNCN_2XF32(                                      \
      IVP_MAXN_2XF32(IVP_MINN_2XF32(IVP_FIRINTN_2XF32(hfvec1), max), min), 0); \
  xb_vecN_2x32v hvec2 = IVP_TRUNCN_2XF32(                                      \
      IVP_MAXN_2XF32(IVP_MINN_2XF32(IVP_FIRINTN_2XF32(hfvec2), max), min), 0); \
  xb_vecNx16 output = IVP_MOVNX16_FROMN_2X32(                                  \
      IVP_SELN_2X32I(hvec2, hvec1, IVP_SELI_8B_EXTRACT_1_OF_2_OFF_0));		   \
	  IVP_SANX8S_IP(output, al_pz, OUTPUT);

dequantize SIMD Computing.

#define DEQUANTIZE_PROCESS_INT8                                              \
  xb_vecNx16 input1;                                                         \
  IVP_LANX8S_IP(input1, al_px, INPUT);                                       \
  input1 = IVP_SUBNX16(input1, v_zero_point);                                \
  xb_vecN_2x32v ll, lh;                                                      \
  ll = IVP_UNPKSNX16_L(input1);                                              \
  lh = IVP_UNPKSNX16_H(input1);                                              \
  xb_vecN_2xf32 output1 = IVP_MULN_2XF32(IVP_FLOATN_2X32(ll, 0), v_scale_0); \
  xb_vecN_2xf32 output2 = IVP_MULN_2XF32(IVP_FLOATN_2X32(lh, 0), v_scale_1);

For the framework part, please refer to the introduction of ping pong IDMA implementation in the DSP Operator section.

Consistency Check

Compare the calculated results with the CPU reference implementation, and pass the verification if they are consistent

#Quantization and Dequantization Introduction

#Principle Introduction