#Video Codec

#Principle

The principle of video coding is the elimination of redundant information and compression of video signals to reduce the amount of data and facilitate storage and transmission. The implementation of video coding usually relies on specific video coding standards, such as H.264, H.265, and so on. These standards specify specific coding algorithms and parameters for efficient video compression and transmission.

#Video Encoding Principle

Taking the H.265 coding protocol as an example, video coding involves the following key steps:

1. Segmentation

H.265 first divides the video into a number of sequences, and a sequence is divided into a number of Groups of Picture (GOP), each GOP represents a set of consecutive video frames.

2. Spatial prediction

H.265 uses spatial prediction to remove redundant information between image blocks. As shown in the figure below, from a spatial point of view, the difference in pixel values between pixel points inside a single video frame is very small. From a temporal perspective, there are also many identical pixel points between two consecutive video frames.

Spatial Sampling Schematic Image	Time Sampling Schematic Image

Predictive coding is a method of data compression based on the statistical properties of an image that utilizes the temporal and spatial correlation of an image to predict the pixels that are currently being coded from the pixel data that has been reconstructed.

Intra-frame prediction means that both the pixels used for prediction and the pixels currently being encoded are within the same video frame and are generally in neighboring regions. Due to the strong correlation between neighboring pixels, the pixel values are generally very close to each other, and the probability of a mutation is very small, with the differences all being zero or very small numbers. Therefore, what is transmitted after intra-frame prediction coding is the difference between the predicted value and the true value, i.e., the value around 0, which is called the prediction error or residual. This enables compression to be achieved with fewer bits transmitted.

H.265 intra-frame predictive coding is done on a block basis, using the rebuilt values of neighboring blocks that have been rebuilt for the block being coded. The prediction component is divided into two, luminance and chrominance, and the corresponding prediction blocks are luminance prediction block and chrominance prediction block, respectively. In order to adapt to the content characteristics of HD video and improve the prediction accuracy, H.265 adopts richer prediction block sizes and prediction modes.

Inter-frame prediction means that the pixels used for prediction and the pixels currently being encoded are not in the same video frame, but are generally in adjacent or nearby locations. In general, inter-frame predictive coding provides better compression than intra-frame prediction, mainly because of the very strong correlation between video frames. If the rate of change of moving objects in a video frame is slow, then the pixel differences between video frames are small and the temporal redundancy is very large.

The method of inter-frame prediction to evaluate the motion condition of a moving object is motion estimation, and its main idea is to search for a matching block for the prediction block from a given range of the reference frame, and calculate the relative displacement between the matching block and the prediction block, and this relative displacement is the motion vector. After obtaining the motion vector, the prediction needs to be corrected, also known as motion compensation. Input the motion vectors into the motion compensation module and "compensate" the reference frame to get the prediction frame of the current encoded frame. The difference between the predicted frame and the current frame is the inter-frame prediction error.

If the inter-frame prediction only uses the previous frame image, it is called forward inter-frame prediction or unidirectional prediction. This predicted frame is also known as a P-frame, and the P-frame can refer to the previous I-frame or P-frame.

Inter-frame prediction is bidirectional if it uses not only the previous frame image to predict the current block, but also the subsequent frame image. This prediction frame is also known as the B-frame, which can refer to the preceding I-frame or P-frame and the following P-frame. Since P-frames need to refer to previous I-frames or P-frames, and B-frames need to refer to previous I-frames or P-frames and later P-frames, if in a video stream, the B-frame arrives first, and the dependent I-frames and P-frames haven't arrived yet, then the B-frame can't be decoded immediately, so what should be done to ensure the playback order?

Actually, PTS and DTS are generated during video encoding. Typically, after generating an I-frame, the encoder skips backward a few frames and encodes a P-frame using the preceding I-frame as a reference frame, and the frame between the I-frame and the P-frame is encoded as a B-frame. The video frame order of the push stream is already coded at the time of encoding in the dependent order of I-frames, P-frames, and B-frames, and the data is directly decoded upon receipt. Therefore, it is not possible to receive B-frames first and then dependent I-frames and P-frames.

Schematic Image

3. Transformation and quantification

In H.265, transform and quantization are used to further compress the data. By transforming the prediction error, the data can be converted from the time domain to the frequency domain for better removal of data redundancy. The transformed data is then quantized to map the data to a lower precision, thus further compressing the data. The process can be referred to the JPEG coding process.

4. Entropy encoding

In the final step of the encoding process, H.265 uses entropy coding to losslessly compress the data. The main purpose of entropy coding is to minimize the redundancy of the coded data in order to improve the data compression efficiency. The process can be referred to JPEG coding process.

#Video Decoding Principle

Video decoding refers to the process of converting compressed video data back to the original video format. The process of video decoding is mainly divided into entropy decoding, inverse quantization, inverse transformation, motion compensation and deconvolution, and post-processing, each of which is designed to recover as close as possible to the original video picture from the highly compressed data. Since the goal of video encoding is to minimize the file size, the decoding process must perform each step of the encoding process precisely in reverse to recover the video content:

1. Entropy decoding

Entropy decoding is the process of converting compressed data back into a more manageable video format. In the process of video encoding, entropy coding techniques such as Hoffman coding or arithmetic coding are usually used to reduce the amount of data. And the purpose of entropy decoding is to recover the symbols used in the video coding process to prepare for the next step of inverse quantization.

2. Inverse quantization

Inverse quantization is the process of flipping the quantization (the step in the encoding process that reduces the precision of the data in order to achieve space savings) in order to restore the precision of the original data, a step that is critical to restoring image quality.

3. Inverse transform

Inverse tranform is the process of reversing the changes used in coding (e.g., the Discrete Cosine Transform, DCT) in order to restore the data from the transform domain (e.g., the frequency domain) to the spatial domain (i.e., the original image), and this step is a key step in image reconstruction.

4. Motion compensation & deblocking

Motion compensation means that, for predicted frames (i.e., frames generated based on the previous/next frame), the complete frame needs to be reconstructed using the run vector data. Deconvolution, on the other hand, is the process of removing block artifacts generated during the coding process. These steps are essential to restore smooth and consistent video playback.

5. Post-processing

Post-processing, as the final step, involves some techniques to enhance video quality such as denoising, sharpening, etc. This step is optional, and whether or not it is carried out depends on the requirements of video playback and whether or not the hardware capability can support it.

#API Interface

#Video Encoding Interface

// Get the default parameter for video encoding
int32_t hbVPGetDefaultVideoEncParam(hbVPVideoEncParam *param);

// Create the video encoding context
int32_t hbVPCreateVideoEncContext(hbVPVideoContext *context,
                                  hbVPVideoEncParam const *param);

// Release the video encoding context
int32_t hbVPReleaseVideoEncContext(hbVPVideoContext context);

// Create video encoding task
int32_t hbVPVideoEncode(hbUCPTaskHandle_t *taskHandle, 
                        hbVPImage const *srcImg,
                        hbVPVideoContext context);

// Get the output buffer where the encoding data is stored
int32_t hbVPGetVideoEncOutputBuffer(hbVPArray *outBuf,
                                    hbUCPTaskHandle_t taskHandle);                             

#Video Decoding Interface

// Get the default parameter for video decoding
int32_t hbVPGetDefaultVideoDecParam(hbVPVideoDecParam *param);

// Create the video decoding context
int32_t hbVPCreateVideoDecContext(hbVPVideoContext *context,
                                  hbVPVideoDecParam const *param);

// Release the video decoding context
int32_t hbVPReleaseVideoDecContext(hbVPVideoContext context);

// Create video decoding tasks
int32_t hbVPVideoDecode(hbUCPTaskHandle_t *taskHandle, 
                        hbVPArray const *srcBuf,
                        hbVPVideoContext const context);

// Get the output buffer where the decoding data is stored
int32_t hbVPGetVideoDecOutputBuffer(hbVPImage *outImg,
                                    hbUCPTaskHandle_t taskHandle);                             

For detailed interface information, please refer to hbVPVideoEncode and hbVPVideoDecode.

#Usage

#Video Encoding Usage

// Include the header
#include "hobot/hb_ucp.h"
#include "hobot/vp/hb_vp.h"
#include "hobot/vp/hb_vp_video_codec.h"

// init encoding param
hbVPVideoEncParam venc_param;
venc_param.videoType = HB_VP_VIDEO_TYPE_H265;

// default set backend to HB_UCP_VPU_CORE_0
hbVPGetDefaultVideoEncParam(&venc_param);
venc_param.height = height;
venc_param.width = width;
venc_param.pixelFormat = image_format;

// create video encoding context and the Video Process Unit internally alloc the output buffer
hbVPVideoContext enc_context{nullptr};
hbVPCreateVideoEncContext(&enc_context, &venc_param)

// init schedule param
hbUCPSchedParam sched_param;
HB_UCP_INITIALIZE_SCHED_PARAM(&sched_param);
sched_param.backend = HB_UCP_VPU_CORE_0;

// init image_buf, alloc memory for image data
hbUCPSysMem image_mem;
hbUCPMalloc(&image_mem, yuv_size, 0);

{
  hbVPImage image_buf;
  image_buf.imageFormat = image_format;
  image_buf.imageType = image_type;
  image_buf.width = width;
  image_buf.height = height;
  image_buf.dataVirAddr = image_mem.virAddr;
  image_buf.dataPhyAddr = image_mem.phyAddr;
  image_buf.uvVirAddr = reinterpret_cast<char *>(image_mem.virAddr) + y_size;
  image_buf.uvPhyAddr = image_mem.phyAddr + y_size;

  // init task handle
  hbUCPTaskHandle_t task_handle{nullptr};

  // create video encoding task
  hbVPVideoEncode(&task_handle, &image_buf, enc_context);

  // submit video encoding task
  hbUCPSubmitTask(task_handle, &sched_param);

  // wait for video encoding task done
  hbUCPWaitTaskDone(task_handle, 10);

  // get output buffer from Video Process Unit
  hbVPArray out_buf;
  hbVPGetVideoEncOutputBuffer(&out_buf, task_handle);

  // process h264 or h265 data

  // release task handle and dequeue output buffer from Video Process Unit
  // since the dequeue output buffer process relies on the context,
  // you must call the hbUCPReleaseTask interface before calling the hbVPReleaseJPEGEncContext interface
  hbUCPReleaseTask(task_handle);
}

// release memory
hbUCPFree(&image_mem);

// release encoding context and the Video Process Unit internally release the output buffer
hbVPReleaseVideoEncContext(enc_context);

#Video Decoding Usage

// Include the header
#include "hobot/hb_ucp.h"
#include "hobot/vp/hb_vp.h"
#include "hobot/vp/hb_vp_video_codec.h"

// init decoding param
hbVPVideoDecParam vdec_param;
vdec_param.videoType = HB_VP_VIDEO_TYPE_H265;

// default set backend to HB_UCP_VPU_CORE_0
hbVPGetDefaultVideoDecParam(&vdec_param);
vdec_param.pixelFormat = HB_VP_IMAGE_FORMAT_YUV420;
vdec_param.inBufSize = ALIGN_1024(height * width * 3 / 2);

// create video decoding context and the Video Process Unit internally alloc the output buffer
hbVPVideoContext dec_context{nullptr};
hbVPCreateVideoDecContext(&dec_context, &vdec_param)

// init schedule param
hbUCPSchedParam sched_param;
HB_UCP_INITIALIZE_SCHED_PARAM(&sched_param);
sched_param.backend = HB_UCP_VPU_CORE_0;

// alloc larger memory for input
hbUCPSysMem src_mem;
ret = hbUCPMalloc(&(src_mem), vdec_param.inBufSize, 0);

{
  // init src_buf, alloc memory for image data
  memset(src_mem.virAddr, src_mem.memSize, 1);
  fread(src_mem.virAddr, 1, feedSize, inFile);
  
  hbVPArray src_buf;
  src_buf.phyAddr = src_mem.phyAddr;
  src_buf.virAddr = src_mem.virAddr;
  src_buf.memSize = feedSize;
  src_buf.size = feedSize;

  // init task handle
  hbUCPTaskHandle_t task_handle{nullptr};

  // create video decoding task
  hbVPVideoDecode(&task_handle, &src_buf, dec_context);

  //  submit video decoding task
  hbUCPSubmitTask(task_handle, &sched_param);

  // wait for video decoding task done
  hbUCPWaitTaskDone(task_handle, 10);

  // get output buffer from Video Process Unit
  hbVPImage out_img;
  hbVPGetVideoDecOutputBuffer(&out_img, task_handle);

  // process yuv data

  // release task handle and dequeue output buffer from Video Process Unit
  // since the dequeue output buffer process relies on the context,
  // you must call the hbUCPReleaseTask interface before calling the hbVPReleaseJPEGDecContext interface
  hbUCPReleaseTask(task_handle);
}

// release memory
hbUCPFree(&src_mem);

// release encoding context and the Video Process Unit internally release the output buffer
hbVPReleaseVideoDecContext(dec_context);

#Additional Notes

#Bit Rate Control Mode

The encoder supports both H.264 and H.265 standards, offering five bitrate control modes: CBR (Constant Bitrate), VBR (Variable Bitrate), AVBR (Adaptive Variable Bitrate), FixQp, and QpMap. These modes ensure flexibility in balancing bitrate stability, image quality, and compression efficiency. For example, CBR maintains a consistent overall bitrate, while VBR focuses on stable image quality. FixQp keeps the quantization parameter (QP) fixed for specific frames, and QpMap assigns QP values to blocks within frames, with H.265 using 32x32 blocks and H.264 using 16x16. The following explanation uses H.265 as an example for rate control settings.

1. Constant Bit Rate(CBR)

CBR (Constant Bitrate) refers to maintaining a stable bitrate throughout the encoding process, ensuring that the overall bitrate of the video stream remains constant. The following are the meanings of the parameters used in CBR mode:

2. Variable Bit Rate(VBR)

VBR (Variable Bitrate) dynamically adjusts the bitrate during the encoding process based on the complexity of the scene. In simple scenes, a larger QP (Quantization Parameter) is assigned to achieve higher compression. In complex scenes, a smaller QP is allocated to maintain stable image quality. The following are the meanings of the parameters used in VBR mode:

3. Average Variable Bit Rate(AVBR)

AVBR (Adaptive Variable Bitrate) is a strategy that maintains a stable average bitrate throughout the encoding process. It combines the advantages of both CBR and VBR: in simple scenes, a lower bitrate is allocated, while in complex scenes, enough bitrate is provided to ensure image quality. This ensures that the bitrate is reasonably distributed across different scenes. At the same time, AVBR ensures that the average bitrate approaches the preset target bitrate over a certain period, helping to control the output file size. AVBR can be viewed as a compromise between CBR and VBR, resulting in a stream with relatively stable bitrate and image quality. The following are the meanings of the parameters used in AVBR mode:

4. Fix QP

In FixQp (Fixed QP) mode, the encoder assigns a fixed Quantization Parameter (QP) value to each I-frame, P-frame, and B-frame without making any dynamic adjustments. This mode is typically used in scenarios where strict control over image quality and compression ratio is required. The following are the meanings of the parameters used in FixQp mode:

5. Qp Map

The QpMap mode allows different Quantization Parameter (QP) values to be assigned to each macroblock within a frame, enabling more fine-grained control over the bitrate. For H.265 encoding, the size of each macroblock is 32x32. The following are the meanings of the parameters used in QpMap mode:

#GOP Structure

H.264 and H.265 encoding support the configuration of GOP structures, allowing you to choose from preset GOP structures. Below are the descriptions of the preset GOP structures: