• English
  • 3.0.31

    • JPEG Codec

    Operator Effect

    Schematic Image
    VP5out_jpegcodec

    Principle

    JPEG (Joint Photographic Experts Group) is a widely used lossy compression standard, developed by the Joint Photographic Experts Group, and has become an internationally recognized standard for image compression. JPEG is not only applicable for static image encoding but has also been extended for intra-frame compression in television image sequences. The JPEG standard supported by JPU hardware is the Baseline Sequential mode of ISO/IEC 10918-1.

    JPEG Encoding Principle

    JPEG encoding, encode YUV format images into JPEG compressed format image files, such as *.jpg.

    Schematic Image
    VP5out_jpegencode

    JPEG Decoding Principle

    JPEG decoding, realize the decoding of .jpg, .jpeg, .JPG, .JPEG image files.

    Schematic Image
    VP5out_jpegdecode

    The main process of JPEG encoding is described by taking an 8x8 subregion of an image as an example.The values of the 8x8 image subregion are shown as belows:

    [52556166706164736359559010985697262596811314410466736358711221541067069676168104126886870796560707768587585716459556165838779696865767894]\begin{aligned}\begin{bmatrix}52 & 55 & 61 & 66 & 70 & 61 & 64 & 73\\ 63 & 59 & 55 & 90 & 109 & 85 & 69 & 72\\62 & 59 & 68 & 113 & 144 & 104 & 66 & 73\\63 & 58 & 71 & 122 & 154 & 106 & 70 & 69\\67 & 61 & 68 & 104 & 126 & 88 & 68 & 70\\79 & 65 & 60 & 70 & 77 & 68 & 58 & 75\\85 & 71 & 64 & 59 & 55 & 61 & 65 & 83\\87 & 79 & 69 & 68 & 65 & 76 & 78 & 94\\\end{bmatrix}\end{aligned}

    A two-dimensional Discrete Cosine Transform (DCT) is first performed on the 8x8 subregion with the aim of transforming the YUV color space into the frequency domain space. Since the range of values accepted for discrete cosine variation is [-128, 127], the following matrix is obtained by subtracting -128 from the values of the 8x8 image sub-region:

    [−76−73−67−62−58−67−64−55−65−69−73−38−19−43−59−56−66−69−60−1516−24−62−55−65−70−57−626−22−58−59−61−67−60−24−2−40−60−58−49−63−68−58−51−60−70−53−43−57−64−69−73−67−63−45−41−49−59−60−63−52−50−34]\begin{aligned}\begin{bmatrix} -76&-73&-67&-62&-58&-67&-64&-55\\ -65&-69&-73&-38&-19&-43&-59&-56\\-66&-69&-60&-15&16&-24&-62&-55\\-65&-70&-57&-6&26&-22&-58&-59\\-61&-67&-60&-24&-2&-40&-60&-58\\-49&-63&-68&-58&-51&-60&-70&-53\\-43&-57&-64&-69&-73&-67&-63&-45\\-41&-49&-59&-60&-63&-52&-50&-34\end{bmatrix}\end{aligned}

    The normalized matrix is subjected to DCT and the transformation formula:

    F(u,v)=14C(u)C(v)[∑m=07∑n=07f(m,n)cos(2m+1)uπ16cos(2n+1)vπ16]F(u,v)=\cfrac{1}{4}C(u)C(v)[\displaystyle\sum_{m=0}^{7} \displaystyle\sum_{n=0}^{7}f(m,n)cos\cfrac{(2m+1)u\pi}{16}cos\cfrac{(2n+1)v\pi}{16}]

    The DCT coefficient matrix is obtained after transformation:

    [−415−30−612756−20−204−22−611013−7−95−47777−25−29105−6−491234−15−1062212−7−13−4−22−33−832−6−2142−100−2−1−34−100−1−4−1012]\begin{aligned}\begin{bmatrix} -415&-30&-61&27&56&-20&-2&0\\ 4&-22&-61&10&13&-7&-9&5\\-47&7&77&-25&-29&10&5&-6\\-49&12&34&-15&-10&6&2&2\\12&-7&-13&-4&-2&2&-3&3\\-8&3&2&-6&-2&1&4&2\\-1&0&0&-2&-1&-3&4&-1\\0&0&-1&-4&-1&0&1&2\end{bmatrix}\end{aligned}

    The coefficients at the points (0, 0) are called the direct current components (DC coefficients) and the coefficients at the remaining 63 points are called the alternating current components (AC coefficients).

    Next, the DCT coefficients of the luminance and chrominance components are quantized, i.e., the DCT coefficients are divided by the quantization table and rounded to the nearest integer. Since the human eye is more sensitive to luminance signals than to chromaticity signals, two quantization scales, for the luminance component and the chromaticity component, are used. The default quantization tables are derived from extensive experiments, and custom quantization tables are also available.

    The default quantization table for the luminance component:

    [1611101624405161121214192658605514131624405769561417222951878062182237566810910377243555648110411392496478871031211201017292959811210010399]\begin{aligned}\begin{bmatrix} 16& 11& 10& 16& 24& 40& 51& 61 \\12& 12& 14& 19& 26& 58& 60& 55 \\14& 13& 16& 24& 40& 57& 69& 56\\14& 17& 22& 29& 51& 87& 80& 62\\18& 22& 37& 56& 68& 109& 103& 77\\24& 35& 55& 64& 81& 104& 113& 92\\49& 64& 78& 87& 103& 121& 120& 101\\72& 92& 95& 98& 112& 100& 103& 99\\\end{bmatrix}\end{aligned}

    The default quantization table for colorimetric components:

    [17182447999999991821266699999999242656999999999947669999999999999999999999999999999999999999999999999999999999999999999999999999]\begin{aligned}\begin{bmatrix} 17 & 18 & 24 & 47 & 99 & 99 & 99 & 99\\18 & 21 & 26 & 66 & 99 & 99 & 99 & 99\\24 & 26 & 56 & 99 & 99 & 99 & 99 & 99\\47 & 66 & 99 & 99 & 99 & 99 & 99 & 99\\99 & 99 & 99 & 99 & 99 & 99 & 99 & 99\\99 & 99 & 99 & 99 & 99 & 99 & 99 & 99\\99 & 99 & 99 & 99 & 99 & 99 & 99 & 99\\99 & 99 & 99 & 99 & 99 & 99 & 99 & 99\\\end{bmatrix}\end{aligned}

    The quantized DCT coefficients can be obtained by dividing the DCT coefficient matrix obtained earlier with the luminance table of the default values and rounding up:

    [−26−3−622−1000−2−411000−315−1−1000−412−1000010000000000000000000000000000000]\begin{aligned}\begin{bmatrix} -26 & -3 & -6 & 2 & 2 & -1 & 0 & 0\\0 & -2 & -4 & 1 & 1 & 0 & 0 & 0\\-3 & 1 & 5 & -1 & -1 & 0 & 0 & 0\\-4 & 1 & 2 & -1 & 0 & 0 & 0 & 0\\1 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\\end{bmatrix}\end{aligned}

    Observe that the quantized DCT coefficients of the quantized data are a bit larger relative to the AC coefficients and that the AC coefficients kind of contain a large number of zeros. Therefore, using Z-shaped encoding allows you to concatenate a large number of zeros together to reduce the size after encoding. The main idea is to organize the quantized DCT coefficients in a zigzag pattern starting from the first pixel in the upper left corner of the quantized DCT coefficients:

    Schematic Image
    VP5out_jpegzigzag

    Since the DC coefficients of the DCT coefficients after zigzag coding have large values and the DC coefficients of the neighboring 8x8 image regions do not vary much, therefore, differential pulse coding technique is used to encode the difference of DC coefficients between neighboring image regions; for characters with AC coefficients that are repeated and occur many times in a row, the stroke length coding is used. Both encoding methods have intermediate formats that are intended to further minimize storage.

    After obtaining the intermediate format of DC coefficients and the intermediate format of AC coefficients, entropy coding of both is required for further compression of image data. Compression is achieved by encoding characters with a higher probability of occurrence with a smaller number of bits. The JPEG basic system specifies the Huffman coding method. Different Huffman coding tables are used for DC coefficient and AC coefficient for Huffman coding, and different Huffman coding tables are used for luminance and chrominance. Therefore, 4 Huffman coding tables are needed to complete the entropy coding, and waiting until the specific Huffman coding is done efficiently by using table lookups. However, there is no default Huffman table defined in the JPEG standard, so you are free to choose one according to the actual application. It is also possible to use the Huffman table recommended by the JPEG standard, or to predefine a generic Huffman table. It is also possible to compute the value of the Huffman table for a particular image by collecting its statistical features before compression coding.

    API Interface

    JPEG Encoding Interface

    // create JPEG encoding context
int32_t hbVPCreateJPEGEncContext(hbVPJPEGContext *context,
                                 hbVPJPEGEncParam *param);

// JPEG encoding API interface
int32_t hbVPJPEGEncode(hbUCPTaskHandle_t *taskHandle, 
                       hbVPImage const *srcImg,
                       hbVPJPEGContext context);

// get JPEG encoding ouptput buffer
int32_t hbVPGetJPEGEncOutputBuffer(hbVPArray *outBuf, 
                                   hbUCPTaskHandle_t taskHandle);

// release JPEG encoding context
int32_t hbVPReleaseJPEGEncContext(hbVPJPEGContext context);

    JPEG Decoding Interface

    // create JPEG decoding context
int32_t hbVPCreateJPEGDecContext(hbVPJPEGContext *context, uint32_t outBufCount, uint8_t imageFormat);

// JPEG decoding API interface
int32_t hbVPJPEGDecode(hbUCPTaskHandle_t *taskHandle, 
                       hbVPArray const *srcBuf,
                       hbVPJPEGContext context);

// get JPEG decoding ouptput buffer
int32_t hbVPGetJPEGDecOutputBuffer(hbVPImage *outImg, 
                                   hbUCPTaskHandle_t taskHandle);

// release JPEG decoding context
int32_t hbVPReleaseJPEGDecContext(hbVPJPEGContext context);

    For detailed interface information, please refer to hbVPJPEGEncode and hbVPJPEGDecode.

    Usage

    JPEG Encoding Usage

    // Include the header
#include "hobot/hb_ucp.h"
#include "hobot/vp/hb_vp.h"
#include "hobot/vp/hb_vp_jpeg_codec.h"

// init image_buf, alloc memory for image data
hbUCPSysMem image_mem;
hbUCPMalloc(&image_mem, yuv_size, 0);
hbVPImage image_buf =  hbVPImage{HB_VP_IMAGE_FORMAT_YUV420,
                                 HB_VP_IMAGE_TYPE_U8C1,
                                 width,
                                 height,
                                 stride,
                                 image_mem.virAddr,
                                 image_mem.phyAddr,
                                 nullptr,
                                 0,
                                 0};

// init jpeg_buf
hbVPArray jpeg_buf{0};

// init encoding param
hbVPJPEGEncParam enc_param;
enc_param.qualityFactor = 50;
enc_param.extendedSequential = 0;
enc_param.width = width;
enc_param.height = height;
enc_param.imageFormat = image_format;
enc_param.outBufCount = 5;

// create encoding context and the JPU internally alloc the output buffer
hbVPJPEGContext enc_context{nullptr};
hbVPCreateJPEGEncContext(&enc_context, &enc_param);

// init task handle and schedule param
hbUCPTaskHandle_t task_handle{nullptr};
hbUCPSchedParam sched_param;
HB_UCP_INITIALIZE_SCHED_PARAM(&sched_param);
sched_param.backend = HB_UCP_JPU_CORE_0;
sched_param.priority = 0;

// create encoding task
hbVPJPEGEncode(&task_handle, &image_buf, enc_context);

// submit encoding task
hbUCPSubmitTask(task_handle, &sched_param);

// wait for encoding task done
hbUCPWaitTaskDone(task_handle, 0);

// get encoded buffer from JPU
hbVPGetJPEGEncOutputBuffer(&jpeg_buf, taskHandle);

// process jpeg data

// release task handle and dequeue output buffer from JPU
// since the dequeue output buffer process relies on the context,
// you must call the hbUCPReleaseTask interface before calling the hbVPReleaseJPEGEncContext interface
hbUCPReleaseTask(task_handle);

// release encoding context and the JPU internally release the output buffer
hbVPReleaseJPEGEncContext(enc_context);

// release memory
hbUCPFree(&image_mem);

    JPEG Decoding Usage

    // Include the header
#include "hobot/hb_ucp.h"
#include "hobot/vp/hb_vp.h"
#include "hobot/vp/hb_vp_jpeg_codec.h"

// init jpeg_buf, alloc memory for jpeg data
hbUCPSysMem jpeg_mem;
hbUCPMalloc(&jpeg_mem, jpeg_size, 0);
hbVPArray jpeg_buf;
jpeg_buf.phyAddr = jpeg_mem.phyAddr;
jpeg_buf.virAddr = jpeg_mem.virAddr;
jpeg_buf.memSize = jpeg_mem.memSize;
jpeg_buf.size = jpeg_mem.memSize;
jpeg_buf.capacity = jpeg_mem.memSize;

// init image_buf
hbVPImage image_buf{0};

// create decoding context and the JPU internally alloc the output buffer
hbVPJPEGContext dec_context{nullptr};
hbVPCreateJPEGDecContext(&dec_context, outbuf_count, image_format);

// init task handle and schedule param
hbUCPTaskHandle_t task_handle{nullptr};
hbUCPSchedParam sched_param;
HB_UCP_INITIALIZE_SCHED_PARAM(&sched_param);
sched_param.backend = HB_UCP_JPU_CORE_0;
sched_param.priority = 0;

// create decoding task
hbVPJPEGDecode(&task_handle, &jpeg_buf, dec_context);

// submit decoding task
hbUCPSubmitTask(task_handle, &sched_param);

// wait for decoding task done
hbUCPWaitTaskDone(task_handle, 0);

// get decoded buffer from JPU
hbVPGetJPEGDecOutputBuffer(&image_buf, task_handle);

// process image data

// release task handle and dequeue output buffer from JPU
// since the dequeue output buffer process relies on the context,
// you must call the hbUCPReleaseTask interface before calling the hbVPReleaseJPEGEncContext interface
hbUCPReleaseTask(task_handle);

// release encoding context and the JPU internally release the output buffer
hbVPReleaseJPEGDecContext(dec_context);

// release memory
hbUCPFree(&jpeg_mem);