De-mosaicking is an image processing technique used to convert Bayer format image data into RGB format image data.
The principle of de-mosaicing is to generate RGB format image data by interpolating the Bayer format image data for subsequent image processing.
In order to better demonstrate how to use DSP to realize the demosaicing function, this chapter starts from the algorithm principle and shows the detailed implementation process of the demosaicing algorithm.
Bayer format image data is a single-channel image, where each pixel contains information of only one color.
Common Bayer formats include RGGB, BGGR, GRBG, and GBRG, among others.
On the other hand, RGB format image data is a three-channel image, where each pixel contains information for red, green, and blue colors.
De-mosaicking algorithms can be classified based on the interpolation method used.
These include nearest-neighbor interpolation, bilinear interpolation, bicubic interpolation, etc.
In this sample, a custom interpolation method based on the principle of constant color difference is used.
For the specific difference principle, please refer to Demosaicing.
Note
The current J6B VDSP operators do not support the FRAME_PADDING_REFLECT_101 edge padding mode. Please review the operator implementation accordingly.
The DSP implementation is mainly optimized around the following points:
By using the tile_manager framework, the data is processed in blocks, and a ping-pong pipeline is constructed for data transfer and computation.
The green_3x3 and green_5x5 interpolation methods are implemented using SIMD (Single Instruction, Multiple Data).
Data classification methods are employed to merge the computation processes for the same data, making full use of the batch processing capabilities of wide-bit registers.
Tile Manager is an operator development framework provided by Cadence. By processing the entire data in blocks, it constructs a ping-pong pipeline for data transfer and computation, thus improving data processing efficiency.
In this framework:
A frame describes the complete input or output data.
A tile describes the partitioning of the frame and serves as the computational processing unit within the framework.
The typical steps for using it in operator development are as follows:
Create an operator parameter structure.
Implement the tile partitioning function to partition the frame data sequentially.
Implement the computation function for each tile to process a single tile.
Use the relevant Tile Manager interfaces to construct the ping-pong pipeline and implement the complete operator functionality.
In the Demosaicing operator, the Frame structure contains one input frame and three output frames. The Tile structure contains one input tile and three output tiles.
This design ensures that in each input tile, the corresponding three RGB tiles can be computed.
The specific parameter structure implementation is as follows:
The next step is to implement the tile partitioning function, which divides the frame data into blocks for sequential processing.
In the Demosaicing operator, the position mapping of each input and output pixel is one-to-one, and the positions do not change.
For example, the G21 point in the Raw image has the same coordinates (2, 1) in each channel of the RGB image.
Therefore, the coordinates and settings of the input tile are consistent with the output.
In the Demosaicing computation, the boundary values of the Raw image require data that extends beyond the original image size, so boundary padding is needed.
In the interpolation algorithm described earlier, the largest range used is 5x5. Therefore, the input boundary size is set to 2, and the output boundary size is set to 0.
The specific parameter structure implementation is as follows:
The third step is to implement the computation function for each tile, which is the core processing logic of the operator, corresponding to the demosaic_kernel function in the source code.
Finally, by calling the relevant framework interfaces, we can achieve the pipeline processing of multiple tiles, corresponding to the demosaic_rggb_comp function in the source code.
#The Implemente of Interpolation Algorithms green_3x3, green_5x5
In the previous analysis of the interpolation algorithms, we can extract the core interpolation logic green_3x3 and green_5x5.
By optimizing with SIMD, the code implementation is as follows:
The input parameters of the function are arrays of different positions needed for interpolation calculation. Each element corresponds to the required value based on the array index.
#Main Calculation Logic Merging and SIMD Optimization
According to the analysis, during the interpolation calculation, the data for odd and even positions can be processed separately.
For example, the elements in even rows are RG, and the R and G values can be computed separately using their parity. Finally, the results can be merged.
In the interpolation calculation, each position's computation involves a maximum of a 5x5 kernel size surrounding it. Therefore, a 5x6 hybrid kernel is defined as follows:
The operator processing follows the maximum processing length of int16 in the DSP, so the first loop iterates over the width with a step size of 1024/16 = 64, and the second loop iterates over the height with a step size of 1.
It can be seen that in the inner processing logic, the first four rows of the hybrid kernel can be reused, which saves redundant computation and reduces processing time.
The process is roughly as follows:
Read the data and split it based on odd and even positions.
Process the split data and apply different demosaicing logic.
Merge the results based on odd and even positions and save them to the output data.
Update the data in the interpolation kernel, making full use of the current computation results for the next round of processing.
Step 1: Read the data and split it based on odd and even positions.
In the DSP implementation, the process of reading the data and splitting it based on odd and even positions is encapsulated in the macro function LOAD_AND_SELECT_ODD_EVEN_WITH_OFFSET.
This function reads up to 64*3 = 192 pieces of data, splits them based on odd and even positions, and then selects the data according to the position of the hybrid kernel.
The selected data is then saved to the corresponding registers.
The definition and usage of this function are as follows:
Step 2: After splitting the data, perform the judgment and execute different demosaicing logic.
During the interpolation process, the computation logic for the same color is consistent. Therefore, the computation logic for the same color can be merged.
In the specific implementation, the odd/even check for rows is determined by the last bit of the sum of the starting y-coordinate and the y-offset, i.e., (y + ys) % 2.
The odd/even check for columns is determined by the last bit of the x-offset, i.e., (x + xs) % 2.
Since the processing step for columns is always even, the x-offset does not affect the odd/even determination.
The specific implementation code is as follows:
uint8_t y_odd_flag = (y ^ ys) & 1; // (y + ys) % 2;xb_vecNx16 dst_vr1, dst_vg1, dst_vb1, dst_vr2, dst_vg2, dst_vb2;if (y_odd_flag) { if (xs & 1) { // x position start from odd position, data is b g b g b g // calc rgb from b green_5x5(dst_vg1, p15, p14, p13, p9, p3, p16, p17, p21, p27); CAL_ROUND_BY_GREEN(dst_vr1, dst_vg1, p8, p10, p20, p22, g33_8, g33_10, p2, p7); dst_vb1 = p15; // calc rgb from g CAL_TB_BY_GREEN(dst_vr2, p16, p10, p22, g33_10, p7); dst_vg2 = p16; CAL_LR_BY_GREEN(dst_vb2, p16, p15, p17, g33_15, g33_17); } else { // x position start from even position, data is g b g b g b // calc rgb from g CAL_TB_BY_GREEN(dst_vr1, p15, p9, p21, g33_9, p5); dst_vg1 = p15; CAL_LR_BY_GREEN(dst_vb1, p15, p14, p16, g33_14, g33_16); // calc rgb from b green_5x5(dst_vg2, p16, p15, p14, p10, p4, p17, p18, p22, p28); CAL_ROUND_BY_GREEN(dst_vr2, dst_vg2, p9, p11, p21, p23, g33_9, g33_11, p5, p12); dst_vb2 = p16; }} else { if (xs & 1) { // x position start from odd position, data is g r g r g r // calc rgb from g CAL_LR_BY_GREEN(dst_vr1, p15, p14, p16, g33_14, g33_16); dst_vg1 = p15; CAL_TB_BY_GREEN(dst_vb1, p15, p9, p21, g33_9, p5); // calc rgb from r dst_vr2 = p16; green_5x5(dst_vg2, p16, p15, p14, p10, p4, p17, p18, p22, p28); CAL_ROUND_BY_GREEN(dst_vb2, dst_vg2, p9, p11, p21, p23, g33_9, g33_11, p5, p12); } else { // x position start from even position, data is r g r g r g // calc rgb from r dst_vr1 = p15; green_5x5(dst_vg1, p15, p14, p13, p9, p3, p16, p17, p21, p27); CAL_ROUND_BY_GREEN(dst_vb1, dst_vg1, p8, p10, p20, p22, g33_8, g33_10, p2, p7); // calc rgb from g CAL_LR_BY_GREEN(dst_vr2, p16, p15, p17, g33_15, g33_17); dst_vg2 = p16; CAL_TB_BY_GREEN(dst_vb2, p16, p10, p22, g33_10, p7); }}
Step 3: Merge the calculation results based on odd and even positions, and save them to the output data. The main implementation method is to use a double-selection operation to combine the data.
It is also important to note that before processing the results, they must be clamped to the valid output range. The specific implementation code is as follows:
Step 4: Update the data in the interpolation kernel and make full use of the current computation results for the next round of processing.
In the inner loop, many parameters can be reused. Since the 5x6 hybrid kernel moves down one row during each loop, only the data in the last row needs to be updated.
The register arrays and the temporary results of green_3x3 can use the old values. The specific implementation code is as follows:
At this point, the entire Demosaicing algorithm implementation process has been completed. This sample is for reference only.
In practical applications, it can be modified and optimized based on this sample, potentially further improving the operator's performance and effectiveness.
