OpticalFlowPyrLK

The OpticalFlowPyrLK operator is used to calculate the changes of specific pixel points in consecutive multi-frame images, which is commonly used in scenarios such as target feature point tracking and improving BOX stability.

Operator Effect

Input continuous frame image	Parameter	Output Image
	pyrLevels = 5 winSize = 7 criteriaEpsilon = 0 maxIteration = 10 minEigThreshold = 1e-4

Principle

The sparse optical flow algorithm, known as the Lucas-Kanade algorithm, is a motion estimation method commonly used in computer vision to estimate the direction and speed of motion of pixels in an image sequence. The Lucas-Kanade algorithm is based on two assumptions:

Brightness constancy assumption: the brightness of pixels of the same object remains constant over a short period of time.

Assuming that the object located at $(x,y)$ pixel position at time $t$ is located at $(x+u,y+v)$ position at time $t+Δt$ , based on the assumption of brightness invariance we have.

$I(x,y,t)=I(x+u,y+v,t+\Delta t)$

A first-order Taylor expansion of the right-hand side of the equation yields:

$I(x+u,y+v,t+\Delta t)=I(x,y,t) + I'_xu + I'_yv+I'_t\Delta t$

It can be obtained from the above two equations:

$I'_xu + I'_yv+I'_t\Delta t=0$

The collation can be expressed as:

$\begin{aligned}\begin{bmatrix} I'_x , I'_y \end{bmatrix}\begin{bmatrix} u \\v \end{bmatrix}=-\Delta I_t\end{aligned}$

Among them, $I'_x$ and $I'_y$ are the partial derivatives of the image luminance in the $x$ direction and the $y$ direction at the $(x,y)$ pixel point, respectively, the gradient of the image in the $x$ and $y$ directions. $ΔIt$ is the $(x,y)$ Brightness difference of the coordinate positions.
Neighborhood optical flow similarity assumption: the direction and size of pixel movement in a small image region is essentially the same.

With the help of this assumption, all pixels in the field of pixel point $(x,y)$ have the following equation.

$\begin{aligned}\begin{bmatrix} I'_{\text{x1}} , I'_{\text{y1}} \\I'_{\text{x2}} , I'_{\text{y2}} \end{bmatrix}\begin{bmatrix} u \\v \end{bmatrix}= \begin{bmatrix} -\Delta I_{\text{t1}} \\-\Delta I_{\text{t2}} \end{bmatrix}\end{aligned}$

The above equation, which is of the form $Ax=b$ , leads to a least squares solution for the optical flow vector:

$x=(A^TA)^{\text{-1}}A^Tb$

One of the requirements is that $A^TA$ is invertible, and in order to fulfill this requirement, the Lucas-Kanade method selects corner points as feature points. In addition to being based on the luminance invariance assumption and the neighborhood optical flow similarity assumption, the Lucas-Kanade algorithm solves for the case of large image offsets with the help of image pyramids, where large offsets become small offsets on high-level low-resolution images, thus solving for the optical flow.

Therefore, the OpticalFlowPyrLK operator requires the pyramid layers of the previous and current frames, as well as the feature points of the previous frame as input, where the feature points are usually chosen to be corner points.

API Interface

int32_t hbVPOpticalFlowPyrLK(hbUCPTaskHandle_t *taskHandle, 
                             hbVPArray *currPoints,
                             hbVPArray *currPointsStatus,
                             hbVPArray *currPointsConf,
                             hbVPArray const *prevPoints,
                             hbVPImage const *currPym,
                             hbVPImage const *prevPym, 
                             hbVPLKOFParam const *lkofParam);

For detailed interface information, please refer to hbVPOpticalFlowPyrLK.

Usage

// Include the header
#include "hobot/hb_ucp.h"
#include "hobot/vp/hb_vp.h"
#include "hobot/vp/hb_vp_opticalflow_pyrlk.h"

// init Image, allocate memory for multilayer gaussian pyramid layer array
int32_t top_k{15};
int32_t_t num_layers{5};
int32_t src_width = 1280;
int32_t src_height = 720;

// Alloc memory used to hold key points, include previous points, current points
hbUCPSysMem arr_mem0;
hbUCPMallocCached(arr_mem0, top_k * sizeof(hbVPKeyPoint), 0);
hbVPArray arr0{arr_mem0.phyAddr,
               arr_mem0.virAddr,
               arr_mem0.memSize,
               top_k};
hbUCPSysMem arr_mem1;  
hbUCPMallocCached(arr_mem1, top_k * sizeof(hbVPKeyPoint), 0);
hbVPArray arr1{arr_mem1.phyAddr,
               arr_mem1.virAddr,
               arr_mem1.memSize,
               top_k};
hbUCPSysMem status_mem;
bUCPMalloc(&status_mem, top_k, 0)
hbVPArray status{status_mem.phyAddr,
                 status_mem.virAddr,
                 status_mem.memSize,
                 top_k};

// Alloc memory used to hold gaussian pyramid images, include previous frame and current frame
std::vector<hbUCPSysMem> prev_frame_mem(num_layers);
std::vector<hbVPImage> prev_frame(num_layers);
std::vector<hbUCPSysMem> cur_frame_mem(num_layers);
std::vector<hbVPImage> cur_frame(num_layers);
for (int32_t i = 0; i < num_layers; i++) {
  if (i != 0) {
    src_width = (src_width + 1) / 2;
    src_height = (src_height + 1) / 2;
  }
  hbUCPMallocCached(&prev_frame_mem[i], src_width * src_height, 0);
  prev_frame[i] = {HB_VP_IMAGE_FORMAT_Y,
                   HB_VP_IMAGE_TYPE_U8C1,
                   src_width,
                   src_height,
                   src_width,
                   prev_frame_mem[i].virAddr,
                   prev_frame_mem[i].phyAddr,
                   nullptr, 
                   0,
                   0};

  hbUCPMallocCached(&cur_frame_mem[i], src_width * src_height, 0);
  cur_frame[i] = {HB_VP_IMAGE_FORMAT_Y,
                  HB_VP_IMAGE_TYPE_U8C1,
                  src_width,
                  src_height,
                  src_width,
                  cur_frame_mem[i].virAddr,
                  cur_frame_mem[i].phyAddr,
                  nullptr, 
                  0,
                  0};
}

// optical flow parameter
hbVPLKOFParam lkof_param;
HB_VP_INITIALIZE_OPTICAL_FLOW_PARAM(&lkof_param);

// init task handle and schedule param
hbUCPTaskHandle_t task_handle{nullptr};
hbUCPSchedParam sched_param;
sched_param.backend = HB_UCP_DSP_CORE_0;
sched_param.priority = 0;

// i frame process
{
  // two blocks of memory read and write alternately
  std::vector<hbVPImage> &prev_frame = i % 2 == 0 ? prev_frame : cur_frame;
  std::vector<hbVPImage> &curr_frame = i % 2 == 0 ? cur_frame : prev_frame;
  std::vector<hbUCPSysMem> &curr_mem = i % 2 == 0 ? cur_frame_mem : prev_frame_mem;

  hbVPArray &prev_points = i % 2 == 0 ? arr0 : arr1;
  hbVPArray &curr_points = i % 2 == 0 ? arr1 : arr0;

    // create task
  hbVPOpticalFlowPyrLK(&task_handle, &curr_points, &status, nullptr, &prev_points, curr_frame.data(), prev_frame.data(), &lkof_param);

  // submit task
  hbUCPSubmitTask(task_handle, &sched_param);

  // wait for task done 
  hbUCPWaitTaskDone(task_handle, 0);

  // release task handle
  hbUCPReleaseTask(task_handle);
}

// release memory
hbUCPFree(&arr_mem0);
hbUCPFree(&arr_mem1);
hbUCPFree(&status_mem);
for (int32_t i = 0; i < num_layers; i++) {
  hbUCPFree(&prev_frame_mem[i]);
  hbUCPFree(&cur_frame_mem[i]);
}

#OpticalFlowPyrLK

#Operator Effect

#Principle

#API Interface

#Usage

OpticalFlowPyrLK

Operator Effect

Principle

API Interface

Usage