CN103618904B - Motion estimation method and device based on pixels - Google Patents
Motion estimation method and device based on pixels Download PDFInfo
- Publication number
- CN103618904B CN103618904B CN201310586718.9A CN201310586718A CN103618904B CN 103618904 B CN103618904 B CN 103618904B CN 201310586718 A CN201310586718 A CN 201310586718A CN 103618904 B CN103618904 B CN 103618904B
- Authority
- CN
- China
- Prior art keywords
- cost map
- cost
- pixel
- candidate
- subunit
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Landscapes
- Image Analysis (AREA)
Abstract
本发明实施例公开了基于像素的运动估计方法及装置。所述方法包括：生成输入图像相对于参考图像在每个候选矢量下的初始代价图；根据所述初始代价图生成候选代价图；根据所述候选代价图为所述输入图像的像素选取代价最小的候选矢量作为该像素的运动矢量。所述装置包括：生成单元，候选单元，选择单元。采用本发明实施例公开的方法及装置，进行基于像素的运动估计，获取基于像素的运动矢量，不需要进行密集运算，处理效率相对较高。
The embodiment of the invention discloses a pixel-based motion estimation method and device. The method includes: generating an initial cost map of an input image relative to a reference image under each candidate vector; generating a candidate cost map according to the initial cost map; selecting a pixel with the smallest cost for the input image according to the candidate cost map The candidate vector of is used as the motion vector of the pixel. The device includes: a generation unit, a candidate unit, and a selection unit. By adopting the method and device disclosed in the embodiment of the present invention, pixel-based motion estimation is performed, and pixel-based motion vectors are obtained without intensive calculation, and the processing efficiency is relatively high.
Description
Technical Field
The present invention relates to the field of video processing, and in particular, to a motion estimation method and apparatus based on pixels.
Background
In the frame sequential images, there is usually a certain correlation between the scenes in adjacent image frames, and different image frames may contain the same scene. However, the spatial position of the same scene may differ in different image frames. The spatial position of a scene in an input image will have a relative offset, commonly referred to as a motion vector, relative to the spatial position of the scene in a reference image. The process of acquiring the Motion vector of the object is called Motion Estimation (Motion Estimation).
Motion estimation is a widely used technique in video coding and video processing. In video coding, motion vectors can be used to find image macroblocks with the highest correlation in a reference image frame, thereby reducing redundancy of video content. The most common motion estimation method in video coding is based on motion estimation of regular macroblocks. Based on motion estimation of regular macroblocks, an input image is usually first divided into several macroblocks of equal size; then calculating the Sum of Absolute differences (SAD, Sum of Absolute Difference) of each macro block under different candidate vectors; and finally, selecting the candidate vector corresponding to the minimum SAD value as the motion vector of the macro block. In video processing, many implementations of the technology also require the use of motion vectors. For example, video de-interlacing, temporal super-resolution, temporal noise reduction, and video frame interpolation all require motion vectors. Unlike motion vectors used in video coding, video processing is typically performed in pixel-based units, and thus pixel-based motion vectors are required for video processing. The pixel-based motion vector needs to be acquired using a pixel-based motion estimation method.
There are a variety of pixel-based motion estimation methods. The more common methods include a motion estimation method based on feature tracking, a global optical flow method, and the like. Firstly, generating a sparse optical flow; then, interpolating the sparse optical flow to obtain a dense optical flow at a pixel level; a motion vector for each pixel is determined from the dense optical flow. The process of interpolating the sparse optical flow requires intensive operations. The global optical flow method needs to uniformly model the cost of the global optical flow according to a certain rule; performing iterative operation and optimizing the optical flow graph, and outputting the optical flow graph corresponding to the minimum cost when the minimum cost is reached; and then determining a motion vector corresponding to each pixel according to the light flow diagram. The process of iterative operation and optimization of the optical flow graph also requires intensive operation.
As can be seen from the above, the motion estimation by the existing method to obtain the motion vector based on the pixel requires intensive operations such as interpolation or iteration, and the processing efficiency is low.
Disclosure of Invention
The embodiment of the invention provides a motion estimation method and device based on pixels, and aims to solve the problems that the existing method needs intensive operation and is low in processing efficiency.
In a first aspect, an embodiment of the present invention provides a pixel-based motion estimation method, including: generating an initial cost map of the input image relative to the reference image under each candidate vector; generating a candidate cost graph according to the initial cost graph; and selecting a candidate vector with the minimum cost for the pixel of the input image according to the candidate cost map as the motion vector of the pixel.
With reference to the first aspect, in a first possible implementation manner of the first aspect, the generating an initial cost map of the input image relative to the reference image under each candidate vector includes: and generating an initial cost map of the input image relative to the reference image under each candidate vector by adopting a mode of calculating the sum of the absolute pixel difference and the absolute gradient difference of the corresponding pixel.
With reference to the first aspect or the first possible implementation manner of the first aspect, in a second possible implementation manner of the first aspect, the generating a candidate cost map according to the initial cost map includes: smoothing each initial cost map to obtain a smooth cost map; and determining a candidate cost map according to the smooth cost map.
With reference to the second possible implementation manner of the first aspect, in a third possible implementation manner of the first aspect, the smoothing of each initial cost map to obtain a smoothed cost map includes: calculating a linear relation between each initial cost map and the input image; and smoothing each initial cost graph according to the linear relation to obtain a smooth cost graph.
With reference to the third possible implementation manner of the first aspect, in a fourth possible implementation manner of the first aspect, before the calculating a linear relationship between each of the initial cost maps and the input image, the method further includes: downsampling each initial cost map to generate a sampling cost map; down-sampling the input image to generate a sampled image; the calculating a linear relationship between each of the initial cost maps and the input image includes: calculating the linear relationship between each of the sampled cost maps and the sampled image.
With reference to the fourth possible implementation manner of the first aspect, in a fifth possible implementation manner of the first aspect, the obtaining a linear relationship between each of the sampling cost maps and the sampling image includes calculating a linear relationship between each of the sampling cost maps and the sampling image by using a linear regression method.
With reference to the fourth possible implementation manner of the first aspect or the fifth possible implementation manner of the first aspect, in a sixth possible implementation manner of the first aspect, the downsampling each initial cost map to generate a sampling cost map further includes: performing mean filtering on each initial cost graph; the downsampling each initial cost map to generate a sampling cost map comprises: and downsampling each initial cost map subjected to mean filtering to generate a sampling cost map.
With reference to the fourth possible implementation manner of the first aspect, the fifth possible implementation manner of the first aspect, or the sixth possible implementation manner of the first aspect, in a seventh possible implementation manner of the first aspect, before the downsampling the input image to generate a sampled image, the method further includes: performing mean filtering on the input image; the downsampling the input image to generate a sampled image comprises: and downsampling the input image after the average filtering to generate a sampling image.
With reference to the fourth possible implementation manner of the first aspect, the fifth possible implementation manner of the first aspect, the sixth possible implementation manner of the first aspect, or the seventh possible implementation manner of the first aspect, in an eighth possible implementation manner of the first aspect, the determining a candidate cost map according to the smooth cost map includes: taking the smoothed cost map as the candidate cost map; or selecting a preselected vector with the minimum cost according to the sampling cost map; and modifying each smooth cost map by using the preselected vector to obtain the candidate cost map.
On the other hand, an embodiment of the present invention further provides a pixel-based motion estimation apparatus, including: a generating unit, configured to generate an initial cost map of the input image relative to the reference image under each candidate vector; a candidate unit, configured to generate a candidate cost map according to the initial cost map generated by the generating unit; and the selecting unit is used for selecting a candidate vector with the minimum cost for the pixel of the input image according to the candidate cost map generated by the candidate unit as the motion vector of the pixel.
With reference to the second aspect, in a first possible implementation manner of the second aspect, the generating unit is configured to generate an initial cost map of the input image relative to the reference image at each candidate vector by calculating a sum of a pixel absolute difference and a gradient absolute difference of a corresponding pixel.
With reference to the second aspect or the first possible implementation manner of the second aspect, in a second possible implementation manner of the second aspect, the candidate unit includes: a smoothing subunit, configured to smooth each initial cost map to obtain a smoothed cost map; and the determining subunit is used for determining the candidate cost map according to the smooth cost map.
With reference to the second possible implementation manner of the second aspect, in a third possible implementation manner of the second aspect, the smoothing subunit includes: a relation calculating subunit, configured to calculate a linear relation between each of the initial cost maps and the input image; and the linear smoothing subunit is used for smoothing each initial cost map according to the linear relationship calculated by the relationship calculation subunit to obtain a smooth cost map.
With reference to the third possible implementation manner of the second aspect, in a fourth possible implementation manner of the second aspect, the smoothing subunit further includes: the first downsampling subunit is used for downsampling each initial cost map to generate a sampling cost map; a second downsampling subunit, configured to downsample the input image to generate a sampled image; the relation calculating subunit is configured to calculate the linear relation between each of the sampling cost maps generated by the first downsampling subunit and the sampling image generated by the second downsampling subunit.
With reference to the fourth possible implementation manner of the second aspect, in a fifth possible implementation manner of the second aspect, the relationship calculating subunit is configured to calculate, by using a linear regression method, a linear relationship between each of the sampling cost maps generated by the first downsampling subunit and the sampling image generated by the second downsampling subunit.
With reference to the fourth possible implementation manner of the second aspect or the fifth possible implementation manner of the second aspect, in a sixth possible implementation manner of the second aspect, the smoothing subunit further includes: the first filtering subunit is configured to perform mean filtering on each initial cost map; the first downsampling subunit is configured to downsample each initial cost map subjected to the average filtering by the first filtering subunit to generate a sampling cost map.
With reference to the fourth possible implementation manner of the second aspect, the fifth possible implementation manner of the second aspect, or the sixth possible implementation manner of the second aspect, in a seventh possible implementation manner of the second aspect, the smoothing subunit further includes: the second filtering subunit is used for performing mean value filtering on the input image; and the second downsampling subunit is configured to downsample the input image subjected to the average filtering by the second filtering subunit to generate a sampled image.
With reference to the fourth possible implementation manner of the second aspect, the fifth possible implementation manner of the second aspect, the sixth possible implementation manner of the second aspect, or the seventh possible implementation manner of the second aspect, in an eighth possible implementation manner of the second aspect, the selecting unit is configured to use the smooth cost map as the candidate cost map; or selecting a preselected vector with the minimum cost according to the sampling cost map; and modifying each smooth cost map by using the preselected vector to obtain the candidate cost map.
In the embodiment of the invention, an initial cost map of an input image relative to a reference image under each candidate vector is generated; generating a candidate cost graph according to the initial cost graph; and selecting a candidate vector with the minimum cost for the pixel of the input image according to the candidate cost map as the motion vector of the pixel. By applying the embodiment of the invention, the motion estimation based on the pixel is carried out, the motion vector based on the pixel is obtained, the dense operation such as interpolation or iteration is not needed, and the processing efficiency is relatively high.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings needed in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings without creative efforts. The above and other objects, features and advantages of the present invention will become more apparent from the accompanying drawings. Like reference numerals refer to like parts throughout the drawings. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the invention.
FIG. 1 is a flow chart of one embodiment of a pixel-based motion estimation method of the present invention;
FIG. 2 is a flow chart of another embodiment of a pixel-based motion estimation method of the present invention;
FIG. 3 is a block diagram of an embodiment of a pixel-based motion estimation apparatus of the present invention;
FIG. 4 is a block diagram of one embodiment of a candidate unit for a pixel-based motion estimation device according to the present invention;
FIG. 5A is a block diagram of one embodiment of a smoothing subunit of the pixel-based motion estimation apparatus of the present invention;
FIG. 5B is a block diagram of one embodiment of a smoothing subunit of the pixel-based motion estimation apparatus of the present invention;
fig. 5C is a block diagram of one embodiment of the smoothing subunit of the pixel-based motion estimation apparatus of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be described clearly and completely with reference to the accompanying drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1, a flow chart of an embodiment of the pixel-based motion estimation method of the present invention is shown. This embodiment comprises the steps of:
step 101, generating an initial cost map of the input image relative to the reference image under each candidate vector.
In order to accelerate the processing speed and keep the consistency of the final result, N vectors can be selected in advance as candidate vectors when the motion estimation based on the pixels is carried out, wherein N is more than or equal to 2; then, one candidate vector with the minimum cost is selected from the N candidate vectors for each pixel of the input image to serve as the motion vector of the pixel. In this case, the motion estimation process is a process of selecting a motion vector for each pixel from the candidate vectors. The candidate vectors may be selected in a variety of ways, for example, motion vectors of pixels of a processed image may be used as the candidate vectors, wherein the processed image may be an image in the same image sequence as the input image.
When selecting a motion vector, an initial cost map of the input image relative to the reference image under each candidate vector needs to be calculated first. The initial cost map consists of the cost of each pixel in the input image relative to the corresponding pixel in the reference image under the candidate vector. If N candidate vectors exist, N initial cost graphs can be obtained, and the N candidate vectors correspond to the N initial cost graphs one by one.
And 102, generating a candidate cost map according to the initial cost map.
After the initial cost maps are obtained, a smooth cost map corresponding to each initial cost map can be generated firstly; and then determining a candidate cost map corresponding to each initial cost map according to the smooth cost map. If N initial cost graphs exist, N smooth cost graphs can be correspondingly generated, and the N candidate cost graphs correspond to the N initial cost graphs one by one.
A direct smoothing mode can be adopted when generating the smooth cost graph. Specifically, the initial cost map may be smoothed according to a linear relationship between the initial cost map and the input image, so as to obtain a smoothed cost map. The initial cost graph is smoothed, so that the selection of the motion vector can be more accurate, and better consistency is achieved. The linear relation between the initial cost map and the input image can be calculated by adopting a linear regression method.
A first sampling and then smoothing mode can be adopted when generating the smooth cost map. Specifically, the initial cost map may be downsampled to obtain a sampling cost map, and the input image may be downsampled to obtain a sampling image; and then smoothing the sampling cost graph according to the linear relation between the sampling cost graph and the sampling image to generate a smooth cost graph. And the initial cost graph and the input image are downsampled, so that the calculation complexity can be reduced on the premise of ensuring the accuracy of motion vector selection. The linear relation between the sampling cost graph and the sampling image can also be calculated by adopting a linear regression method.
Before sampling the initial cost graph, mean value filtering can be performed on the initial cost graph; similarly, before sampling the input image, the input image may be subjected to mean filtering. The initial cost graph and the input image are subjected to mean filtering, and then sampling is performed, so that the robustness of the processing process can be increased, and the influence of image noise is reduced.
There are also various ways to determine the candidate cost map from the smoothed cost map. For example, after the smooth cost map is obtained, the smooth cost map may be directly used as a candidate cost map; or, a preselected vector may be selected for each pixel according to the initial cost map, then the smooth cost map is modified by using the preselected vector, and the modified smooth cost map is used as a candidate cost map, so as to improve the probability of selecting the preselected vector.
When the initial cost maps are N, each initial cost map can be smoothed by the method described above.
And 103, selecting a candidate vector with the minimum cost for the pixel of the input image according to the candidate cost map as the motion vector of the pixel.
From the foregoing process of obtaining the candidate cost map, according to N candidate cost maps corresponding to N candidate vectors, the cost of the pixel j under each candidate vector can be known relative to the pixel mi + j, where the pixel j is a pixel in the input image, and the pixel mi + j is a pixel in the reference image corresponding to the pixel j; so that for each pixel in the input image I a least costly candidate vector can be selected as the motion vector. When selecting a motion vector for each pixel, the selection process may be based on the Winner Take All (WTA) rule, and the detailed selection process is not described herein again.
In the embodiment of the invention, an initial cost map of an input image relative to a reference image under each candidate vector is generated; generating a candidate cost graph according to the initial cost graph; and selecting a candidate vector with the minimum cost for the pixel of the input image according to the candidate cost map as the motion vector of the pixel. By applying the embodiment, the motion estimation based on the pixel is carried out, the motion vector based on the pixel is obtained, the intensive operation is not needed, and the processing efficiency is relatively high.
Referring to fig. 2, a flow chart of another embodiment of the pixel-based motion estimation method of the present invention is shown, which illustrates the pixel-based motion estimation method of the present invention in detail.
Step 201, an initial cost map of the input image relative to the reference image under each candidate vector is generated by adopting a mode of calculating the sum of the absolute difference of the pixel of the corresponding pixel and the absolute difference of the gradient.
For convenience of description, the input image may be denoted as I, the reference image may be denoted as R, the candidate vector may be denoted as I, and the initial cost map may be denoted as C. The pixel of the input image I may be denoted as j, the pixel corresponding to the pixel j in the reference image R may be denoted as mi + j, and the initial cost map of the input image I relative to the reference image R under the candidate vector I may be denoted as Ci. The initial cost for a pixel j in the input image I under the candidate vector I may be referred to as the initial cost for the pixel j for short, relative to the pixel mi + j in the reference image R corresponding to the pixel j.
Initial cost graph CiMay be constituted by the cost of each pixel in the input image I. The initial cost for pixel j can be calculated as the pixel absolute difference of pixel j and pixel mi + j and the gradient absolute difference of pixel j and pixel mi + j. Initial cost C of pixel j under candidate vector ii,j＝|Ij-Rmi+j|+|Dj-Dmi+jI, wherein Ij-Rmi+j| is the absolute difference between pixel j and pixel mi + j, | Dj-Dmi+jAnd | is the absolute difference of the gradient of the pixel j and the pixel mi + j. Since each pixel can be represented by the coordinate value of the pixel in the image, the pixel absolute difference between the pixel j and the pixel mi + j can be calculated according to the coordinate value of the pixel j and the coordinate value of the pixel mi + j, and the gradient absolute difference between the pixel j and the pixel mi + j can be calculated by using the gradient value of the pixel j and the gradient value of the pixel mi + j, and the specific calculation process is not repeated here.
When there are multiple candidate vectors, the above method may be adopted to obtain the initial cost map corresponding to each candidate vector. The sum of the absolute difference of the pixels and the absolute difference of the gradient is adopted to calculate the cost of each pixel, the calculation process is simple, and the edge characteristics of the input image can be kept in the cost map.
And 202, performing mean filtering on each initial cost map.
The mean filtering is a typical linear filtering algorithm, and refers to setting a filtering template for a target pixel on an image, where the filtering template includes neighboring pixels around the target pixel, for example, 8 surrounding pixels centered on the target pixel, to form a filtering template; the average value of all pixels in the template is used to replace the original pixel value, and the average value and the pixel value can be color value, gray value, etc. There are many specific implementation methods for the mean filtering, and the detailed description thereof is omitted here. And performing mean filtering on the initial cost graph to reduce the influence of image noise on the motion vector selection result.
And 203, down-sampling each initial cost map after mean filtering to obtain a sampling cost map.
To reduce the complexity of the calculation process, the mean filtered initial cost map may be downsampled. The down-sampling ratio can be set as desired. For example, the initial cost map may be sampled four times horizontally and three times vertically. And if N initial cost graphs exist, respectively downsampling each initial cost graph to generate a sampling cost graph corresponding to each initial cost graph.
And 204, performing mean filtering on the input image.
And the mean value filtering is carried out on the input image, so that the influence of noise on the motion vector selection result can be reduced.
And step 205, down-sampling the input image after the average filtering to obtain a sampling image.
In order to reduce the complexity of the calculation process, the average-filtered input image can be downsampled. The down-sampling may be in the same scale as the down-sampling of the initial cost map.
It should be noted that, step 202 to step 203 may be executed first, and then step 204 to step 205 may be executed; step 204 to step 205 may be executed first, and then step 202 to step 203 may be executed, which is not limited to the present invention.
Step 206, calculating the linear relation between each sampling cost map and the sampling image;
and the linear relation between the sampling cost graph and the sampling image is formed by the linear relation between each pixel in the sampling image and the cost corresponding to the pixel in the sampling cost graph. The linear relationship between the pixel and the cost corresponding to the pixel can be calculated according to the pixel brightness and the cost corresponding to the pixel. When there are N sampling cost maps, a linear regression method can be adopted to obtain the linear relationship between each sampling cost map and the sampling image one by one.
The linear relationship between the pixel in the sampling image and the cost corresponding to the pixel in the sampling cost image is determined by the linear relationship coefficient between the pixel in the sampling image and the cost corresponding to the pixel in the sampling cost image, and the linear relationship between the pixel in the sampling image and the cost can be determined as long as the linear relationship coefficient between the pixel in the sampling image and the cost is obtained. When calculating the linear relation coefficient of a certain pixel, firstly, a local image window is set on the sampling image for the pixel, and the local image window comprises the pixel. The size of the partial image can be set according to requirements. For example, 13 × 13 image blocks may be used as local image windows. The calculation process may include the steps of: calculating the average brightness of each pixel in the local image windowAnd the standard deviation of the luminance var (I 'I'), whereinCalculating the average cost value of the pixels in the local image window under the candidate vector iAnd covariance cov of pixel cost values in local image windows (C)i'I'), whereinAccording to the average brightnessStandard deviation of luminance var (I 'I'), average cost valueAnd covariance of cost cov (C)i'I'), the pixel can be calculated to calculate the linear relation coefficient. The coefficient of the linear relationship includes a slope aiAnd intercept biWhereinas shown in the foregoing formula, the standard deviation and covariance can be calculated by using the mean of the local image window. The computational complexity can be made independent of the window size using techniques such as integral mapping. There are various methods for calculating the linear relation coefficient, and they will not be described in detail here.
If there are N sampling cost maps, the linear relationship between each sampling cost map and the input image can be obtained by the method, so as to obtain N linear relationships corresponding to the N sampling cost maps one to one. The linear relation between the brightness of the down-sampling image and the sampling cost graph is calculated by adopting a linear regression method, and only one-time scanning can be performed, so that the operation complexity is reduced.
And step 207, smoothing each initial cost map according to the linear relation to obtain a smooth cost map.
After the linear relationship between the sampling cost map and the sampling image is obtained, the sampling cost map can be smoothed according to the linear relationship. And a sampling cost map Ci' the corresponding smoothed cost map can be denoted as Ci". For sampling cost graph Ci' smoothing may be performed by applying a sample cost map CiThe cost of each pixel included in' is smoothly achieved. For example, under the candidate vector I, a pixel j in the input image I corresponds to a pixel j in the sample image corresponding to the input imageThe cost of the pixel j after smoothing can be recorded as C "ikKnown as C "ik＝aik*Ij+bikWherein a isikIs the slope corresponding to pixel j, bikIs the intercept corresponding to pixel j, IjIs the luminance of pixel j.
And 208, selecting a preselected vector with the minimum cost according to the sampling cost map.
After each initial cost map is downsampled to obtain a sampling cost map, a motion vector with the minimum cost can be preselected for each pixel according to the sampling cost map to serve as a preselected vector. When selecting the preselected vector for each pixel, the selection process may be based on the Winner Take All (WTA) rule, and the detailed selection process is not described herein.
And 209, modifying each smooth cost map by using the preselected vector to generate a candidate cost map.
After the preselected vector is obtained, the smooth cost map can be corrected according to the preselected vector to generate a candidate cost map, and the probability of selecting the preselected vector is improved. For example, if the pre-selected vector for pixel k in the sample image corresponding to pixel j in the input image I is t. Smooth cost graph C corresponding to preselected vector tt"where the cost corresponding to pixel k can be represented as Ctk", then C may be substitutedtk"is multiplied by a coefficient less than 1, thereby increasing the probability that the motion vector t becomes the least costly candidate vector for pixel j in the input image I. Preselecting the vector as a reference for vector selection improves the consistency of motion vectors in input images with sparse texture or high noise.
And step 210, selecting a candidate vector with the minimum cost for the pixel of the input image according to the candidate cost map as the motion vector of the pixel.
After the candidate cost map is acquired, a motion vector with the minimum cost is selected for each pixel according to the WTA rule.
It should be noted that, after step 207 is executed, step 210 may be directly executed, that is, after the smooth cost map is obtained, the smooth cost map may not be modified, and the smooth cost map is directly used as a candidate cost map; a least costly motion vector is then selected for each pixel. The smooth cost graph is directly used as a candidate cost graph, so that the processing process can be simplified, and the data calculation amount can be reduced.
It can be seen from the above embodiments that, when the embodiments are applied to the pixel-based motion estimation, the calculation is simple, the processing efficiency is relatively high, and the embodiments are suitable for being implemented by using hardware, and the motion vectors can be accurately selected, so that the selected motion vectors have relatively good consistency and relatively good robustness.
Corresponding to the pixel-based motion estimation method, the invention also provides a pixel-based motion estimation device.
Referring to fig. 3, a block diagram of an embodiment of the apparatus for estimating motion based on pixels according to the present invention is shown.
The device includes: a generating unit 301, a candidate unit 302, and a selecting unit 303.
Wherein the generating unit 301 is configured to generate an initial cost map of the input image relative to the reference image under each candidate vector.
The specific way in which the generating unit 301 generates the initial cost map of the input image with respect to the reference image under each candidate vector may be various. For example, the generating unit 301 may be configured to generate an initial cost map of the input image relative to the reference image at each candidate vector in a manner of calculating a sum of a pixel absolute difference and a gradient absolute difference of a corresponding pixel. The specific generation process is not described herein.
The candidate unit 302 is configured to generate a candidate cost map according to the initial cost map generated by the generating unit 301.
After the generating unit 301 generates the initial cost maps, the candidate unit 302 may first generate a smooth cost map corresponding to each initial cost map; and then determining a candidate cost map corresponding to each initial cost map according to the smooth cost map.
As shown in fig. 4, the candidate unit 302 may include: a smoothing subunit 401 and a determination subunit 402. The smoothing subunit 401 is configured to smooth each of the initial cost maps to obtain a smoothed cost map; the determining subunit 402 is configured to determine a candidate cost map from the smoothed cost map.
As shown in fig. 5A, the smoothing subunit 401 may include: a relation calculation subunit 501 and a linear smoothing subunit 502. The relationship calculation subunit 501 is configured to calculate a linear relationship between each initial cost map and the input image; the linear smoothing subunit 502 is configured to smooth each initial cost map according to the linear relationship calculated by the calculation subunit 501 to obtain a smooth cost map.
To reduce the amount of data calculation and increase the processing speed, as shown in fig. 5B, the smoothing subunit 401 may further include: a first downsampling subunit 503 and a second downsampling subunit 504. The first down-sampling subunit 503 is configured to down-sample each initial cost map to generate a sampling cost map; a second downsampling subunit 504, configured to downsample the input image to generate a sampled image. When the smoothing sub-unit 401 includes the first downsampling sub-unit 503 and the second downsampling sub-unit 504, the relation calculating sub-unit 501 may be configured to calculate the linear relation between each of the sampling cost maps generated by the first downsampling sub-unit 503 and the sampling image generated by the second downsampling sub-unit 504.
To reduce the influence of noise points in the image, as shown in fig. 5C, the smoothing subunit 401 may further include: a first filtering subunit 505 and a second filtering subunit 506. The first filtering subunit 505 is configured to perform mean filtering on each initial cost map; a second filtering subunit 506, configured to perform mean filtering on the input image. When the smoothing sub-unit includes the first filtering sub-unit 505, the first downsampling sub-unit 503 may be configured to downsample each of the initial cost maps after mean filtering by the first filtering sub-unit 505 to generate a sampling cost map. When the smoothing sub-unit comprises a second filtering sub-unit 506, the second downsampling sub-unit 504 may be configured to downsample the input image mean-filtered by the second filtering sub-unit 506 to generate a sampled image.
If the smoothing subunit 401 includes the first lower sampling subunit 503, the determining subunit 402 may be configured to select a preselected vector with a minimum cost according to the sampling cost map generated by the first lower sampling subunit 503; and then modifying each smooth cost map by using the preselected vector to obtain the candidate cost map.
The selecting unit 303 is configured to select, as the motion vector of the pixel, a candidate vector with the smallest cost for the pixel of the input image according to the candidate cost map generated by the candidate unit 302.
As can be seen from the foregoing embodiments, the pixel-based motion estimation apparatus provided in this embodiment is used to perform pixel-based motion estimation and obtain a pixel-based motion vector, and therefore, intensive operations are not required, and the processing efficiency is relatively high. Because the calculation process is simple, interpolation or repeated scanning is not needed, and the method is suitable for hardware implementation.
Those skilled in the art will readily appreciate that the techniques of the embodiments of the present invention may be implemented as software plus a required general purpose hardware platform. Based on such understanding, the technical solutions in the embodiments of the present invention may be essentially or partially implemented in the form of a software product, which may be stored in a storage medium, such as ROM/RAM, magnetic disk, optical disk, etc., and includes several instructions for enabling a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method according to the embodiments or some parts of the embodiments.
The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the system embodiment, since it is substantially similar to the method embodiment, the description is simple, and for the relevant points, reference may be made to the partial description of the method embodiment.
The above-described embodiments of the present invention do not limit the scope of the present invention. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (17)
1. A method for pixel-based motion estimation, comprising:
generating an initial cost map of the input image relative to the reference image under each candidate vector, wherein the initial cost map is formed by the cost of each pixel in the input image relative to the corresponding pixel in the reference image under the candidate vector;
generating a candidate cost graph according to the initial cost graph;
and selecting a candidate vector with the minimum cost for the pixel of the input image according to the candidate cost map as the motion vector of the pixel.
2. The method of claim 1, wherein generating an initial cost map for the input image relative to the reference image at each candidate vector comprises:
and generating an initial cost map of the input image relative to the reference image under each candidate vector by adopting a mode of calculating the sum of the absolute pixel difference and the absolute gradient difference of the corresponding pixel.
3. The method according to claim 1 or 2, wherein the generating a candidate cost map from the initial cost map comprises:
smoothing each initial cost map to obtain a smooth cost map;
and determining a candidate cost map according to the smooth cost map.
4. The method of claim 3, wherein said smoothing each of said initial cost maps results in a smoothed cost map, comprising:
calculating a linear relation between each initial cost map and the input image;
and smoothing each initial cost graph according to the linear relation to obtain a smooth cost graph.
5. The method of claim 4, further comprising, prior to said computing a linear relationship between each of said initial cost maps and said input image:
downsampling each initial cost map to generate a sampling cost map;
down-sampling the input image to generate a sampled image;
the calculating a linear relationship between each of the initial cost maps and the input image includes:
calculating the linear relationship between each of the sampled cost maps and the sampled image.
6. The method of claim 5, wherein said computing said linear relationship between each of said sampled cost maps and said sampled image comprises:
and calculating the linear relation between each sampling cost graph and the sampling image by adopting a linear regression method.
7. The method of claim 5 or 6, wherein said down-sampling each of said initial cost maps generates a sampled cost map, further comprising:
performing mean filtering on each initial cost graph;
the downsampling each initial cost map to generate a sampling cost map comprises:
and downsampling each initial cost map subjected to mean filtering to generate a sampling cost map.
8. The method of claim 5, wherein prior to downsampling the input image to generate a sampled image, further comprising:
performing mean filtering on the input image;
the downsampling the input image to generate a sampled image comprises:
and downsampling the input image after the average filtering to generate a sampling image.
9. The method of claim 5, wherein determining a candidate cost map from the smoothed cost map comprises:
taking the smoothed cost map as the candidate cost map; or,
selecting a preselected vector with the minimum cost according to the sampling cost graph; and modifying each smooth cost map by using the preselected vector to obtain the candidate cost map.
10. An apparatus for pixel-based motion estimation, comprising:
a generating unit, configured to generate an initial cost map under each candidate vector of the input image relative to the reference image, where the initial cost map is composed of costs under the candidate vectors of each pixel in the input image relative to a corresponding pixel in the reference image;
a candidate unit, configured to generate a candidate cost map according to the initial cost map generated by the generating unit;
and the selecting unit is used for selecting a candidate vector with the minimum cost for the pixel of the input image according to the candidate cost map generated by the candidate unit as the motion vector of the pixel.
11. The apparatus of claim 10,
and the generating unit is used for generating an initial cost map of the input image relative to the reference image under each candidate vector by adopting a mode of calculating the sum of the absolute difference of the pixel of the corresponding pixel and the absolute difference of the gradient.
12. The apparatus of claim 10 or 11, wherein the candidate unit comprises:
a smoothing subunit, configured to smooth each initial cost map to obtain a smoothed cost map;
and the determining subunit is used for determining the candidate cost map according to the smooth cost map.
13. The apparatus of claim 12, wherein the smoothing subunit comprises:
a relation calculating subunit, configured to calculate a linear relation between each of the initial cost maps and the input image;
and the linear smoothing subunit is used for smoothing each initial cost map according to the linear relationship calculated by the relationship calculation subunit to obtain a smooth cost map.
14. The apparatus of claim 13, wherein the smoothing subunit further comprises:
the first downsampling subunit is used for downsampling each initial cost map to generate a sampling cost map;
a second downsampling subunit, configured to downsample the input image to generate a sampled image;
the relation calculating subunit is configured to calculate the linear relation between each of the sampling cost maps generated by the first downsampling subunit and the sampling image generated by the second downsampling subunit.
15. The apparatus of claim 14,
and the relation calculating subunit is configured to calculate a linear relation between each of the sampling cost maps generated by the first down-sampling subunit and the sampling image generated by the second down-sampling subunit by using a linear regression method.
16. The apparatus of claim 14 or 15, wherein the smoothing subunit further comprises:
the first filtering subunit is configured to perform mean filtering on each initial cost map;
the first downsampling subunit is configured to downsample each initial cost map subjected to the average filtering by the first filtering subunit to generate a sampling cost map.
17. The apparatus of claim 14, wherein the smoothing subunit further comprises:
the second filtering subunit is used for performing mean value filtering on the input image;
and the second downsampling subunit is configured to downsample the input image subjected to the average filtering by the second filtering subunit to generate a sampled image.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310586718.9A CN103618904B (en) | 2013-11-20 | 2013-11-20 | Motion estimation method and device based on pixels |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310586718.9A CN103618904B (en) | 2013-11-20 | 2013-11-20 | Motion estimation method and device based on pixels |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103618904A CN103618904A (en) | 2014-03-05 |
| CN103618904B true CN103618904B (en) | 2017-02-22 |
Family
ID=50169608
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310586718.9A Active CN103618904B (en) | 2013-11-20 | 2013-11-20 | Motion estimation method and device based on pixels |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103618904B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| GB2536904B (en) | 2015-03-30 | 2017-12-27 | Imagination Tech Ltd | Image filtering based on image gradients |
| WO2017036399A1 (en) * | 2015-09-02 | 2017-03-09 | Mediatek Inc. | Method and apparatus of motion compensation for video coding based on bi prediction optical flow techniques |
| WO2018166357A1 (en) * | 2017-03-16 | 2018-09-20 | Mediatek Inc. | Method and apparatus of motion refinement based on bi-directional optical flow for video coding |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5825909A (en) * | 1996-02-29 | 1998-10-20 | Eastman Kodak Company | Automated method and system for image segmentation in digital radiographic images |
| DE69427801D1 (en) * | 1993-11-03 | 2001-08-30 | Matsushita Electric Industrial Co Ltd | Method and device for determining motion vector fields by suppressing local deviations |
| CN101416515A (en) * | 2006-03-31 | 2009-04-22 | 索尼德国有限责任公司 | Method and apparatus for improving convergence speed of recursive motion estimator |
| CN101629966A (en) * | 2009-08-18 | 2010-01-20 | 清华大学深圳研究生院 | Particle image velocimetry (PIV) processing method |
| CN101826211A (en) * | 2009-03-05 | 2010-09-08 | 索尼株式会社 | Method and system for providing reliable motion vectors |
| CN102014281A (en) * | 2009-09-04 | 2011-04-13 | 夏普株式会社 | Methods and systems for motion estimation with nonlinear motion-field smoothing |
| KR20110120215A (en) * | 2010-04-28 | 2011-11-03 | 소니 주식회사 | An image processing apparatus, an image processing method, an imaging apparatus, and a program |
| CN102819764A (en) * | 2012-07-18 | 2012-12-12 | 郑州金惠计算机系统工程有限公司 | Method for counting pedestrian flow from multiple views under complex scene of traffic junction |
| CN103247058A (en) * | 2013-05-13 | 2013-08-14 | 北京工业大学 | Fast optical flow field calculation method based on error-distributed multilayer grid |
-
2013
- 2013-11-20 CN CN201310586718.9A patent/CN103618904B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| DE69427801D1 (en) * | 1993-11-03 | 2001-08-30 | Matsushita Electric Industrial Co Ltd | Method and device for determining motion vector fields by suppressing local deviations |
| US5825909A (en) * | 1996-02-29 | 1998-10-20 | Eastman Kodak Company | Automated method and system for image segmentation in digital radiographic images |
| CN101416515A (en) * | 2006-03-31 | 2009-04-22 | 索尼德国有限责任公司 | Method and apparatus for improving convergence speed of recursive motion estimator |
| CN101826211A (en) * | 2009-03-05 | 2010-09-08 | 索尼株式会社 | Method and system for providing reliable motion vectors |
| CN101629966A (en) * | 2009-08-18 | 2010-01-20 | 清华大学深圳研究生院 | Particle image velocimetry (PIV) processing method |
| CN102014281A (en) * | 2009-09-04 | 2011-04-13 | 夏普株式会社 | Methods and systems for motion estimation with nonlinear motion-field smoothing |
| KR20110120215A (en) * | 2010-04-28 | 2011-11-03 | 소니 주식회사 | An image processing apparatus, an image processing method, an imaging apparatus, and a program |
| CN102819764A (en) * | 2012-07-18 | 2012-12-12 | 郑州金惠计算机系统工程有限公司 | Method for counting pedestrian flow from multiple views under complex scene of traffic junction |
| CN103247058A (en) * | 2013-05-13 | 2013-08-14 | 北京工业大学 | Fast optical flow field calculation method based on error-distributed multilayer grid |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103618904A (en) | 2014-03-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110268712B (en) | Method and apparatus for processing image property maps | |
| US8768069B2 (en) | Image enhancement apparatus and method | |
| US8306121B2 (en) | Method and apparatus for super-resolution of images | |
| US9615039B2 (en) | Systems and methods for reducing noise in video streams | |
| US8958484B2 (en) | Enhanced image and video super-resolution processing | |
| TWI455588B (en) | Bi-directional, local and global motion estimation based frame rate conversion | |
| US10818018B2 (en) | Image processing apparatus, image processing method, and non-transitory computer-readable storage medium | |
| JP5107409B2 (en) | Motion detection method and filtering method using nonlinear smoothing of motion region | |
| CN106127689B (en) | Image and video super-resolution method and device | |
| Hou et al. | Underwater image dehazing and denoising via curvature variation regularization | |
| JP5890547B2 (en) | Image processing device | |
| CN113221925A (en) | Target detection method and device based on multi-scale image | |
| CN115578255B (en) | A super-resolution reconstruction method based on inter-frame sub-pixel block matching | |
| CN109903315B (en) | Method, apparatus, device and readable storage medium for optical flow prediction | |
| KR20150032822A (en) | Method and apparatus for filtering an image | |
| JP2021184594A (en) | Video frame interpolation device and method | |
| JPWO2016189901A1 (en) | Image processing apparatus, image processing method, program, recording medium recording the same, video photographing apparatus, and video recording / reproducing apparatus | |
| CN113012061A (en) | Noise reduction processing method and device and electronic equipment | |
| CN114640885B (en) | Video frame insertion method, training method, device and electronic equipment | |
| CN103618904B (en) | Motion estimation method and device based on pixels | |
| CN116391203A (en) | Method and image processing device for improving signal-to-noise ratio of image frame sequence | |
| CN103685858A (en) | Real-time video processing method and equipment | |
| CN111784733A (en) | Image processing method, device, terminal and computer readable storage medium | |
| JP2019067169A (en) | Moving vector calculation method, device, program, and moving vector calculation method including noise cancellation processing | |
| CN113438386B (en) | Dynamic and static judgment method and device applied to video processing |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant |