CN102447917A - Three-dimensional image matching method and equipment thereof - Google Patents

Abstract

A stereoscopic image matching method and device thereof are provided, wherein the device includes: a cost calculation unit configured to calculate a matching cost between pixels of a reference image and pixels of a target image, wherein the cost calculation unit calculates the matching costs of the reference image respectively A non-parametric statistical cost function and a color mean absolute difference function between pixels and pixels of the target image, and linearly combining the non-parametric statistical cost function and the color mean absolute difference function to use the combined result as a reference image The matching cost between the pixel of the pixel and the pixel of the target image; the cost aggregation unit is used to gather the matching cost between the pixel of the reference image calculated by the cost calculation unit and the pixel of the target image; and the parallax determination unit is used for The disparity between the pixels of the reference image and the target image is determined based on the result of the aggregation of the matching cost by the cost aggregation unit, so as to realize stereoscopic image matching.

Description

Stereo image matching method and device thereof

technical field

The present invention relates to a method and device for image processing, in particular to a method and device for stereoscopic image matching between a reference image and a target image.

Background technique

Stereo image matching is an important technical topic in the field of computer vision. The disparity map is obtained during the process of implementing stereo image matching between a reference image (for example, left image) and a target image (for example, right image). Visual disparity) refers to the position vector between different projections of the same spatial point in the left image and right image seen by the binocular vision in binocular vision. In practice, since the perception of depth by the human eye depends on the parallax in the horizontal direction, the parallax may only refer to the horizontal component of the position vector.

As mentioned above, parallax is the basis for generating depth perception. Therefore, in the development of 3D image technology, stereoscopic image matching through parallax estimation is the core technology in calculating scene depth, 3D scene reconstruction and 3D display. In existing implementations of stereo image matching, the main focus is on the requirements of matching accuracy and processing efficiency. However, although there are many systems for stereoscopic image matching, it is difficult for these systems to achieve greater improvements in matching accuracy and processing efficiency at the same time. Often, the improvement in one aspect leads to the sacrifice of the other. performance. For example, many stereo image matching schemes with better accuracy have the following things in common: 1) use a large support window for robust cost aggregation operations; 2) use complex global optimizers to perform energy Optimal operation; 3) Mainly rely on the segmentation results of matching images to achieve the improvement of matching accuracy. Although the above features can successfully suppress the fuzzy phenomenon in matching, they all require complex calculation methods as the basis, which makes calculation and processing take a lot of time, making it difficult to apply to situations that require real-time performance. In order to achieve real-time matching performance, some other stereo image matching schemes adopt simplified measures, such as image pyramid, gradual matching, robust measurement and optimizer based on simple dynamic programming, etc., as a result, the matching accuracy in these systems cannot be achieved Not lowered.

Therefore, it is necessary to provide a stereoscopic image matching scheme that can significantly improve both matching accuracy and processing speed, so as to achieve high-precision matching and meet the computing requirements for real-time processing.

Contents of the invention

The purpose of the present invention is to provide a fast stereoscopic image matching method and system capable of achieving high-precision matching. In addition, the performance of stereoscopic image matching can be further improved by performing thinning processing on the generated disparity map.

According to an aspect of the present invention, there is provided a stereoscopic image matching device, the device comprising: a cost calculation unit for calculating matching costs between pixels of a reference image and pixels of a target image, wherein the cost calculation unit calculates the reference The non-parametric statistical cost function and the color mean absolute difference function between the pixels of the image and the pixels of the target image, and the non-parametric statistical cost function and the color mean absolute difference function are linearly combined to use the result of the combination as a matching cost between the pixels of the reference image and the pixels of the target image; a cost aggregation unit for gathering the matching costs between the pixels of the reference image calculated by the cost calculation unit and the pixels of the target image; and a parallax determination unit, It is used to determine the disparity between the pixels of the reference image and the target image based on the result of the aggregation of the matching cost by the cost aggregation unit, so as to realize stereoscopic image matching.

For each pixel in the reference image, the cost aggregation unit can determine the dynamic support region of the pixel, and in each determined dynamic support region, the difference between the pixel of the reference image calculated by the cost calculation unit and the pixel of the target image The matching costs among them are aggregated.

The disparity determination unit may construct the global energy of the matching cost with a depth smoothness constraint, and determine the set of disparity levels that minimize the global energy as the final stereoscopic image matching disparity map.

The cost aggregation unit may include: a pixel dynamic support domain construction unit for constructing a dynamic support domain for each pixel in the reference image; a dynamic support domain cost aggregation unit for constructing each dynamic support domain for which All pixels of are cost aggregated.

The stereo image matching device may further include: a disparity map refinement unit, configured to refine the disparity map output by the disparity determination unit.

The disparity map refinement unit may include at least one of the following items: an occlusion point processing unit, configured to detect an occlusion point in the disparity map, and use images around the occlusion point to estimate the disparity of the occlusion point; a discontinuous boundary correction unit, It is used to detect a discontinuous boundary in which the disparity value jumps in the disparity map, and for pixels on the discontinuous boundary, use pixels on both sides of the discontinuous boundary that are at a certain distance from the pixel to re-determine the discontinuous boundary The disparity of the pixels above; the sub-pixel level difference unit is used to perform sub-pixel level difference operation on the disparity map.

In the linear combination, the ratio between the non-parametric statistical cost function and the color mean absolute difference function may be adaptively set based on each pixel's local texture, local gradient and/or matching confidence information.

According to another aspect of the present invention, a stereoscopic image matching method is provided, the method comprising: calculating the matching cost between the pixels of the reference image and the pixels of the target image, wherein the pixels of the reference image and the pixels of the target image are respectively calculated Between the non-parametric statistical cost function and the color mean absolute difference function, and the non-parametric statistical cost function and the color mean absolute difference function are linearly combined to use the result of the combination as the pixel of the reference image and the pixel of the target image The matching cost between the pixels; the calculated matching costs between the pixels of the reference image and the pixels of the target image are aggregated; and the disparity of the pixels of the reference image to the target image is determined based on the result of the aggregation of the matching costs, to Realize stereo image matching.

The step of aggregating may include: for each pixel in the reference image, determine the dynamic support region of the pixel, and in each determined dynamic support region, the calculated matching between the pixel of the reference image and the pixel of the target image The cost is aggregated.

The step of determining the disparity may include: constructing a global energy of a matching cost with a depth smoothness constraint, and determining a set of disparity levels that minimize the global energy as a final stereoscopic image matching disparity map.

The stereo image matching method may further include: performing thinning processing on the final stereo image matching disparity map.

The step of refining the final stereo image matching disparity map may include at least one of the following steps: detecting an occlusion point in the disparity map, using images around the occlusion point to estimate the disparity of the occlusion point; detecting a disparity in the disparity map The discontinuous boundary where the difference value jumps, for the pixels on the discontinuous boundary, use the pixels at a certain distance from the pixel on both sides of the discontinuous boundary to re-determine the disparity of the pixels on the discontinuous boundary; for the disparity The image performs the difference operation at the sub-pixel level.

According to another aspect of the present invention, there is provided a generating device for 3D content, including: a 3D data input unit, a 3D data analysis unit, and a 3D content generating unit, and the generating device also includes: the stereoscopic image matching according to the present invention equipment.

According to another aspect of the present invention, there is provided a display device for 3D content, including: a 3D data input unit, a 3D data analysis unit, a 3D content generation unit, and a 3D content display unit, and the display device further includes: according to the present invention Invented stereo image matching device.

According to another aspect of the present invention, there is provided a stereoscopic image matching device, which includes: a cost calculation unit for calculating the matching cost between pixels of a reference image and pixels of a target image; a cost aggregation unit for Each pixel in the reference image determines the dynamic support region of the pixel, and in each determined dynamic support region, the matching costs between the pixels of the reference image calculated by the cost calculation unit and the pixels of the target image are gathered; and The disparity determination unit is configured to determine the disparity between the pixels of the reference image and the target image based on the result of the aggregation of the matching cost by the cost aggregation unit, so as to achieve stereoscopic image matching.

According to another aspect of the present invention, there is provided a stereoscopic image matching method, the method comprising: calculating the matching cost between a pixel of a reference image and a pixel of a target image; a support domain, in each determined dynamic support domain, the matching costs between the pixels of the reference image calculated by the cost calculation unit and the pixels of the target image are aggregated; and the reference image is determined based on the result of the aggregation of the matching costs The pixel-to-target image disparity for stereo image matching.

According to the present invention, the matching cost function can be calculated based on specific rules, and the cost aggregation is further carried out for the dynamic support domain of each pixel. On the basis of the cost aggregation, the global depth smoothness can be considered to generate the disparity map of the reference image. Therefore, the matching accuracy and the processing speed are simultaneously improved in the process of realizing stereoscopic image matching. In addition, the performance of image matching can be further improved by performing thinning and correction processing on the generated disparity map.

Description of drawings

The above and/or other objects and advantages of the present invention will become more clear through the following description of the embodiments in conjunction with the accompanying drawings, wherein:

FIG. 1 is a block diagram illustrating a stereoscopic image matching device according to an exemplary embodiment of the present invention;

FIG. 2 is a flowchart illustrating a stereo image matching method according to an exemplary embodiment of the present invention;

3 is a block diagram showing a detailed structure of a cost aggregation unit according to an exemplary embodiment of the present invention;

FIG. 4 shows an example of constructing a pixel dynamic support domain according to an exemplary embodiment of the present invention;

FIG. 5 shows an example of cost aggregation based on dynamic support domains according to an exemplary embodiment of the present invention;

6 is a block diagram illustrating a detailed structure of a disparity map refinement unit according to an exemplary embodiment of the present invention;

Fig. 7 shows an example of occlusion point processing according to an exemplary embodiment of the present invention;

FIG. 8 shows an example of discontinuous boundary correction according to an exemplary embodiment of the present invention; and

FIG. 9 shows the improvement in performance of the stereo image matching system according to the exemplary embodiment of the present invention compared with the prior art.

Detailed ways

Reference will now be made in detail to embodiments of the invention, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to like parts throughout. The embodiments are described below in order to explain the present invention by referring to the figures.

FIG. 1 is a block diagram illustrating a stereoscopic image matching device according to an exemplary embodiment of the present invention. As shown in FIG. 1 , the stereoscopic image matching device according to an exemplary embodiment of the present invention includes: a cost calculation unit 10 configured to calculate a matching cost between pixels of a reference image and pixels of a target image, wherein the cost calculation unit 10 is respectively Calculate the non-parametric statistical cost function and the color mean absolute difference function between the pixels of the reference image and the pixels of the target image, and linearly combine the non-parametric statistical cost function and the color mean absolute difference function to combine the combined The result is used as the matching cost between the pixels of the reference image and the pixels of the target image; the cost aggregation unit 20 is used to gather the matching costs between the pixels of the reference image calculated by the cost calculation unit 10 and the pixels of the target image; and The disparity determination unit 30 is configured to determine the disparity between the pixels of the reference image and the target image based on the result of the aggregation of the matching costs by the cost aggregation unit 20, so as to achieve stereoscopic image matching. As a preferred but non-essential implementation, a disparity map refinement unit 40 may be additionally included for performing refinement processing on the disparity map formed by the disparity output by the disparity determination unit 30, so as to further improve the performance of stereoscopic image matching.

An example of implementing the stereo image matching method according to the present invention by using the stereo image matching device shown in FIG. 1 will be described below with reference to FIG. 2 .

FIG. 2 is a flowchart illustrating a stereoscopic image matching method according to an exemplary embodiment of the present invention. Referring to Fig. 2, in step S100, the matching cost between the pixels of the reference image and the pixels of the target image is calculated by the cost calculation unit 10, wherein, the cost calculation unit 10 calculates the difference between the pixels of the reference image and the pixels of the target image respectively A parametric statistical cost function and a color mean absolute difference function, and the non-parametric statistical cost function and the color mean absolute difference function are linearly combined, so that the result of the combination is used as the difference between the pixels of the reference image and the pixels of the target image Matching cost; in step S200, the matching cost between the pixels of the reference image calculated by the cost calculation unit 10 and the pixels of the target image is gathered by the cost aggregation unit 20; in step S300, the parallax determination unit 30 is based on the cost aggregation unit 20 Aggregate the matching cost to determine the disparity between the pixels of the reference image and the target image, so as to achieve stereo image matching. As a preferred but non-essential implementation, step S400 may also be additionally included, wherein the disparity map refinement unit 40 performs refinement processing on the disparity map formed by the disparity output by the disparity determination unit 30, thereby further improving the accuracy of stereo image matching. performance.

In the following, first, the process of calculating the matching cost between the pixels of the reference image and the pixels of the target image by the cost calculation unit 10 at step S100 will be described.

The cost calculation unit 10 receives a reference image and a target image, wherein, as an example, the reference image is a left image and the target image is a right image. Then, for each pixel p(x, y) in the left image, calculate its matching cost with the pixel pd(x-d, y) in the right image, where x, y indicate the abscissa and ordinate of pixel p respectively , d represents a given disparity level, and the value range of d can be set in units of integer pixels according to actual needs.

In the current exemplary embodiment according to the present invention, the cost calculation unit 10 respectively calculates the non-parametric statistical cost function and the color mean absolute difference function between the pixels of the reference image and the pixels of the target image, and calculates the non-parametric statistical cost function The cost function is linearly combined with the color mean absolute difference function, so that the result of the combination is used as the matching cost between the pixels of the reference image and the pixels of the target image.

Specifically, the cost calculation unit 10 first calculates the non-parametric statistical cost function between the pixel p(x, y) and the pixel pd(xd, y), in this case, selects the pixels around the pixel p(x, y) square window N _p and the square window N _pd around the corresponding pixel pd(xd, y), and respectively represent the pixels in the selected square window N _p and N _pd as a bit string in a certain order, in the In the above bit string, bit Bit _p (p ₁ ) represents the relative relationship between the intensity of pixel p(x, y) and the intensity of any pixel p ₁ in its square window N _p (for example, greater than, less than or approximately ), for example, when the intensity of the pixel p(x, y) is greater than the intensity of the pixel p ₁ , the value of Bit _p (p ₁ ) is set to 1, otherwise it is set to 0, which can be determined according to actual needs Set various specific comparison relationships; similarly, the relative relationship between the intensity of pixel pd (xd, y) and the intensity of any pixel pd1 in its square window N _pd is represented by bit Bit _pd (pd ₁ ) (for example , greater than, less than or approximately), for example, when the intensity of pixel pd(xd, y) is greater than the intensity of pixel pd ₁ , the value of Bit _pd (pd ₁ ) is set to 1, otherwise it is set to 0, where , various specific comparison relationships can be set according to actual needs.

Based on the above settings, the cost calculation unit 10 calculates the non-parametric statistical cost function C _census (p, d) between the pixel p (x, y) and the pixel pd (xd, y) according to the following equation 1:

$C_{\mathrm{census}}(p,d)=\underset{{\mathrm{<figure-callout\; id="1"\; label="p"\; filenames="HSA00000297375900062.png"\; state="\{\{state\}\}">p</figure-callout>}}_1\&Element;N,{\mathrm{<figure-callout\; id="1"\; label="pd"\; filenames="HSA00000297375900062.png"\; state="\{\{state\}\}">pd</figure-callout>}}_1\&Element;N_{\mathrm{pd}}}{\mathrm{\&Sigma;}}W({\mathrm{<figure-callout\; id="1"\; label="p"\; filenames="HSA00000297375900062.png"\; state="\{\{state\}\}">p</figure-callout>}}_1,p)\&CenterDot;W({\mathrm{<figure-callout\; id="1"\; label="pd"\; filenames="HSA00000297375900062.png"\; state="\{\{state\}\}">pd</figure-callout>}}_1,\mathrm{pd})\&Center\; Dot;({\mathrm{bit}}_p\left(p_1\right)\&CircleTimes;{\mathrm{bit}}_{\mathrm{pd}}\left({\mathrm{pd}}_1\right))$ Equation 1

Among them, W(p ₁ , p) and W(pd ₁ , pd) are the change weights of the Hamming distance between the square window N _p and N _pd respectively, and the value of W(p ₁ , p) can be determined according to the pixel p ₁ and The relationship between pixels _p in space and color can be calculated. Specifically, the value of W(p ₁ , p) can be obtained according to the following rules: when the distance between pixel p ₁ and pixel p is closer, W The larger the value of (p ₁ , p) is, at the same time, when the color of pixel p ₁ is closer to that of pixel p, the value of W(p ₁ , p) is also larger; similarly, the value of W(pd ₁ , pd) can be calculated according to pixel The relationship between pd ₁ and pixel pd in terms of space and color can be calculated. Specifically, the value of W(pd ₁ , pd) can be obtained according to the following rules: when the distance between pixel pd ₁ and pixel pd is greater Recently, the larger the value of W(pd ₁ , pd), at the same time, the closer the color of the pixel pd ₁ to the pixel pd, the larger the value of W(pd ₁ , pd).

After the cost calculation unit 10 calculates the non-parametric statistical cost function C _census (p, d) between the pixel p (x, y) and the pixel pd (xd, y) as above, the cost calculation unit 10 calculates the reference image The color mean absolute difference function between the pixels of and the pixels of the target image. As an example, Equation 2 can be used to calculate the color average absolute difference between the pixel p(x, y) and the pixel pd(xd, y) on the three color channels of R (red), G (green), and B (blue) Function C _AD (p, d):

$C_{\mathrm{AD}}(p,d)=\min \left(\frac{1}{3}\underset{R,G,B}{\mathrm{\&Sigma;}}\right|I\left(p\right)-I\left(\mathrm{pd}\right)|,{\mathrm{\&lambda;}}_{\mathrm{AD}})$ Equation 2

Among them, I(p) and I(pd) represent the intensity values of pixel p(x, y) and pixel pd(xd, y) under the corresponding R, G, B channels respectively, and λ _AD is the truncation threshold.

Then, the cost calculation unit 10 linearly combines the non-parametric statistical cost function C _census (p, d) and the color average absolute difference function C _AD (p, d) to use the result of the combination as the pixel of the reference image and The matching cost C(p,d) between pixels of the target image.

As an example, the cost calculation unit 10 calculates the matching cost C(p, d) between the pixels of the reference image and the pixels of the target image according to Equation 3:

C(p,d)=z(p)·C _census (p,d)+(1-z(p))·C _AD (p,d) Equation 3

Among them, z(p) is the weight used to control the proportional relationship between the non-parametric statistical cost function C _census (p, d) and the color average absolute difference function C _AD (p, d), and its value is between 0 and 1 . For example, z(p) may be adaptively set in consideration of local texture, local gradient and/or matching confidence information of each pixel p(x,y).

After obtaining the pixel matching cost function in the reference image as described above, in step S200, the cost aggregation unit 20 aggregates the matching costs between the pixels of the reference image calculated by the cost calculation unit 10 and the pixels of the target image, Thus, the cost aggregation result C _aggr (p, d) is generated.

Here, as an optional manner, the preferred aggregation scheme according to the exemplary embodiment of the present invention may be adopted, that is, the cost aggregation is performed based on the dynamic support region of each pixel. Hereinafter, the detailed structure of the cost aggregation unit according to the exemplary embodiment of the present invention will be described in detail with reference to FIG. 3 .

FIG. 3 is a block diagram illustrating a detailed structure of a cost aggregation unit according to an exemplary embodiment of the present invention. Referring to FIG. 3 , as an exemplary structure, the cost aggregation unit 20 may include: a pixel dynamic support domain construction unit 201 for constructing a dynamic support domain for each pixel in a reference image; a dynamic support domain cost aggregation unit 202 for For each dynamic support domain constructed, cost aggregation is performed on all pixels in it.

As known to those skilled in the art, the constructed dynamic support domain serves as the basis for cost aggregation for each pixel, and its shape and size have a great influence on the accuracy of the matching result. For example, a larger support domain needs to be constructed for an area with less texture, but if the support domain crosses the boundary of an object in the image, the matching performance at the object boundary will be poor. On the other hand, smaller support domains are also more sensitive to image noise and inflection points. Therefore, considering the above factors, the pixel dynamic support region construction unit 201 can follow the following two criteria to construct the dynamic support region of each pixel: (1) the pixels in the dynamic support region have a color similar to the central pixel of the support region (2) The boundary of the dynamic support domain should follow the existing edges in the reference image, where the local image gradient information of the pixel can be used to detect the existing edges in the reference image. Fig. 4 shows an example of constructing a pixel dynamic support domain according to an exemplary embodiment of the present invention. As shown in FIG. 4 , the dynamic support domains for pixel P and pixel Q are respectively constructed based on the above two criteria.

After the pixel dynamic support domain construction unit 201 constructs the dynamic support domain of each pixel, the dynamic support domain cost aggregation unit 202 performs cost aggregation on all the pixels in each constructed dynamic support domain.

As an example, the dynamic support region cost aggregation unit 202 may perform cost aggregation on all pixels in the dynamic support region of the pixel P in the manner shown in FIG. 5 . As shown in Figure 5, the dynamic support region cost aggregation unit 202 can perform cost aggregation on all pixels in the dynamic support region of pixel P according to two paths: first, perform cost aggregation on all pixels in each row along the horizontal direction, and obtain Each aggregation result in the horizontal direction is stored, and then the stored focusing results in the horizontal direction are aggregated along the vertical direction.

Those skilled in the art should understand that the above-mentioned example of performing cost matching processing based on the dynamic support domain of pixels by the cost aggregation unit 20 shown in FIG. It is not limited to the specific cost function generated by the cost calculation unit 10 of FIG. 1 , but can be applied to a matching cost function of various pixels. Although the dynamic domain described above is a two-dimensional spatial domain, the three-dimensional support domain of the spatial and disparity domains can also be applied in the present invention. In addition, those skilled in the art should also understand that the pixel dynamic support domain construction unit 201 and the dynamic support domain cost aggregation unit 202 shown in FIG. . That is to say, the cost aggregation unit 20 can be used as a single entity unit to implement the overall operation performed by the pixel dynamic support domain construction unit 201 and the dynamic support domain cost aggregation unit 202, or the cost aggregation unit 20 can be further subdivided into more units to perform the operations.

其中，p表示参考图像中的任一像素，d_p为该像素p的视差值。如果直接对聚集的代价结果应用局部WTA(Winner Take All，胜者通吃)算法来选择分别对每个像素p而言具有最小匹配代价的视差，则产生的视差图中会存在很多噪声深度值。因此，作为一种优选的方式，可采用包括对深度平滑性进行附加约束的全局算法，将确定视差的处理构建为确定使全局能量得到最小值的视差级别集合

作为确定的视差图。Referring back to FIG. 2, after the aggregation result of the cost function is obtained as described above, in step S300, the parallax of the pixels of the reference image to the target image can be determined by the parallax determination unit 30 based on the result of the aggregation of the matching cost by the cost aggregation unit 20 . Specifically, the disparity determining unit 30 can use an energy optimizer to calculate a disparity map, that is, based on a specific energy operation, determine a disparity level set that minimizes the matching cost

Among them, p represents any pixel in the reference image, and d _p is the disparity value of the pixel p. If the local WTA (Winner Take All) algorithm is directly applied to the aggregated cost results to select the disparity with the smallest matching cost for each pixel p, the resulting disparity map will contain many noisy depth values . Therefore, as a preferred method, a global algorithm including additional constraints on depth smoothness can be used, and the process of determining the disparity can be constructed as a set of disparity levels that minimize the global energy

as a definite disparity map.

As an example, the disparity determining unit 30 may construct the global energy of the matching cost with a depth smoothness constraint according to Equation 4, and the disparity level set that makes the global energy obtain the minimum value is determined as the final stereoscopic image matching disparity map:

${\mathrm{E.}}_{\min }\left(\mathrm{D.}\right)=\underset{p}{\mathrm{\&Sigma;}}(C_{\mathrm{aggr}}(p,d_p)+\underset{q\&Element;N}{\mathrm{\&Sigma;}}{P}_{1}T[|d_p-d_q|=1]+\underset{q\&Element;N}{\mathrm{\&Sigma;}}{P}_{2}T[|d_p-d_q|>1\left]\right)$ Equation 4

Among them, C _aggr (p, d _p ) is the aggregation result of the matching cost, N is the neighborhood of pixel p, which can be determined according to actual needs, q is the pixel in the neighborhood N, and d _q is the disparity value of pixel q , T(A) is a truth function, that is, when the A equation is established, the value of T(A) is 1, otherwise, the value of T(A) is 0, and P ₁ and P ₂ are penalty parameters, which can be adjusted according to actual needs to set. In addition, as a preferred method, the 4-direction scan line optimization method can be used to effectively perform the above determination processing flow.

Referring back to FIG. 2 , after the disparity map from the pixels of the reference image to the target image is determined by the disparity determination unit 30, as a preferred but not necessary implementation, step S400 may be additionally included, wherein the disparity map refinement unit 40 Thinning processing is performed on the disparity map output by the disparity determination unit 30, so as to further improve the performance of stereoscopic image matching.

Hereinafter, the detailed structure of the disparity map refinement unit 40 according to an exemplary embodiment of the present invention will be described with reference to FIG. 6 .

FIG. 6 is a block diagram illustrating detailed results of a disparity map refinement unit according to an exemplary embodiment of the present invention. Referring to FIG. 6, as an exemplary structure, the disparity map refinement unit 40 may include: an occlusion point processing unit 401, configured to detect an occlusion point in the disparity map, and use images around the occlusion point to estimate the disparity of the occlusion point; The continuous boundary correction unit 402 is configured to detect a discontinuous boundary where the disparity value jumps in the disparity map processed by the occlusion point processing unit 401, and for pixels on the discontinuous boundary, use The disparity of the pixels on the discontinuous boundary is re-determined by pixels at a certain distance from the pixel; the sub-pixel level difference unit 403 is used to perform sub-pixel level difference on the disparity map corrected by the discontinuous boundary correction unit 402 Value operations.

Specifically, the occlusion point processing unit 401 detects occlusion points in the disparity map (ie, points whose disparity values cannot be determined by corresponding pixels in the target image) by checking the left-right consistency between the reference image and the target image. Then, as shown in (a) in Figure 7, the disparity value of the occlusion point is first estimated by using other pixels in the dynamic support domain of the occlusion point, for example, the disparity values of all pixels in the dynamic support domain can be Among the values, the most common disparity value is used as the disparity value of the occluded point. In this way, the parallax values of some or all occlusion points can be determined. For occlusion points whose parallax values still cannot be determined, as shown in (b) in Figure 7, starting from the occlusion point along multiple directions Scanning is performed in (for example, 16 directions), and the disparity value of the pixel whose color is closest to the occlusion point among the pixels scanned along these directions is determined as the disparity value of the occlusion point.

Then, the discontinuous boundary correction unit 402 detects a discontinuous boundary whose disparity value jumps (for example, the difference of the disparity value>2) in the disparity map processed by the occlusion point processing unit 401, and then, for the discontinuous boundary For pixels on the discontinuous boundary, the discontinuous boundary correction unit 402 uses two pixels on both sides of the discontinuous boundary separated from the pixel by a certain distance (for example, 4 pixels) to re-determine the disparity of the pixel on the discontinuous boundary. For example, the color values of the two pixels may be compared with the pixels on the discontinuous boundary, and the disparity value of the pixel whose color is closer to the pixel on the discontinuous boundary is selected as the boundary The disparity value of the pixel on . As a preferred manner, the color information of the dynamic support domain of the pixel can be used to represent the color value of the pixel, thereby reducing the corresponding calculation consumption. The results of the discontinuous boundary correction processing described above are shown in FIG. 8 . In FIG. 8 , the left figure shows the detected discontinuous boundary before correction, and the right figure shows the result after correction.

Then, the sub-pixel level difference unit 403 can perform a sub-pixel level difference operation on the disparity map corrected by the discontinuous boundary correction unit 402 . Since the previous processing is based on the integer pixel level, in order to further refine the generated disparity map, for example, the difference operation at the sub-pixel level can be performed according to Equation 5:

$d^{*}=d-\frac{\mathrm{E.}(p,d+1)-\mathrm{E.}(p,d-1)}{2(\mathrm{E.}(p,d+1)+\mathrm{E.}(p,d-1)-2\mathrm{E.}(p,d))}$ Equation 5

where d ^* is the disparity value at the sub-pixel level, and E represents the energy calculated by Equation 4.

Those skilled in the art should understand that the above-mentioned occlusion point processing unit 401, discontinuous boundary correction unit 402 and sub-pixel level difference unit 403 shown in FIG. Structural constraints of unit 40. That is to say, the disparity map refinement unit 40 can completely implement the operations performed by the occlusion point processing unit 401, the discontinuous boundary correction unit 402, and the sub-pixel level difference unit 403 as a single entity unit, or the disparity map refinement unit 40 can also be further subdivided into more units to perform the operations. In addition, the above-mentioned occlusion point processing unit 401, discontinuous boundary correction unit 402 and sub-pixel level difference unit 403 respectively perform three independent disparity map refinement processes, therefore, the disparity map refinement unit 40 may only include the above three units Any one or two of them, and there is no fixed order of setting among the three units.

The above shows the device and method for stereo image matching according to the exemplary embodiments of the present invention. Fig. 9 shows the improvement in performance of the stereo image matching system according to an exemplary embodiment of the present invention compared with the prior art, wherein (a) and (b) in Fig. 9 respectively show the error threshold =1 and the test result of error threshold=0.5, as shown in the figure, the stereoscopic image matching scheme (displayed as "YOUR METHOD" in the test result interface) according to the exemplary embodiment of the present invention is all obvious in terms of matching accuracy and processing efficiency ahead of other solutions in the prior art.

According to the present invention, the matching cost function can be calculated based on specific rules, and the cost aggregation is further carried out for the dynamic support domain of each pixel. On the basis of the cost aggregation, the global depth smoothness can be considered to generate the disparity map of the reference image. Therefore, the matching accuracy and the processing speed are simultaneously improved in the process of realizing stereoscopic image matching. In addition, the performance of image matching can be further improved by performing thinning and correction processing on the generated disparity map.

It should be noted that the stereoscopic image matching method and apparatus according to the exemplary embodiments of the present invention may be included in a generating device for 3D content, and may also be included in a display device for 3D content. In the above device, in addition to the stereoscopic image matching device according to the exemplary embodiment of the present invention, it also includes a 3D data input unit, a 3D data analysis unit, a 3D content generation unit or a 3D content display unit, since these units all belong to the present invention Therefore, in order to avoid confusing the subject matter of the present invention, no detailed description is given here.

The above respective embodiments of the present invention are merely exemplary, and the present invention is not limited thereto. Those skilled in the art should understand that any method for performing stereo image matching involving the above-mentioned matching cost processing, pixel dynamic support domain construction, disparity map generation and thinning respectively falls within the scope of the present invention. Changes may be made in these embodiments without departing from the principles and spirit of the invention, the scope of which is defined in the claims and their equivalents.

Claims (15)

1. stereo-picture matching unit, said equipment comprises:

The cost computing unit; Be used to calculate the coupling cost between the pixel of pixel and target image of reference picture; Wherein, The cost computing unit calculates nonparametric statistics cost function and the colored mean absolute difference value function between the pixel of pixel and target image of reference picture respectively, and said nonparametric statistics cost function and colored mean absolute difference value function are carried out linear combination, with the result that will the make up coupling cost between the pixel of pixel and the target image of image as a reference;

The cost accumulation unit is used for the coupling cost between the pixel of the pixel of the reference picture that is calculated by the cost computing unit and target image is assembled; And

Parallax is confirmed the unit, is used for confirming the parallax of the pixel of reference picture to target image based on the result that the cost accumulation unit is assembled the coupling cost, to realize the stereo-picture coupling.

2. equipment as claimed in claim 1; Wherein, Said cost accumulation unit is confirmed the dynamic support region of this pixel to each pixel in the reference picture; In each dynamic support region of confirming, the coupling cost between the pixel of the reference picture that will be calculated by the cost computing unit and the pixel of target image is assembled.

3. equipment as claimed in claim 1, wherein, said parallax is confirmed the global energy of the coupling cost of the subsidiary degree of depth flatness constraint of cell formation, and the parallax level set that will make global energy obtain minimum value is confirmed as final stereo-picture coupling disparity map.

4. equipment as claimed in claim 3, wherein, said cost accumulation unit comprises:

The dynamic support region construction unit of pixel is used for making up dynamic support region to each pixel of reference picture;

Dynamically support region cost accumulation unit is used for to each the dynamic support region that makes up, and all pixels is wherein carried out cost assemble.

5. equipment as claimed in claim 3 also comprises: disparity map refinement unit is used for the disparity map of being confirmed unit output by parallax is carried out thinning processing.

6. equipment as claimed in claim 5, wherein, said disparity map refinement unit comprises at least one in following:

Block a processing unit, be used for detecting blocking a little of disparity map, utilize the image around blocking a little to estimate to block parallax a little;

Discontinuous border amending unit; Be used for detecting the discontinuous border that saltus step appears in parallax value at disparity map; For discontinuous borderline pixel, the pixel that is utilized in said discontinuous boundaries on either side and said pixel separation certain distance to confirm again the parallax of discontinuous borderline pixel;

The sub-pixel-level difference unit is used for disparity map is carried out other difference computing of sub-pixel-level.

7. equipment as claimed in claim 1; Wherein, In said linear combination, the ratio between said nonparametric statistics cost function and the colored mean absolute difference value function comes adaptively to be provided with based on local grain, partial gradient and/or the coupling confidential information of each pixel.

8. stereo-picture matching process, said method comprises:

Coupling cost between the pixel of calculating reference picture and the pixel of target image; Wherein, Calculate nonparametric statistics cost function and colored mean absolute difference value function between the pixel of pixel and target image of reference picture respectively; And said nonparametric statistics cost function and colored mean absolute difference value function carried out linear combination, with the result that will the make up coupling cost between the pixel of pixel and the target image of image as a reference;

Coupling cost between the pixel of the pixel of the reference picture that calculates and target image is assembled; And

Confirm the parallax of the pixel of reference picture based on the result that the coupling cost is assembled, to realize the stereo-picture coupling to target image.

9. method as claimed in claim 8; Wherein, The step of assembling comprises: the dynamic support region of confirming this pixel to each pixel in the reference picture; In each dynamic support region of confirming, the coupling cost between the pixel of the pixel of the reference picture that calculates and target image is assembled.

10. method as claimed in claim 8; Wherein, The step of confirming parallax comprises: make up the global energy of the coupling cost of subsidiary degree of depth flatness constraint, and the parallax level set that will make global energy obtain minimum value is confirmed as final stereo-picture coupling disparity map.

11. method as claimed in claim 10 also comprises: the stereo-picture coupling disparity map to final carries out thinning processing.

12. method as claimed in claim 11, wherein, at least one during the step that final stereo-picture coupling disparity map is carried out thinning processing may further comprise the steps:

Detect blocking a little in the disparity map, utilize the image around blocking a little to estimate to block parallax a little;

In disparity map, detect the discontinuous border that saltus step appears in parallax value, for discontinuous borderline pixel, the pixel that is utilized in said discontinuous boundaries on either side and said pixel separation certain distance to confirm again the parallax of discontinuous borderline pixel;

Disparity map is carried out other difference computing of sub-pixel-level.

13. method as claimed in claim 8; Wherein, In said linear combination, the ratio between said nonparametric statistics cost function and the colored mean absolute difference value function comes adaptively to be provided with based on local grain, partial gradient and/or the coupling confidential information of each pixel.

14. a stereo-picture matching unit, said equipment comprises:

The cost computing unit is used to calculate the coupling cost between the pixel of pixel and target image of reference picture;

The cost accumulation unit; Be used for confirming the dynamic support region of this pixel to each pixel of reference picture; In each dynamic support region of confirming, the coupling cost between the pixel of the reference picture that will be calculated by the cost computing unit and the pixel of target image is assembled; And

Parallax is confirmed the unit, is used for confirming the parallax of the pixel of reference picture to target image based on the result that the cost accumulation unit is assembled the coupling cost, to realize the stereo-picture coupling.

15. a stereo-picture matching process, said method comprises:

Coupling cost between the pixel of calculating reference picture and the pixel of target image;

Confirm the dynamic support region of this pixel to each pixel in the reference picture, in each dynamic support region of confirming, the coupling cost between the pixel of the reference picture that will be calculated by the cost computing unit and the pixel of target image is assembled; And

Confirm the parallax of the pixel of reference picture based on the result that the coupling cost is assembled, to realize the stereo-picture coupling to target image.

Images