KR101326279B1 - Motion estimation system for stereoscopic videos and method thereof - Google Patents

Abstract

The present invention provides an initial difference estimator for calculating initial difference information using a first left image frame and a first right image frame associated with a first view, and a predetermined size of the first left image frame and the first right image frame. A block matching unit which selects a left matching block and a right matching block by dividing into blocks each having a block, a motion estimating unit estimating a left motion vector and a right motion vector using the left matching block and the right matching block, and the left motion Provided is a motion estimation system for stereo video including a difference map updater for updating the initial difference information using a vector and the right motion vector.

Description

MOTION ESTIMATION SYSTEM FOR STEREOSCOPIC VIDEOS AND METHOD THEREOF

The present invention relates to a method for estimating the motion of stereo video.

1 is a diagram illustrating left and right images of a stereo video according to the prior art.

Referring to Figure 1, stereoscopic video (stereoscopic video) allows a person to feel a three-dimensional feeling using binocular parallax. That is, when the image of the parallax is shown to both eyes, the distance of the object is virtually recognized by the binocular parallax and the distance relationship between the object in the human brain. What is important here is that all objects of the images 110 and 120 input by the two eyes have a horizontal position difference only, which is called disparity (hereinafter, referred to as “difference”).

When the left image 110 and the left image 120 and the right view 120 are overlapped as one, it can be seen that only a horizontal difference exists (130). This difference is inversely proportional to the distance between the camera and the object. In other words, when the object is far away, the difference is small, and when the object comes near the camera, the difference is large.

In video playback, frame rate conversion may be applied to provide more vivid image quality to the user. Frame rate conversion is a technology that provides a clearer picture quality to a person viewing the content by increasing the frame rate of the content. That is, increasing the frame rate means inserting and playing artificially created frames between adjacent screen frames in the content. At this time, the most effective method of creating an inserted frame is the MCFI (Motion Compensated Frame Interpolation) method that detects the movement of all the objects in the screen and places each object in the middle of the movement to create an insertion frame.

2 is a diagram illustrating an example of generating an intermediate image frame according to the prior art.

Referring to FIG. 2, on the current 3D TV that plays stereo video, the MCFI scheme is independently applied to the left image (a) and the right image (b) to perform frame rate conversion. That is, at n time point 210, the left image (a) coming into the left eye is regarded as the existing 2D video and the MCFI scheme is applied, and the right image (b) applies the MCFI scheme as the right image. If the image frame of the n + 1 time point 230 generated after the n time point 210 is an ideal case in which the motion of each object can be accurately estimated, the MCFI method is applied to the left image (a) and the right image (b). Even if it is applied independently, by placing the objects in the intermediate position of the movement, it is possible to make the intermediate image frame 220 that there is only a horizontal position difference between the objects.

However, although many MCFI algorithms have been developed to improve performance, accurate motion estimation for all images is limited. This means that in case of applying the MCFI method independent of each time as described above, due to inaccurately estimated motion information, not only the horizontal position difference but also the vertical position difference may be generated between objects in the intermediate image frame. it means. This does not meet the difference condition stereo video should have, and if the same object is seen in both left and right sides in different vertical positions, our brain will not be able to accurately perceive the stereoscopic feeling and feel uncomfortable.

One embodiment of the present invention constrains the corresponding portions in the left and right views to have the same vertical movement with respect to the same object in consideration of the left and right views at the same time, so that the final movement of the corresponding portions has only a difference. And a motion estimation system and method for stereo video.

According to an embodiment of the present invention, initial difference information is calculated using left and right image frames with respect to a first viewpoint, and thus difference information of left and right image frames generated after the first viewpoint is obtained using initial difference information. Provided are a predictable, motion estimation system and method for stereo video.

According to an embodiment of the present invention, the left and right image frames are divided into blocks having a predetermined size to calculate left and right motion vectors, and the calculated left and right motion vectors are reflected on the left and right image frames. A motion estimation system and method for stereo video, which can calculate position difference information of a right image frame, are provided.

According to an embodiment of the present invention, a motion estimation system for stereo video includes an initial difference estimator configured to calculate initial difference information using a first left image frame and a first right image frame associated with a first viewpoint, and the first left side. A block matching unit for dividing the image frame and the first right image frame into blocks having a predetermined size to select a left matching block and a right matching block, a left motion vector and a right using the left matching block and the right matching block. A motion estimator for estimating a motion vector, and a difference map updater for updating the initial difference information using the left motion vector and the right motion vector.

The block matching unit calculates average difference information by using difference information in each of the divided blocks, and selects left and right blocks that overlap with the average difference information by more than a reference value as left matching blocks and right matching blocks.

The motion estimator may calculate position difference information by reflecting the left motion vector and the right motion vector in a second left image frame and a second right image frame respectively associated with a second viewpoint different from the first viewpoint.

The difference map updater may predict final difference information associated with the second time point by using the updated initial difference information.

The motion estimator may calculate a motion vector pair by using the left motion vector, the right motion vector, and the position difference information.

The difference map updater may obtain final difference information associated with the second viewpoint by using the motion vector pair.

The motion estimator may estimate a left motion vector and a right motion vector such that the left matching block and the right matching block have the same vertical displacement.

According to an embodiment of the present invention, a motion estimation method for stereo video includes calculating initial difference information using a first left image frame and a first right image frame associated with a first viewpoint, the first left image frame and Selecting a left matching block and a right matching block by dividing the first right image frame into blocks having a predetermined size, and estimating a left motion vector and a right motion vector using the left matching block and the right matching block. And updating the initial difference information by using the left motion vector and the right motion vector.

According to an embodiment of the present invention, the left and right views may be considered at the same time so that the corresponding parts in the left and right views have the same vertical movement with respect to the same object, so that the final movement of the corresponding portions may have only a difference. have.

According to an embodiment of the present invention, by calculating the initial difference information using the left and right image frames for the first time point, the difference between the left and right image frames generated after the first time point using the initial difference information. Information can be predicted.

1 is a diagram illustrating left and right images of a stereo video according to the prior art.
2 is a diagram illustrating an example of generating an intermediate image frame according to the prior art.
3 is a block diagram illustrating a configuration of a motion estimation system for stereo video according to an embodiment of the present invention.
4 is a diagram illustrating an example of obtaining initial difference information according to an embodiment of the present invention.
5 is a diagram illustrating an example of selecting a matching block according to an embodiment of the present invention.
6 illustrates an example of finding a matching block in an image frame according to an embodiment of the present invention.
7 is a diagram illustrating an example of calculating a motion vector and position difference information according to an embodiment of the present invention.
8 is a diagram illustrating an example of obtaining final difference information according to an embodiment of the present invention.
9 is a flowchart illustrating a procedure of a motion estimation method for stereo video according to an embodiment of the present invention.

Hereinafter, various embodiments of the present invention will be described in detail with reference to the accompanying drawings and accompanying drawings, but the present invention is not limited to or limited by the embodiments.

3 is a block diagram illustrating a configuration of a motion estimation system for stereo video according to an embodiment of the present invention.

Referring to FIG. 3, the motion estimation system 300 (hereinafter, referred to as a 'motion estimation system') for stereo video may include a block matching unit 310, an initial difference estimation unit 320, a motion estimation unit 330, and a difference. The map updater 340 is included.

을 이용하여 제1 좌측 영상 프레임에 대한 좌측 초기 차이정보와 제1 우측 영상 프레임에 대한 우측 초기 차이정보 (

)를 산출할 수 있다. The initial difference estimator 320 calculates initial disparity information using the first left image frame and the first right image frame associated with the first viewpoint. Here, the first viewpoint may be a current viewpoint, and the initial difference estimation unit 320 is a left and right image frame input at the current viewpoint.

Left initial difference information for the first left image frame and right initial difference information for the first right image frame using

) Can be calculated.

)는 상기 제1 시점 이후에 발생된 제2 시점의 좌측, 우측 영상 프레임(

)의 움직임을 예측하는데 사용될 수 있다.There are various methods of calculating the initial difference information, but obtaining accurate difference information has limitations similar to motion estimation. Therefore, the initial difference information is approximate and can be used according to the required speed and accuracy according to the environment of the system. The initial difference information is then updated to more accurate final difference information using a motion vector obtained after the final motion prediction is performed in the difference map updater 340, and the updated final difference information (

) Is the left and right image frames (2) of the second view generated after the first view.

) Can be used to predict

4 is a diagram illustrating an example of obtaining initial difference information according to an embodiment of the present invention.

Referring to FIG. 4, the initial difference information 420 may be obtained by dividing an image of 410 by a small block unit. Subsequently, the initial difference information 420 may be updated by the difference map updater 340 to obtain final difference information 430. The final difference information 430 may include the second difference generated after the first time. Used to predict video frame movement.

Since stereoscopic video is composed of two videos with different parallaxes, one object in space is simultaneously displayed in left and right image frames. Therefore, when the left and right image frames are divided into small blocks, the block matching unit 310 determines which blocks of the other image frames contain the contents contained in the blocks in one image frame, that is, one part of the same object is left and right. The motion estimation unit 330 may inform the motion estimation unit 330 of which blocks of the image frame are contained in each other.

To this end, the block matching unit 310 divides the first left image frame and the first right image frame into blocks having a predetermined size, and calculates average difference information by using difference information in each of the divided blocks. do. The average difference information indicates how shifted the contents of the block appear in different parallax images.

5 is a diagram illustrating an example of selecting a matching block according to an embodiment of the present invention.

Referring to FIG. 5, the block matching unit 310 may divide the first left image frame 510 and the first right image frame 530 into blocks having a predetermined size in a rectangular shape. The block matching unit 310 may calculate average difference information by using difference information in each of the divided blocks. The block matching unit 310 may move (arrow) the left block by the average difference information from the first left image frame 510 (520). The block matching unit 310 may move (arrow) the right block by the average difference information from the first right image frame 530 (540). Arrows indicate average difference information.

In general, since the average difference information is expressed in pixel units, it does not exactly match the block grid. Therefore, the block matching unit 310 first selects the most overlapping block as a matching block. For example, the block matching unit 310 may select a block overlapping with a reference value (50%) or more as a matching block. In FIG. 5, block 3 may be selected as the left matching block and block d may be selected as the right matching block. The block matching unit 310 may go through a process of verifying once more that the left and right matching blocks are correct. For example, the block matching unit 310 calculates the average difference information of the block d, and when moving by the calculated average difference information, it is possible to check whether the first overlap block 3 most overlap. Accordingly, the block matching unit 310 may select {3, d} as the left matching block and the right matching block that have passed the verification.

The block matching unit 310 selects a left matching block for all the divided blocks of the first left image frame 510, and selects a right matching block for all the divided blocks of the first right image frame 530. Can be selected.

6 illustrates an example of finding a matching block in an image frame according to an embodiment of the present invention.

Referring to FIG. 6, the block matching unit 310 selects a left matching block 630 for all blocks included in the first left image frame 610 and is included in the first right image frame 620. The right matching block 640 may be selected for all blocks. In the drawing, blocks indicated by white have selected matching blocks, and black blocks have shown failures in selecting matching blocks.

In general, an error function (Error (/ Cost) Function) must be defined first to determine a motion vector. The motion estimator 330 may calculate an error value using an error function defined for various possible motion vector candidates, and determine a motion vector representing a minimum error as the final motion vector.

The motion estimator 330 estimates a left motion vector and a right motion vector using the left matching block and the right matching block. That is, the motion estimation unit 330 may estimate a left motion vector and a right motion vector such that the left matching block and the right matching block have the same vertical displacement (y displacement). This is for the motion estimator 330 to estimate the motion vector such that the left / right matching blocks have the same vertical displacement even though the horizontal displacement (x displacement) is different.

The motion estimation unit 330 may estimate a left motion vector and a right motion vector with respect to one left matching block and one right matching block among the plurality of left matching blocks and the plurality of right matching blocks.

을 산출할 수 있다. 이때 가능한 움직임 벡터쌍 후보들 집합

(

)은 동일한 수직적 움직임을 갖는 벡터들로 제한한다. 따라서, 동일 물체를 담고 있는 좌, 우측의 두 매칭 블록이 동일한 수직적 움직임을 갖게 돼, 이후 프레임율 변환 시 만들어지는 중간 영상 프레임에서 차이정보만 존재하게 된다.For example, the motion estimator 330 uses the left motion vector, the right motion vector, and position difference information to pair a motion vector.

Can be calculated. Set of possible motion vector pair candidates

(

) Is limited to vectors with the same vertical movement. Accordingly, two matching blocks on the left and right sides containing the same object have the same vertical movement, and thus only the difference information exists in the intermediate image frame generated during the frame rate conversion.

V ^L is the left motion vector, V ^R is the right motion vector, M ^L is the difference information with the motion-compensated position by the left motion vector, M ^R is the difference information with the motion-compensated position by the right motion vector, and M ^D is the difference A subfunction that indicates inconsistency of information, depending on V ^L , V ^R , and D.

The motion estimation unit 330 may calculate the position difference information M ^D using Equation 1. For example, the motion estimation unit 330 calculates position difference information by reflecting the left motion vector and the right motion vector in the second left image frame and the second right image frame respectively associated with the second viewpoint different from the first viewpoint. can do.

The motion estimator 330 may calculate M ^L and M ^R using Sum of Squared Differences (SSD) or Mean of Absolute Differences (MAD). Since the two values depend only on the motion vectors to be examined, the two values can be expressed as a function of the motion vectors.

7 is a diagram illustrating an example of calculating a motion vector and position difference information according to an embodiment of the present invention.

Referring to FIG. 7, since the two motion blocks 710 and 720 represent the same object in the real space, the left motion vector and the right motion vector represent the exact motion of the object. This relationship must be maintained between the regions indicated by the two motion vectors in the left and right image frames (730 and 740).

에 의해 n+1번째 좌우 영상 내에 지시되는 블록 영역

과

을 찾을 수 있는데, 이 두 블록 영역도 서로 동일한 물체를 담고 있어야 한다. 이를 확인하기 위해, 움직임 추정부(330)는 상기 제1 시점 이후(시간상 늦음)에 발생된 제2 시점(n+1번째) 좌측 영상 프레임과 우측 영상 프레임에 대해 만들어진 차이정보를 이용할 수 있다. 움직임 추정부(330)는

영역 내의 평균 차이정보를 D라 하면, 수학식 2가 성립된다.That is, the motion estimator 330 candidates for the motion vector pair for the two matching blocks indicated by the thick solid line in the left image frame and the right image frame of the first viewpoint (n-th).

Block area indicated in the n + 1th left and right images by

and

We can find that these two block areas must contain the same object. In order to confirm this, the motion estimator 330 may use the difference information generated for the left image frame and the right image frame of the second view (n + 1th) generated after the first view (late in time). The motion estimation unit 330

If the average difference information in the area is D, equation (2) is established.

Here, P ^L is the center position of the left matching block, P ^R is the center position of the right matching block, V ^L is the left motion vector, V ^R is the right motion vector, and D is the average difference information.

The motion estimator 330 may set the equation 3 as an error term M ^D indicating an inconsistency of difference information using Equation 2.

α is a constant, which determines the final motion vector pair, and adjusts the degree of influence of difference information mismatch.

이 결정되면, 실제 해당 부분

과

의 차이정보는 각각

과

이라 생각할 수 있기 때문에, 이미 얻어진 초기 차이정보에 해당 값을 적용하여 최종 차이정보의 정확도를 높일 수 있다.The difference map data unit 340 may obtain final difference information associated with the second viewpoint by using the motion vector pair. For example, the difference map data unit 340 is a final motion vector pair from equation (4).

Once this is determined, the actual corresponding part

and

The difference information of each

and

Since it can be considered, it is possible to increase the accuracy of the final difference information by applying the corresponding value to the already obtained initial difference information.

8 is a diagram illustrating an example of obtaining final difference information according to an embodiment of the present invention.

가 위치되고,

값으로 채워진 블록이 있다면, 최종 차이정보(820)는 수학식 5와 같이 초기 차이정보와 가중 평균하여 업데이트될 수 있다.Referring to FIG. 8, since the initial difference information obtains average difference information in units of blocks, a block form remains in the difference information. In 810, determined by the final motion vector

Is located,

If there is a block filled with values, the final difference information 820 may be updated by weighted average of the initial difference information as shown in Equation 5.

는 n+1번째 좌측 영상 프레임에 대한 최종 차이정보,

는 n+1번째 좌측 영상 프레임에 대한 초기 차이정보이다.β is a constant,

Is the final difference information for the n + 1th left image frame,

Is initial difference information for the n + 1th left image frame.

Accordingly, the difference map updater 340 may predict final difference information associated with the second time point by using the initial difference information.

9 is a flowchart illustrating a procedure of a motion estimation method for stereo video according to an embodiment of the present invention.

Referring to FIG. 9, in step S910, a motion estimation method (hereinafter, referred to as a 'motion estimation method') for stereo video uses an initial difference using a first left image frame and a first right image frame associated with a first viewpoint. Calculate the information.

In operation S920, the motion estimation method selects a left matching block and a right matching block by dividing the first left image frame and the first right image frame into blocks having a predetermined size, respectively.

For example, the motion estimation method calculates average difference information by using difference information in each of the divided blocks, and replaces the left and right blocks that overlap the average difference information with a reference value or more as a left matching block and a right matching block. Can be selected. In this case, the motion estimation method may select a matching block for each of the divided blocks.

In step S930, the motion estimation method estimates a left motion vector and a right motion vector using the left matching block and the right matching block. In this case, the motion estimation method may estimate a left motion vector and a right motion vector such that the left matching block and the right matching block have the same vertical displacement.

The motion estimation method may calculate position difference information by reflecting the left motion vector and the right motion vector in a second left image frame and a second right image frame respectively associated with a second viewpoint different from the first viewpoint. The motion estimation method may calculate a motion vector pair by using the left motion vector, the right motion vector, and the position difference information.

In step S940, the motion estimation method updates the initial difference information by using the left motion vector and the right motion vector. The motion estimation method may predict final difference information associated with the second viewpoint by using the updated initial difference information. In addition, the motion estimation method may obtain final difference information associated with the second viewpoint by using the motion vector pair.

Methods according to an embodiment of the present invention may be implemented in the form of program instructions that can be executed by various computer means and recorded in a computer readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the medium may be those specially designed and constructed for the present invention or may be available to those skilled in the art of computer software. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

While the invention has been shown and described with reference to certain preferred embodiments thereof, it will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. This is possible. Therefore, the scope of the present invention should not be limited to the described embodiments, but should be determined by the equivalents of the claims, as well as the claims.

300: motion estimation system for stereo video
310: block matching unit
320: initial difference estimation unit
330: motion estimation unit
340: difference map updating unit

Claims (13)

An initial difference estimator configured to calculate initial disparity information using the first left image frame and the first right image frame associated with the first viewpoint;
A block matching unit which selects a left matching block and a right matching block by dividing the first left image frame and the first right image frame into blocks having a predetermined size;
A motion estimation unit estimating a left motion vector and a right motion vector using the left matching block and the right matching block; And
A difference map updating unit updating the initial difference information by using the left motion vector and the right motion vector
Motion estimation system for stereo video comprising a.
The method of claim 1,
The block matching unit,
Average difference information is calculated using the divided difference information in each of the divided blocks, and the left and right blocks overlapping the average difference information with a reference value or more are selected as a left matching block and a right matching block, motion estimation for stereo video. system.
The method of claim 1,
The motion estimation unit,
And calculating position difference information by reflecting the left motion vector and the right motion vector to a second left image frame and a second right image frame respectively associated with a second viewpoint different from the first viewpoint. .
The method of claim 3,
The difference map update unit,
And predicting final difference information associated with the second viewpoint using the updated initial difference information.
The method of claim 3,
The motion estimation unit,
Calculating a motion vector pair consisting of the left motion vector and the right motion vector.
The method of claim 5,
The difference map update unit,
And using the motion vector pair to obtain final difference information associated with the second viewpoint.
The method of claim 1,
The motion estimation unit,
Estimating a left motion vector and a right motion vector such that the left matching block and the right matching block have the same vertical displacement.
Calculating initial difference information using the first left image frame and the first right image frame associated with the first viewpoint;
Selecting a left matching block and a right matching block by dividing the first left image frame and the first right image frame into blocks having a predetermined size;
Estimating a left motion vector and a right motion vector using the left matching block and the right matching block; And
Updating the initial difference information by using the left motion vector and the right motion vector.
Motion estimation method for stereo video comprising a.
9. The method of claim 8,
Selecting a left matching block and a right matching block,
Calculating average difference information using difference information in each of the divided blocks; And
Selecting a left block and a right block that overlap with the average difference information by more than a reference value as a left matching block and a right matching block
Motion estimation method for stereo video comprising a.
9. The method of claim 8,
Estimating the left motion vector and the right motion vector,
Calculating position difference information by reflecting the left motion vector and the right motion vector in a second left image frame and a second right image frame respectively associated with a second viewpoint different from the first viewpoint.
Further comprising a motion estimation method for stereo video.
The method of claim 10,
Estimating the left motion vector and the right motion vector,
Calculating a motion vector pair consisting of the left motion vector and the right motion vector
Further comprising a motion estimation method for stereo video.
12. The method of claim 11,
Updating the initial difference information,
Acquiring final difference information associated with the second viewpoint by using the motion vector pair
Including a motion estimation method for stereo video.
9. The method of claim 8,
Estimating the left motion vector and the right motion vector,
Estimating a left motion vector and a right motion vector such that the left matching block and the right matching block have the same vertical displacement.
Including a motion estimation method for stereo video.

Images