JP2014183544A - Image processing unit and image processing method - Google Patents

Abstract

Motion vector detection accuracy is improved and image quality deterioration is suppressed.
An image processing apparatus is determined by a motion vector search unit for a block of a processing target image or a sub-block obtained by dividing the block, each of a plurality of sub-blocks obtained by dividing the block of interest, and a motion vector of the block of interest. A first error calculation unit for calculating an error between each of the plurality of sub-blocks and a reference image region, and a predetermined sub-block based on the magnitude of the absolute value of the plurality of first errors A reference image area corresponding to the predetermined sub-block determined by a sub-block specifying unit for specifying the block, a motion vector of an adjacent block adjacent to the block of interest, and a relative position in the block of interest, and the predetermined sub-block A second error calculation unit for calculating a second error between the first error and the predetermined error based on the first error and the second error. And a motion vector determiner for determining the motion vector of the sub-blocks.
[Selection] Figure 2

Description

  The present invention relates to an image processing apparatus and an image processing method for processing an image such as a video.

  As one of video compression methods, image prediction using block motion vectors is used. The motion vector is a value indicating a spatial deviation in units of blocks obtained by using a block matching method for searching for a position in the reference image that is most similar to the block of the processing target image. It is possible to reduce the amount of video data by obtaining a difference value between pixel values of a reference image area determined by the motion vector and a block of the processing target image.

  In block matching type motion vector search, in order to suppress the degradation of detection accuracy due to thinning out pixels in the block, blocks containing the same motion are integrated, and the pixels in the block are thinned out according to the size of the integrated block. After matching is performed, there is a technique in which the integrated block is divided again into blocks, and the motion vector obtained by the integrated block is used as an initial transition of each block to search again in a minute region (for example, Patent Document 1). reference).

  In addition, a small block and a large block motion vector candidate that minimize the block matching error calculated for each small block and each large block in the first search mode are searched, and a block calculated for each small block in the second search mode. There exists a technique for searching for a small block motion vector candidate that minimizes the matching error and determining a final motion vector candidate based on a block matching error corresponding to the motion vector candidate searched in the first search mode (for example, , See Patent Document 2).

JP-A-8-223578 JP 2003-111082 A

  Since the motion vector detection accuracy affects the compression efficiency of the video, a motion vector detection method with higher accuracy is desired.

  In particular, when a plurality of individually moving objects exist in the video, there is a small partial area (sub-block) having another motion included in a large motion detection unit (block) in which the accuracy of the motion vector is ensured. There is a case. In this case, since a small partial region having another motion is caught by a motion of a large motion detection unit, it may be difficult to detect an accurate motion vector.

  In order to cope with this, there is a case where the size of a block for detecting a motion vector is reduced to detect the motion vector in detail. However, when the block size is reduced, it is generally difficult to detect a motion vector that accurately captures the motion of an actual object.

  However, in one aspect of video encoding, it is not always necessary to accurately follow the motion of the video, and a high compression rate can be achieved if a vector similar to the surroundings and a region similar to the encoding target image is found. . For this reason, a decrease in the accuracy of the motion vector when the block size is reduced is not always a problem.

  However, if the accuracy of capturing motion vectors decreases, motion vectors that are different from the motion of the video are often processed. Therefore, noise motion different from the motion of the video is perceived around a moving object in the video. There is a problem of image deterioration due to being performed.

  In one aspect of the embodiment, in order to cope with the above-described problem, an object is to improve the accuracy of motion vector detection and suppress such image quality degradation.

  According to one embodiment, a motion vector search unit that searches for a motion vector indicating from which region of a reference image a block that is a partial region of a processing target image or a sub-block obtained by dividing the block has moved, and a block of interest First error calculation for calculating a first error between each of the divided sub-blocks and a reference image area of each of the plurality of sub-blocks determined by the motion vector of the block of interest And a sub-block specifying unit that specifies a predetermined sub-block based on the absolute values of the plurality of first errors calculated by the first error calculating unit, and an adjoining adjacent block of interest Corresponding to the predetermined sub-block determined by the motion vector of the block and the relative position of the predetermined sub-block in the target block. Based on the second error calculation unit that calculates a second error between the area of the reference image and the predetermined sub-block, the first error, and the second error, the predetermined error There is provided an image processing apparatus including a motion vector determination unit that determines a motion vector of a sub-block.

  According to the embodiment, the accuracy of motion vector detection can be improved and image quality degradation can be suppressed.

It is a figure which shows the example of a detection of a motion vector. It is a block diagram which shows the example of a structure of the image processing apparatus of one Embodiment. It is a figure which shows the example of a motion vector. It is a figure which shows the specific example of the area | region corresponding to the subblock of a process target image, and a reference image. It is a figure which shows several corresponding area | regions in the subblock of a process target image, and a reference image. It is a figure which shows the example by which the area | region of the reference image with the smallest error was searched with respect to the subblock. It is a figure which shows the example of a block division. It is a flowchart which shows the outline | summary of operation | movement of one Embodiment. It is a flowchart which shows the example of the 1st operation | movement of a subblock specific part. It is a flowchart which shows the example of the 2nd operation | movement of a subblock specific part. It is a flowchart which shows the example of the 3rd operation | movement of a subblock specific part. It is a flowchart which shows the example of the 4th operation | movement of a subblock specific part. It is a flowchart which shows the example of the 5th operation | movement of a subblock specific part. It is a flowchart which shows the example of the 1st operation | movement of a motion vector determination part. It is a flowchart which shows the example of the 2nd operation | movement of a motion vector determination part. It is a flowchart which shows the example of the 3rd operation | movement of a motion vector determination part. It is a flowchart which shows the example of the block division | segmentation of one Embodiment. It is a figure which shows the outline | summary of the hardware constitutions of one Embodiment.

  H. is a standard for efficiently processing moving picture information. In the H.264 / AVC (Advanced Video Coding) or HEVC (High Efficiency Video Coding) video compression encoding method, each frame (image) of the video is divided into rectangular areas (hereinafter referred to as blocks) and encoded.

  In the above standards, motion vectors are also used. Note that the motion vector detection method itself is not defined as a standard. For example, a block matching method can be used for the reference image for each block of the processing target image.

  FIG. 1 shows an example of motion vector detection. The moving image information includes a plurality of frames (images). For example, the processing target image 110 represents a moving image at time t = 0. The reference image 120 represents a moving image at time t = 1. Each image is composed of a plurality of blocks. In order to search for a motion vector of the block 112 existing in the processing target image 110, a search range 121 is set in the reference image 120.

  When the block matching method is used, the SAD (Sum of Absolute Difference :) of the block 112 on the processing target image 110 and the image in the region within the search range 121 set around the same spatial position 122 on the reference image: The sum of absolute values of errors) or SSD (Sum of Square Difference) is calculated. The motion vector 124 is determined from the relationship between the spatial position of the region 123 on the reference image having the smallest error value and the block 112 of the processing target image 110.

  AVC / H. In the H.264 and HEVC standards, motion vector detection with 1/2 or 1/4 pixel accuracy is performed in motion vector detection between a processing target image and a reference image.

  FIG. 2 is a block diagram illustrating an example of the configuration of the image processing apparatus 200 according to the embodiment. The image processing apparatus 200 may include a motion vector search unit 210, a first error calculation unit 220, a sub-block specifying unit 230, a second error calculation unit 240, and a motion vector determination unit 260.

  In addition, the motion vector determination unit 260 may include a comparison unit 262 and a motion vector adoption unit 264.

  Then, the processing target image 110 and the reference image 120 may be provided to the motion vector search unit 210, the first error calculation unit 220, and the second error calculation unit 240.

  Hereinafter, the configuration of each unit shown in FIG. 2 will be described.

  The motion vector search unit 210 searches for a motion vector that estimates from which region of the reference image the block that is a partial region of the processing target image or a sub-block obtained by further dividing the block.

  FIG. 3 is a diagram illustrating an example of a motion vector. For convenience of explanation of the motion vector, the spatial positions of the plurality of blocks of the processing target image and the spatial positions of the plurality of regions of the reference image are drawn on the same plane. It should be noted that the processing target image and the reference image are actually frame images at different times.

  In FIG. 3, a block 312 and a block 314 are shown as some blocks of the processing target image. As described above, it is assumed that the error between the reference image area 322 and the block 312 of the processing target image is minimized as a result of searching the block 312 of the processing target image within the reference image search range by the block matching technique. In this case, the motion vector of the block 312 is represented by a motion vector 332.

  Similarly, the block 314 of the processing target image has the smallest error with the reference image region 324 within the search range, and thus the motion vector is obtained as the motion vector 334. Although a rectangular block shape is used in FIG. 3, the shape of the block and the block obtained by re-dividing the block are defined for each image standard. Therefore, the shape of the block is not limited to a rectangle.

  Returning to FIG. The first error calculation unit 220 further divides the target block for which the motion vector is calculated into sub-blocks. When the processing target block is divided, the region of the reference image corresponding to the divided sub-block is uniquely determined. Therefore, the corresponding region of the reference image can be specified corresponding to each of the plurality of sub-blocks of the processing target image.

  A specific example of the corresponding region in the reference image for the sub-block of the processing target image will be described with reference to FIG. Here, the sub-block 410 of the block of interest 314 will be described as an example. A region of the reference image corresponding to the sub block 410 is a region 424. The positions of the sub-block 410 and the region 424 can be specified by a vector 334a. This vector 334a has the same direction and length as the motion vector 334 of block 314.

  Then, an error between the sub-block 410 and the region 424 is calculated. The error may be calculated using, for example, the brightness value of the corresponding pixel using the above-described SAD or SSD. The luminance value is not limited to this. In FIG. 4, only one sub-block 410 is shown, but the entire block 314 may be divided into a plurality of sub-blocks, and an error from the corresponding region of the reference image may be calculated for each sub-block. Good.

  It should be noted that the error calculation result of the pixel unit or the predetermined region unit already calculated when calculating the motion vector of the block to which the sub block belongs can be used as the error in the sub block. Therefore, when the motion vector of the block is obtained, it is desirable to store the calculated pixel unit or predetermined region unit error in a storage unit such as a memory. By doing so, it is possible to simplify the calculation of errors in units of sub-blocks.

  Returning to FIG. The sub-block specifying unit 230 processes errors calculated for a plurality of sub-blocks. One of the purposes of the sub-block specifying unit 230 is to specify a sub-block having a large error absolute value. Note that the processing method is not limited to this, and one of the following processing examples may be executed. The error here is an example of a first error.

  (1) A sub-block corresponding to an error exceeding the first threshold among a plurality of absolute values of errors is specified as a predetermined sub-block. A predetermined constant may be set in advance as the first threshold value. Alternatively, the average value of errors corresponding to a plurality of sub-blocks in a block may be multiplied by a predetermined numerical value and used as the first threshold value.

  (2) When the sum of absolute values of errors exceeding the first threshold among a plurality of absolute values of errors exceeds the second threshold, a sub-block corresponding to the error exceeding the first threshold is set as a predetermined sub-block. It may be specified as a block. The second threshold value may be determined in advance. In this case, if the sum of absolute values of errors exceeding the first threshold does not exceed the second threshold, the total error of the block of interest is considered to be small, so the error exceeding the first threshold It may be determined that noise generated in association with the sub-block corresponding to can be ignored.

  (3) A value obtained by dividing the average value of the absolute values of errors exceeding the first threshold among the absolute values of the plurality of errors by the average value of the absolute values of the plurality of errors in all the sub-blocks in the block is When the threshold value of 3 is exceeded, a sub-block corresponding to an error exceeding the first threshold value may be specified as a predetermined sub-block. The third threshold value may be determined in advance. In this case, when the average value of the absolute values of the first errors divided by the average value of the absolute values of the plurality of errors of all the sub-blocks in the block is larger than the third threshold, It may be determined that an error exceeding the threshold is a value that cannot be ignored, and a sub-block that satisfies this condition may be treated as a predetermined sub-block.

  (4) The average of the absolute values of the first errors that exceed the first threshold among the absolute values of the plurality of errors, and the average of the absolute values of the errors that do not exceed the first threshold among the absolute values of the plurality of errors When the value divided by the value exceeds the fourth threshold value, the sub-block corresponding to the error exceeding the first threshold value may be specified as the predetermined sub-block. The fourth threshold value may be set in advance. This condition is a modification of the condition (3). This is an example in which an average value of absolute values of errors that do not exceed the first threshold value among a plurality of absolute values of errors is used as a denominator when the ratio is calculated.

  (5) When the number of errors exceeding the first threshold among the plurality of absolute values of errors exceeds the fifth threshold, the sub-block corresponding to the error exceeding the first threshold is specified as a predetermined sub-block. May be. The fifth threshold value may be set in advance. For example, if the error of a relatively large number of sub-blocks exceeds the first threshold when the block is divided into sub-blocks, it is determined that noise may be perceived in a relatively large number of blocks. Since the number of sub-blocks when dividing one block into sub-blocks may not be constant, the fifth threshold value is obtained by multiplying the number divided into sub-blocks by a predetermined ratio (for example, 1/3). It may be a value. In this case, if the number of sub-blocks whose error exceeds the first threshold exceeds 1/3 of the total number of sub-blocks, the sub-block corresponding to the error exceeding the first threshold is a predetermined sub-block. It will be specified as a block.

  In FIG. 2, the second error calculation unit 240 corresponds to a predetermined sub-block determined by a predetermined sub-block, a motion vector of an adjacent block adjacent to the target block, and a relative position of the predetermined sub-block in the target block. The error between the reference image area to be calculated is calculated. This error is an example of a second error.

  The second error calculation unit 240 will be described with reference to FIG. In FIG. 4, it is assumed that the sub-block 410 is specified as a predetermined sub-block in the sub-block specifying unit 230. Therefore, the absolute value of the error between the block 410 of the processing target image and the region 424 of the reference image is larger than the first threshold value.

  This means that there is a high possibility that it is not appropriate to use the motion vector 334a (that is, the motion vector of the block 314 of the processing target image as it is) as the sub-block 410.

  Therefore, the second error calculation unit 240 calculates an error with respect to another region of the reference image in order to obtain a more appropriate motion vector for the sub-block 410.

  For this reason, in the processing target image, considering the motion vector information of blocks (or sub-blocks) around the block to which the sub-block belongs, a plurality of highly likely reference image areas are specified, and errors are determined. calculate.

  In FIG. 4, there is an adjacent block 312 adjacent to the block 314 to which the sub-block 410 belongs. The adjacent block 312 has the motion vector 332 as described above. When the motion vector 332 is obtained, the area of the reference image corresponding to the block 312 is the area 322.

  In this case, it is expected that the sub-block 410 may have a motion vector close to the motion vector 332 of the adjacent block 312 adjacent to the sub-block 410 instead of the block 314 to which the sub-block 410 belongs.

  By using the relative position of the sub-block 410 to the block 314 and the motion vector 322 of the adjacent block 312, the corresponding reference image region 432 can be specified.

  Then, the second error calculation unit 240 calculates an error between the sub-block 410 of the processing target image and the specified region 432 of the reference image.

  In general, there are a plurality of blocks adjacent to the block 314.

FIG. 5 shows a plurality of corresponding regions in the reference image for the sub-block of the processing target image. That is, there are the following four candidates for the area in the reference image corresponding to the sub-block 501a of the block 501 of the processing target image.
(1) A region 531b existing in the reference image region 531
(2) Region 532b existing outside reference image region 532
(3) Region 533b existing outside reference image region 533
(4) Region 534b existing outside reference image region 534
Among the above, the area 531 of (1) is an area of the reference image used when the motion vector of the block 501 is obtained. Then, the region 532b, the region 533b, and the region 534b are used when calculating a new motion vector of the sub-block 201a.

  Therefore, the second error calculation unit 240 outputs the error between the sub block 501a and the region 532b, the error between the sub block 501a and the region 533b, and the error between the sub block 501a and the region 534b to the motion vector determination unit 260. May be.

  Returning to FIG. The comparison unit 362 in the motion vector determination unit 260 subtracts the absolute value of the error from the second error calculation unit 240 from the absolute value of the error from the first error calculation unit 220 and compares it with the sixth threshold value. May be. When the subtraction result exceeds the sixth threshold value, the motion vector determination unit 260 in the motion vector determination unit 260 determines whether the motion vector determination unit 260 is adjacent to the second error that outputs the largest subtraction value. The motion vector of the block may be adopted as the motion vector of the identified sub-block and output.

  Conversely, when the subtraction result does not exceed the sixth threshold value, the motion vector adopting unit 264 determines that the motion vector has been slightly improved (or cannot be improved), and adopts a new motion vector. It does not have to be.

  FIG. 6 is a diagram illustrating an example in which the motion vector adoption unit 264 searches for a motion vector. FIG. 6A shows a vector search range 510 in addition to FIG.

  The motion vector adopting unit 264 may search the search range 510 around the region 432 in the reference image to search for the most appropriate motion vector for the sub-block 410. Note that the motion vector adoption unit 264 may set the search range 510 as a fixed range in advance. Alternatively, the search range 510 may be changed according to the time interval between the processing target image and the reference image. For example, the search range 510 may be expanded as the time interval becomes longer. Note that the search range does not need to be an ellipse as shown in FIG. 6, and a search range having an arbitrary shape such as a rectangle may be set. As a search method, a block matching technique may be used.

  FIG. 6B shows an example in which a reference image area 532 having the smallest error with respect to the block 410 is searched in the search range 510. As a result, a motion vector 332b for the sub-block 410 is shown.

  FIG. 7 is a diagram illustrating an example of block division. As illustrated in FIG. 7A1, it is assumed that the sub-block 715 and the sub-block 716 of the processing target image are associated with the region 762 and the region 764 of the reference image, respectively.

  In this case, the motion vector of the block 710 of the processing target image and the sub-block 715 and the sub-block 716 have different motion vectors. If the coding standard allows the block structure as shown in FIG.

  For example, MPEG-4 or AVC / H. H.264 allows square or rectangular blocks and sub-blocks. In this case, as shown in FIG. 7B1, the block 710b of the processing target image can be divided into sub-blocks 711 to 718. Also, the block 726b can be divided into a sub-block 712 and a sub-block 722. In this example, the image processing apparatus 200 also divides 720b, which is an adjacent block of the block 710b, into a subblock 721 and a subblock 722. This is because the encoding standard to which a specific rule regarding block division is applied is followed. Therefore, the block division method may be in accordance with the applied coding standard. For this reason, the block division in FIG. 7B1 is only an example of the block division.

  As shown in FIG. 7 (B2), the reference image areas 750b and 760b are also divided into areas corresponding to the processing target image and referred to.

<Operation>
FIG. 8 shows an outline of the operation of the embodiment. The following will be described in order.

  In step S805, the motion vector search unit 210 searches for the motion vector of the block to obtain the motion vector of the block.

  In step S810, the first error calculation unit 220 divides the block into sub-blocks, and calculates a first error between each of the sub-blocks and the corresponding region of the reference image. This first error may be stored in a storage unit (for example, the main storage unit 1802 in FIG. 18).

  In step S820, the sub-block specifying unit 230 specifies a predetermined sub-block based on the absolute value of the first error. For example, the first error may be compared with a predetermined threshold, and the sub-block corresponding to the error exceeding the predetermined threshold may be specified as the predetermined sub-block. Details of this point will be described later.

  In step S830, the second error calculation unit 240 calculates a second error between a predetermined sub-block and a reference image area corresponding to the predetermined sub-block in an adjacent block of the processing target block.

  In step S840, the motion vector determination unit calculates a motion vector of a predetermined sub-block based on the absolute value of the first error and the absolute value of the second error. For example, when the value obtained by subtracting the absolute value of the second error from the absolute value of the first error exceeds a predetermined threshold, the motion vector of the adjacent block is used as the motion vector of the predetermined sub block. May be. Details of this point will be described later.

  FIG. 9 shows an example of a first operation of the sub-block specifying unit 230.

  In step S910, the sub-block specifying unit 230 extracts one of the first errors of the sub-block from the storage unit (for example, the main storage unit 1802 in FIG. 18).

  In step S920, the sub-block specifying unit 230 determines whether or not the absolute value of the extracted first error exceeds the first threshold value. If the determination result is “Yes”, the process proceeds to step S930. If the determination result is “No”, the process proceeds to step S940.

  In step S930, the sub-block specifying unit 230 may adopt a sub-block corresponding to the extracted error as a predetermined sub-block.

  In step S940, the sub-block specifying unit 230 determines whether or not processing has been completed for all sub-blocks included in the block. If the determination result is “No”, the process returns to step S910. If the determination result is “Yes”, the process ends.

  FIG. 10 shows an example of the second operation of the sub-block specifying unit 230.

  Steps S1010 and S1020 are the same as steps S910 and S920 of FIG. 9, respectively.

  In step S1030, when the absolute value of the first error exceeds the first threshold, the sub-block specifying unit 230 adds the absolute value of the error to Sum.

  Step S1040 is the same as step S940 in FIG.

  In step S1050, the sub-block specifying unit 230 determines whether the value of Sum exceeds the second threshold value. If the determination is “yes”, the process moves to step S1050. If the determination is “No”, the process ends.

  In step S1060, the sub-block specifying unit 230 specifies a sub-block whose absolute value of the first error exceeds the first threshold as a predetermined sub-block.

  FIG. 11 shows an example of the third operation of the sub-block specifying unit 230.

  Step S1110, Step S1120, Step S1130, and Step S1140 are the same as Step S1010, Step S1020, and Step S1030 in FIG. 10, respectively.

  In step S1150, the sub-block specifying unit 230 subtracts the average of the absolute values of errors of all the sub-blocks of the block from the average of the sum of errors in which the absolute value of the first error exceeds the first threshold. Is determined to exceed the third threshold. If the determination result is “Yes”, the process moves to step S1160. If the determination result is “No”, the process ends.

  In step S1160, the sub-block specifying unit 230 specifies a sub-block whose absolute value of the first error exceeds the first threshold as a predetermined sub-block.

  FIG. 12 shows an example of the fourth operation of the sub-block specifying unit 230.

  Step S1210, step S1220, step S1230, and step S1240 are the same as step S1010, step S1020, step S1030, and step S1040 of FIG. 10, respectively.

  In step S1250, the sub-block specifying unit 230 calculates the remaining sub-block of the block (the absolute value of the error of the sub-block is the first sum) from the average sum of errors in which the absolute value of the first error exceeds the first threshold. It is determined whether the value obtained by subtracting the average of the absolute values of the errors of the sub-block that does not exceed the threshold value of 1 exceeds the fourth threshold value. If the determination result is “Yes”, the process moves to step S1260. If the determination result is “No”, the process ends.

  In step S1260, subblock specifying section 230 specifies a subblock whose absolute value of the first error exceeds the first threshold as a predetermined subblock.

FIG. 13 illustrates a fifth operation example of the sub-block specifying unit 230.
Steps S1310 and S1320 are the same as steps S1010 and S1020 in FIG. 10, respectively.

  In step S1330, when the absolute value of the first error exceeds the first threshold, the sub-block specifying unit 230 increments n to store the number of times in n.

  Step S1340 is the same as step S1040 in FIG.

  In step S1350, the sub-block specifying unit 230 determines whether the number n of errors in which the absolute value of the first error exceeds the first threshold exceeds the fifth threshold. If this determination is “Yes”, the flow proceeds to step S 1360. If the determination result is “No”, the process ends.

  In step S1360, sub-block specifying unit 230 specifies a sub-block whose absolute value of the first error exceeds the first threshold as a predetermined sub-block.

  FIG. 14 illustrates an example of a first operation of the motion vector determination unit 260.

  In step S1410, the comparison unit 262 of the motion vector determination unit 260 may determine whether or not a value obtained by subtracting the absolute value of the second error from the absolute value of the first error exceeds a sixth threshold value. . If the determination result is “yes”, the process proceeds to step S1420. If the determination result is “no”, the process proceeds to step S1430.

  In step S1420, the motion vector employing unit 264 may employ the motion vector of the adjacent block as the motion vector of the predetermined sub-block. As shown in FIG. 5, when there are a plurality of adjacent blocks and there are a plurality of adjacent block motion vectors, the motion vector of the adjacent block having the largest subtracted value may be used.

  In step S1430, the motion vector adopting unit 264 may adopt the motion vector of the target block to which the predetermined sub-block belongs as the motion vector of the predetermined sub-block (that is, the vector may not be changed).

  FIG. 15 is a flowchart illustrating an example of a second operation of the motion vector determination unit 260.

  Step S1510 is the same as step S1410 of FIG.

  In step S1520, the motion vector adopting unit 264, as a motion vector of a predetermined subblock, refers to a motion vector of an adjacent block that is adjacent and a reference corresponding to the predetermined subblock determined by the relative position of the predetermined subblock in the target block. A motion vector of a predetermined sub-block may be searched for a predetermined peripheral region of the image area, and the searched motion vector may be adopted as the motion vector of the predetermined sub-block. In addition, you may utilize the existing block matching technique for a search.

  In step S1530, the motion vector adopting unit 264 may adopt the motion vector of the target block to which the predetermined sub-block belongs as the motion vector of the predetermined sub-block (that is, the vector may not be changed).

  FIG. 16 is a flowchart illustrating a third operation example of the motion vector determination unit 260.

  Step S1610 is the same as step S1410 of FIG.

  When the comparison unit of the motion vector determination unit 260 determines in step S1620 that the value obtained by subtracting the absolute value of the second error from the absolute value of the first error exceeds the sixth threshold value, the number of times is counted. Therefore, the value of m is incremented.

  In step S1630, it is determined whether all sub-blocks have been processed. If “no”, the process returns to step S1610. If “yes”, the process moves to step S1640.

  In step S1640, it is determined whether the value of m exceeds a seventh threshold value. If it is determined that the value does not exceed (“Yes”), the process proceeds to step S1650. When it is determined that the number exceeds (“No”), the process of FIG. 14 or 15 may be performed (more specifically, the process of step S1420 or step S1520 may be performed).

  In step S1650, the motion vector of the target block is adopted as the motion vector of the predetermined sub-block. When there are a plurality of predetermined sub-blocks, the motion vector of the target block is adopted as the motion vector of all the predetermined sub-blocks.

  FIG. 17 is a flowchart illustrating an example of block division according to an embodiment.

  In step S1710, it is determined whether the motion vector of the target block has been adopted as the motion vector of all the predetermined sub-blocks. That is, if this determination is “Yes”, the motion vectors for all the sub-blocks are unified with the motion vector of the target block to which the sub-block belongs. If this determination is “Yes”, the process moves to step S1720. If this determination is “NO”, the process moves to step S1730.

  In step S1720, it may be determined not to divide the block of interest into sub-blocks. The reason for this is that, as described above, the motion vectors of all the sub-blocks included in the target block are the same as those of the target block. Therefore, the processing divided into sub-blocks causes an increase in the data amount. Because there are cases.

  In step S1730, processing for dividing the block of interest is performed. For this processing, for example, the block division technique described with reference to FIG. 7 can be used. In addition, what is necessary is just to process about the division | segmentation of a block so that it may conform to each video coding standard.

  The block division information can be encoded and included in the bitstream together with the encoding information of the video information. The decoder that has received the bitstream performs decoding using this block division information.

  FIG. 18 is a block diagram illustrating an example of a schematic configuration of the image processing apparatus 1800. An image processing apparatus 1800 illustrated in FIG. 18 is an example of an apparatus in which the image processing described in the above-described embodiment is implemented by software.

  In the above embodiment, the block and the sub-block obtained by dividing the block have been described as an example. Needless to say, the above-described embodiment can be applied to a case where the sub-block is further divided.

  As shown in FIG. 18, the image processing apparatus 1800 includes a control unit 1801, a main storage unit 1802, an auxiliary storage unit 1803, a drive device 1804, a network I / F unit 1806, an input unit 1807, and a display unit. 1808. These components are connected to each other via a bus so as to be able to transmit and receive data. Each unit is connected by a bus 1809.

  The control unit 1801 is a CPU (Central Processing Unit) that controls each device, calculates data, and processes in the computer. The control unit 1801 is an arithmetic device that executes an image processing program stored in the main storage unit 1802 or the auxiliary storage unit 1803. The control unit 1801 receives data from the input unit 1807 and the storage device, calculates and processes the data, and then outputs the data to the display unit 1808 or the storage device.

  Further, the control unit 1801 can realize the processing described in the embodiment by executing an image processing program.

  The main storage unit 1802 is a ROM (Read Only Memory) or a RAM (Random Access Memory). The main storage unit 1802 is a storage device that stores or temporarily stores programs and data such as OS (Operating System) or application software which is basic software executed by the control unit 1801.

  The auxiliary storage unit 1803 is an HDD (Hard Disk Drive) or the like, and is a storage device that stores data related to application software or the like.

  The drive device 1804 reads the program from the recording medium 1805, for example, a flexible disk, and installs it in the storage unit.

  A predetermined program is stored in the recording medium 1805, and the program stored in the recording medium 1805 is installed in the image processing apparatus 1800 via the drive device 1804. The installed predetermined program can be executed by the image processing apparatus 1800.

  A network I / F unit 1806 has a communication function connected via a network such as a LAN (Local Area Network) or a WAN (Wide Area Network) constructed by a data transmission path such as a wired and / or wireless line. This is an interface between the device and the image processing apparatus 1800.

  The input unit 1807 includes a keyboard having cursor keys, numeric input, various function keys, and the like, and a mouse and a slide pad for performing key selection on the display screen of the display unit 1808. The display unit 1808 is configured by an LCD (Liquid Crystal Display) or the like, and performs display according to display data input from the control unit 1801.

  2 can be realized by, for example, the control unit 1801 and the main storage unit 1802 as a work memory. The main storage unit 1802 may store the above-described error information and the like.

  A program executed by the image processing apparatus 1800 has a module configuration including each unit described in the embodiment. As actual hardware, when the control unit 1801 reads out and executes a program from the auxiliary storage unit 1803, one or more of the above-described units are loaded onto the main storage unit 1802, and one or more of the units are main. It is generated on the storage unit 1802.

  As described above, the image processing described in the above-described embodiment may be realized as a program for causing a computer to execute. The processing described in the embodiment can be realized by installing this program from a server or the like and causing the computer to execute the program.

  It is also possible to record the program in a recording medium 1805 and cause the processing medium such as a computer or a portable terminal to read the recording medium 1805 on which the program is recorded, thereby realizing the above-described image processing.

  Note that the recording medium 1805 is a recording medium that records information optically, electrically, or magnetically, such as a CD-ROM, flexible disk, magneto-optical disk, etc., and information is electrically stored, such as a ROM or flash memory. Various types of recording media such as a semiconductor memory for recording can be used.

  Further, the image processing described in each of the above embodiments may be implemented in one or a plurality of integrated circuits. Note that the image processing apparatus 1800 may have a function as at least a part of the image processing apparatus 200 as described above.

  Further, the image processing apparatus in the above-described embodiment can be applied to an encoding technique. It is not limited to H.264 / AVC or HEVC.

  Each embodiment has been described in detail above. However, the present invention is not limited to the specific embodiment, and various modifications and changes other than the above-described modification are possible within the scope described in the claims. .

The following notes are disclosed for the above embodiment.
(Appendix 1)
A motion vector search unit for searching for a motion vector indicating from which region of the reference image a block that is a partial region of the processing target image or a sub-block obtained by dividing the block has moved;
A first error for calculating a first error between each of a plurality of sub-blocks obtained by dividing the block of interest and a reference image area of each of the plurality of sub-blocks determined by a motion vector of the block of interest. An error calculation unit of
A sub-block specifying unit that specifies a predetermined sub-block based on the magnitudes of absolute values of a plurality of first errors calculated by the first error calculating unit;
A region of a reference image corresponding to the predetermined subblock determined by a motion vector of an adjacent block adjacent to the target block and a relative position of the predetermined subblock in the target block; and the predetermined subblock A second error calculation unit for calculating a second error between;
A motion vector determination unit that determines a motion vector of the predetermined sub-block based on the first error and the second error;
An image processing apparatus.
(Appendix 2)
The image according to claim 1, wherein the sub-block specifying unit specifies, as the predetermined sub-block, the sub-block corresponding to a first error that exceeds a first threshold value among the plurality of first error absolute values. Processing equipment.
(Appendix 3)
The sub-block specifying unit, when the sum of absolute values of first errors that exceed a first threshold among the absolute values of the plurality of first errors exceeds a second threshold, determines the first threshold. The image processing apparatus according to appendix 1, wherein the sub-block corresponding to the first error that exceeds is specified as the predetermined sub-block.
(Appendix 4)
The sub-block specifying unit calculates an average value of absolute values of first errors that exceed a first threshold among absolute values of the plurality of first errors, and calculates an average value of absolute values of the plurality of first errors. The image processing apparatus according to appendix 1, wherein when the value divided by 1 exceeds a third threshold, the sub-block corresponding to the first error exceeding the first threshold is specified as the predetermined sub-block.
(Appendix 5)
The sub-block specifying unit calculates an average value of absolute values of first errors exceeding a first threshold among the absolute values of the plurality of first errors, and calculates the average value of the absolute values of the plurality of first errors. When the value obtained by dividing the average value of the absolute values of the first errors not exceeding the first threshold exceeds the fourth threshold, the sub-block corresponding to the first error exceeding the first threshold is The image processing apparatus according to appendix 1, which is specified as a predetermined sub-block.
(Appendix 6)
The sub-block specifying unit, when the number of first errors that exceed a first threshold among the absolute values of the plurality of first errors exceeds a fifth threshold, The image processing apparatus according to appendix 1, wherein the sub-block corresponding to the error is identified as the predetermined sub-block.
(Appendix 7)
The motion vector determination unit
A comparison unit that compares a value obtained by subtracting the absolute value of the second error from the absolute value of the first error with a sixth threshold;
When the subtracted value exceeds the sixth threshold, the motion vector of the adjacent block is adopted as the motion vector of the predetermined sub-block, and when the subtracted value does not exceed the sixth threshold, A motion vector adopting unit that adopts a motion vector of the block of interest as a motion vector of a predetermined sub-block;
including,
The image processing apparatus according to any one of appendices 1 to 6.
(Appendix 8)
When the subtracted value exceeds the sixth threshold, the motion vector adopting unit, as the motion vector of the predetermined sub-block, as the motion vector of the adjacent block, and the attention of the predetermined sub-block The motion vector search unit is instructed to search for a motion vector of the predetermined sub-block for a predetermined peripheral region of a reference image region corresponding to the predetermined sub-block, which is determined by a relative position in the block, and searches The image processing apparatus according to appendix 7, wherein the motion vector thus adopted is adopted as a motion vector of the predetermined sub-block.
(Appendix 9)
The motion vector adopting unit, with respect to the block of interest, if the number when the subtracted value exceeds the sixth threshold does not exceed the seventh threshold, the subtracted value exceeds the sixth threshold Even if it is a case, The image processing apparatus of Additional remark 7 or 8 which employ | adopts the motion vector of the said block of interest as a motion vector of the said predetermined | prescribed subblock.
(Appendix 10)
The motion vector adopter does not divide the target block into sub-blocks when the motion vector adopter determines to adopt the motion vector of the target block as all the motion vectors of the predetermined sub-block. The image processing apparatus according to any one of appendices 7 to 9.
(Appendix 11)
Computer
Searching for a motion vector indicating from which region of the reference image a block that is a partial region of the processing target image or a sub-block obtained by dividing the block has moved;
Calculating a respective first error between each of the plurality of sub-blocks obtained by dividing the block of interest and a reference image area of each of the plurality of sub-blocks determined by a motion vector of the block of interest; ,
Determining a predetermined sub-block based on the magnitudes of absolute values of the plurality of first errors calculated by the calculating the first error;
A region of a reference image corresponding to the predetermined subblock determined by a motion vector of an adjacent block adjacent to the target block and a relative position of the predetermined subblock in the target block; and the predetermined subblock Calculating a second error between;
Determining a motion vector of the predetermined sub-block based on the first error and the second error;
An image processing method.
(Appendix 12)
The specifying of the sub-block includes specifying the sub-block corresponding to a first error exceeding a first threshold among the plurality of first error absolute values as the predetermined sub-block. Image processing method.
(Appendix 13)
The step of identifying the sub-block includes the step of identifying the first block when the sum of the absolute values of the first errors exceeding the first threshold among the absolute values of the plurality of first errors exceeds the second threshold. The image processing method according to appendix 11, wherein the sub-block corresponding to the first error exceeding a threshold is specified as the predetermined sub-block.
(Appendix 14)
The step of identifying the sub-block includes calculating an average value of absolute values of the first errors exceeding a first threshold among the absolute values of the plurality of first errors, and calculating an absolute value of the plurality of first errors. The image processing method according to claim 11, wherein when the value divided by the average value exceeds a third threshold, the sub-block corresponding to the first error exceeding the first threshold is specified as the predetermined sub-block. .
(Appendix 15)
The step of identifying the sub-block includes calculating an average value of absolute values of the first errors exceeding a first threshold among the absolute values of the plurality of first errors, and calculating an absolute value of the plurality of first errors. The sub-block corresponding to the first error exceeding the first threshold when the value obtained by dividing the average value of the absolute values of the first error not exceeding the first threshold exceeds the fourth threshold 12. The image processing method according to appendix 11, wherein: is specified as the predetermined sub-block.
(Appendix 16)
The step of identifying the sub-block exceeds the first threshold when the number of first errors that exceed a first threshold among the absolute values of the plurality of first errors exceeds a fifth threshold. The image processing method according to claim 11, wherein the sub-block corresponding to the first error is specified as the predetermined sub-block.
(Appendix 17)
Determining the motion vector comprises:
Comparing a value obtained by subtracting the absolute value of the second error from the absolute value of the first error with a sixth threshold;
When the subtracted value exceeds the sixth threshold, the motion vector of the adjacent block is adopted as the motion vector of the predetermined sub-block, and when the subtracted value does not exceed the sixth threshold, Adopting a motion vector of the block of interest as a motion vector of a predetermined sub-block;
including,
The image processing method according to any one of appendices 11 to 16.
(Appendix 18)
In the step of adopting the motion vector, when the subtracted value exceeds the sixth threshold, the motion vector of the adjacent block and the motion vector of the adjacent sub-block as the motion vector of the predetermined sub-block. A motion vector of the predetermined sub-block is searched for a predetermined peripheral region of a reference image region corresponding to the predetermined sub-block, which is determined by a relative position in the target block, and the searched motion vector is searched for the predetermined motion vector 18. The image processing method according to appendix 17, which is employed as a sub-block motion vector.
(Appendix 19)
In the step of adopting the motion vector, regarding the target block, when the number when the subtracted value exceeds the sixth threshold does not exceed the seventh threshold, the subtracted value is the sixth threshold. 19. The image processing method according to appendix 17 or 18, wherein the motion vector of the block of interest is adopted as the motion vector of the predetermined sub-block even when the value exceeds.
(Appendix 20)
When the step of adopting the motion vector determines to adopt the motion vector of the block of interest as all the motion vectors of the predetermined sub-block, the step of employing the motion vector subtracts the block of interest 20. The image processing method according to any one of supplementary notes 17 to 19, wherein the image processing method is not divided into blocks.
(Appendix 21)
Search for a motion vector indicating from which region of the reference image a block that is a partial region of the processing target image or a sub-block obtained by dividing the block has moved,
Calculating a plurality of first errors between each of a plurality of sub-blocks obtained by dividing the block of interest and a reference image area of each of the plurality of sub-blocks determined by a motion vector of the block of interest;
Identifying a predetermined sub-block based on the magnitudes of absolute values of a plurality of first errors calculated by the process of calculating the first error;
A region of a reference image corresponding to the predetermined subblock determined by a motion vector of an adjacent block adjacent to the target block and a relative position of the predetermined subblock in the target block; and the predetermined subblock Calculate the second error between
Determining a motion vector of the predetermined sub-block based on the first error and the second error;
An image processing program that causes a computer to execute processing.
(Appendix 22)
The processing for identifying the sub-block identifies the sub-block corresponding to the first error that exceeds a first threshold among the plurality of first error absolute values as the predetermined sub-block. Image processing program.
(Appendix 23)
The process of specifying the sub-block is performed when the sum of the absolute values of the first errors that exceed the first threshold among the absolute values of the plurality of first errors exceeds the second threshold. The image processing program according to appendix 21, wherein the sub-block corresponding to the first error exceeding the threshold is specified as the predetermined sub-block.
(Appendix 24)
The process of specifying the sub-block includes calculating an average value of absolute values of the first errors exceeding a first threshold among the absolute values of the plurality of first errors, and calculating an absolute value of the plurality of first errors. The image processing program according to appendix 21, wherein when the value divided by the average value exceeds a third threshold value, the sub-block corresponding to the first error exceeding the first threshold value is specified as the predetermined sub-block. .
(Appendix 25)
The process of specifying the sub-block includes calculating an average value of absolute values of the first errors exceeding a first threshold among the absolute values of the plurality of first errors, and calculating an absolute value of the plurality of first errors. The sub-block corresponding to the first error exceeding the first threshold when the value obtained by dividing the average value of the absolute values of the first error not exceeding the first threshold exceeds the fourth threshold The image processing program according to appendix 21, wherein the image processing program is specified as the predetermined sub-block.
(Appendix 26)
The process of specifying the sub-block exceeds the first threshold when the number of first errors that exceed a first threshold among the absolute values of the plurality of first errors exceeds a fifth threshold. The image processing program according to appendix 21, wherein the sub-block corresponding to the first error is specified as the predetermined sub-block.
(Appendix 27)
The process of determining the motion vector includes:
A value obtained by subtracting the absolute value of the second error from the absolute value of the first error is compared with a sixth threshold value;
When the subtracted value exceeds the sixth threshold, the motion vector of the adjacent block is adopted as the motion vector of the predetermined sub-block, and when the subtracted value does not exceed the sixth threshold, Adopting the motion vector of the block of interest as the motion vector of the predetermined sub-block,
Including processing,
27. The image processing program according to any one of appendices 21 to 26.
(Appendix 28)
When the subtracted value exceeds the sixth threshold, the process of adopting the motion vector includes the motion vector of the adjacent block and the motion vector of the adjacent sub-block as the motion vector of the predetermined sub-block. A motion vector of the predetermined sub-block is searched for a predetermined peripheral region of a reference image region corresponding to the predetermined sub-block, which is determined by a relative position in the target block, and the searched motion vector is searched for the predetermined motion vector 28. The image processing program according to appendix 27, which is employed as a sub-block motion vector.
(Appendix 29)
In the process of adopting the motion vector, the number of cases where the subtracted value exceeds the sixth threshold for the block of interest does not exceed the seventh threshold, the subtracted value is the sixth threshold. 29. The image processing program according to appendix 27 or 28, wherein the motion vector of the block of interest is adopted as the motion vector of the predetermined sub-block even if the value exceeds.
(Appendix 30)
When the process of adopting the motion vector has decided to adopt the motion vector of the target block as all the motion vectors of the predetermined sub-block, the process of adopting the motion vector subtracts the target block. 30. The image processing program according to any one of appendices 27 to 29, which is not divided into blocks.

200 Image processing device 210 Motion vector search unit 220 First error calculation unit 230 Sub-block specifying unit 240 Second error calculation unit 260 Vector determination unit 262 Comparison unit 264 Vector adoption unit 1800 Image processing device

Claims (12)

A motion vector search unit for searching for a motion vector indicating from which region of the reference image a block that is a partial region of the processing target image or a sub-block obtained by dividing the block has moved;
A first error for calculating a first error between each of a plurality of sub-blocks obtained by dividing the block of interest and a reference image area of each of the plurality of sub-blocks determined by a motion vector of the block of interest. An error calculation unit of
A sub-block specifying unit that specifies a predetermined sub-block based on the magnitudes of absolute values of a plurality of first errors calculated by the first error calculating unit;
A region of a reference image corresponding to the predetermined subblock determined by a motion vector of an adjacent block adjacent to the target block and a relative position of the predetermined subblock in the target block; and the predetermined subblock A second error calculation unit for calculating a second error between;
A motion vector determination unit that determines a motion vector of the predetermined sub-block based on the first error and the second error;
An image processing apparatus.   The subblock specifying unit specifies the subblock corresponding to a first error exceeding a first threshold among the plurality of first error absolute values as the predetermined subblock. Image processing device.   The sub-block specifying unit, when the sum of absolute values of first errors that exceed a first threshold among the absolute values of the plurality of first errors exceeds a second threshold, determines the first threshold. The image processing apparatus according to claim 1, wherein the sub-block corresponding to the first error that exceeds is specified as the predetermined sub-block.   The sub-block specifying unit calculates an average value of absolute values of first errors that exceed a first threshold among absolute values of the plurality of first errors, and calculates an average value of absolute values of the plurality of first errors. 2. The image processing apparatus according to claim 1, wherein when the value divided by 1 exceeds a third threshold, the sub-block corresponding to the first error exceeding the first threshold is specified as the predetermined sub-block.   The sub-block specifying unit calculates an average value of absolute values of first errors exceeding a first threshold among the absolute values of the plurality of first errors, and calculates the average value of the absolute values of the plurality of first errors. When the value obtained by dividing the average value of the absolute values of the first errors not exceeding the first threshold exceeds the fourth threshold, the sub-block corresponding to the first error exceeding the first threshold is The image processing apparatus according to claim 1, wherein the image processing apparatus is specified as a predetermined sub-block.   The sub-block specifying unit, when the number of first errors that exceed a first threshold among the absolute values of the plurality of first errors exceeds a fifth threshold, The image processing apparatus according to claim 1, wherein the sub-block corresponding to the error is identified as the predetermined sub-block. The motion vector determination unit
A comparison unit that compares a value obtained by subtracting the absolute value of the second error from the absolute value of the first error with a sixth threshold;
When the subtracted value exceeds the sixth threshold, the motion vector of the adjacent block is adopted as the motion vector of the predetermined sub-block, and when the subtracted value does not exceed the sixth threshold, A motion vector adopting unit that adopts a motion vector of the block of interest as a motion vector of a predetermined sub-block;
including,
The image processing apparatus according to claim 1.   When the subtracted value exceeds the sixth threshold, the motion vector adopting unit, as the motion vector of the predetermined sub-block, as the motion vector of the adjacent block, and the attention of the predetermined sub-block The motion vector search unit is instructed to search for a motion vector of the predetermined sub-block for a predetermined peripheral region of a reference image region corresponding to the predetermined sub-block, which is determined by a relative position in the block, and searches The image processing apparatus according to claim 7, wherein the determined motion vector is adopted as a motion vector of the predetermined sub-block.   The motion vector adopting unit, with respect to the block of interest, if the number when the subtracted value exceeds the sixth threshold does not exceed the seventh threshold, the subtracted value exceeds the sixth threshold The image processing apparatus according to claim 7 or 8, wherein the motion vector of the block of interest is adopted as the motion vector of the predetermined sub-block even if it is a case.   When the motion vector adopting unit decides to adopt the motion vector of the target block as all motion vectors of the predetermined sub-block, the motion vector adopting unit does not divide the target block into sub blocks. An image processing apparatus according to any one of claims 7 to 9. Computer
Searching for a motion vector indicating from which region of the reference image a block that is a partial region of the processing target image or a sub-block obtained by dividing the block has moved;
Calculating a respective first error between each of the plurality of sub-blocks obtained by dividing the block of interest and a reference image area of each of the plurality of sub-blocks determined by a motion vector of the block of interest; ,
Determining a predetermined sub-block based on the magnitudes of absolute values of the plurality of first errors calculated by the calculating the first error;
A region of a reference image corresponding to the predetermined subblock determined by a motion vector of an adjacent block adjacent to the target block and a relative position of the predetermined subblock in the target block; and the predetermined subblock Calculating a second error between;
Determining a motion vector of the predetermined sub-block based on the first error and the second error;
An image processing method. Search for a motion vector indicating from which area of the reference image a block that is a partial area of the processing target image or a sub-block obtained by dividing the block has moved;
Calculating a plurality of first errors between each of a plurality of sub-blocks obtained by dividing the block of interest and a reference image area of each of the plurality of sub-blocks determined by a motion vector of the block of interest;
Identifying a predetermined sub-block based on the magnitudes of absolute values of a plurality of first errors calculated by the process of calculating the first error;
A region of a reference image corresponding to the predetermined subblock determined by a motion vector of an adjacent block adjacent to the target block and a relative position of the predetermined subblock in the target block; and the predetermined subblock Calculate the second error between
Determining a motion vector of the predetermined sub-block based on the first error and the second error;
An image processing program that causes a computer to execute processing.

Images