Disclosure of Invention
In order to solve the technical problems, the invention provides a method and a device for processing image blocks in immersive media coding.
For at least one image block in the multi-view image, for at least one dividing line which penetrates through and is perpendicular to the boundary of the image block in the image block, calculating the effective information density of the image sub-blocks at two sides of the dividing line respectively, wherein the effective information is the additional information of the image sub-blocks relative to other view points in the multi-view.
When the image sub-blocks at two sides of the dividing line meet the cutting condition, the cutting condition is that the effective information density of the image sub-block at one side is larger than a threshold value and the effective information density of the image sub-block at the other side is smaller than the threshold value, cutting is carried out along the dividing line, the image block is cut into two image blocks, if the image block containing the image sub-block with the effective information density smaller than the threshold value does not have the dividing line meeting the cutting condition, the image block is discarded and is not assembled into the splicing diagram, otherwise, the image block is not cut.
Further, for the image block which contains the image sub-block with the effective information density smaller than the threshold value after cutting, if a dividing line meeting the cutting condition exists, cutting the image block again.
Further, for the image block with a plurality of dividing lines for which the image sub-blocks at two sides meet the cutting condition, a dividing line with the largest difference of effective information density of the image sub-blocks at two sides of the dividing line is selected as an actual dividing line.
Further, when the image sub-blocks at two sides of the dividing line meet the cutting condition, updating a threshold according to the pixel rate limit of the spliced image and the effective information density of the image blocks which are not discarded.
Further, the invention provides a device for processing image blocks, which comprises the following modules:
The effective information density calculating module is used for respectively calculating the effective information densities of the image sub-blocks at two sides of the dividing line along the dividing line which penetrates through and is perpendicular to the boundary of the image block in the multi-view image block, wherein the effective information is the extra information of the image sub-blocks relative to other views in the multi-view
The threshold judging module is used for judging whether one of the effective information densities of the image sub-blocks at the two sides of the dividing line is larger than a threshold value, one is smaller than the threshold value, cutting is performed when the condition is met, and cutting is not performed when the condition is not met
The dividing line decision module is used for selecting the dividing line with the largest difference of the effective information density of the image sub-blocks at the two sides of the dividing line as the actual dividing line for the image block when the image sub-blocks at the two sides meet the dividing line of the cutting condition
Cutting along the dividing line under the condition of meeting the cutting condition
The assembly and splicing decision module is used for discarding image blocks which contain image sub-blocks with effective information density smaller than a threshold value and do not have a dividing line meeting the cutting condition, and does not assemble the image blocks into a splicing diagram
And the threshold updating module is used for updating the threshold according to the pixel rate limit of the spliced image and the effective information density of the image blocks which are not discarded when the image sub-blocks at the two sides of the dividing line meet the cutting condition.
Further, the image block cutting device cuts the image block containing the image sub-block with the effective information density smaller than the threshold value again if the dividing line meeting the cutting condition exists.
The method has the advantages that full cutting of the image blocks can be achieved through judgment of the granularity of the image sub-blocks, the image blocks can be discarded more accurately, the image blocks with higher effective information density can be organized under the limit of the limited pixel rate in the multi-view image splicing process, the information utilization rate of the spliced image is improved, the rendering quality is improved at the decoding end, meanwhile, invalid cutting is reduced under the condition that discarding is not needed through threshold value control cutting, the coding code rate is saved, the threshold value is updated according to the limit of the pixel rate and the effective information density of the image sub-blocks which are not discarded after each cutting, the threshold value is more accurate, and the overall coding efficiency is improved.
Detailed Description
Example 1
As shown in fig. 1, A, B, C is three image sub-blocks of image block ABC, the colors of the image sub-blocks representing the effective density of the corresponding image sub-blocks from light to deep as low as high.
In this example, the initial threshold lies between the effective information densities of image sub-block a and image sub-block B.
The common boundary of the image sub-block A and the image sub-block B is a dividing line which penetrates through and is perpendicular to the boundary of the image block, the effective information density of the image sub-block A is smaller than a threshold value, and the effective information density of the image sub-block B is larger than the threshold value, so that the cutting condition is met;
Therefore, the image block ABC is cut into the image block A and the image block BC by taking the common boundary of the image sub-block A and the image sub-block B as a dividing line, and the effective information density of the image block A is smaller than a threshold value, and no dividing line meeting the cutting condition exists, so that the image block ABC is discarded and not assembled into the spliced graph.
Removing image block ABC, adding image block BC, and recalculating a threshold value, wherein the updated threshold value is the minimum value of the effective information density of the image blocks and the image sub-blocks which are not discarded in the pixel rate limit of the mosaic image.
In this example, the updated threshold is less than the effective information density of image sub-block B.
For the image block BC generated after cutting, repeating the process, wherein the common boundary of the image sub-block B and the image sub-block C is a dividing line which penetrates through and is perpendicular to the boundary of the image block, but the effective information density of the image sub-block B and the effective information density of the image sub-block C are both larger than a threshold value, and the cutting condition is not met;
the cut is terminated, as shown in fig. 4, with discarded tiles within the dashed box, and the final tile ABC is cut into tile a and tile BC, where tile a is discarded and not entered into the mosaic.
Example 2
As shown in fig. 1, A, B, C is three image sub-blocks of image block ABC, the colors of the image sub-blocks representing the effective density of the corresponding image sub-blocks from light to deep as low as high.
In this example, the initial threshold lies between the effective information densities of image sub-block B and image sub-block C.
The common boundary of the image sub-block A and the image sub-block B is a dividing line which penetrates through and is perpendicular to the boundary of the image block, but the effective information density of the image sub-block A and the effective information density of the image sub-block B are smaller than a threshold value and do not meet the cutting condition;
The image block ABC is cut into an image block AB and an image block C by taking the common boundary of the image sub-block B and the image sub-block C as a dividing line, and the image block AB comprises an image sub-block A with the effective information density smaller than a threshold value, but the effective information density of the image sub-block A, B is smaller than the threshold value, and the dividing line meeting the cutting condition does not exist, so that the image block AB is discarded and not assembled into the spliced graph.
The cut is terminated, as shown in fig. 5, with discarded tiles within the dashed box, and the final tile ABC is cut into tiles AB and C, where tile AB is discarded and not entered into the mosaic.
Removing image block ABC, adding image block C, and recalculating a threshold value, wherein the updated threshold value is the minimum value of effective information density of the image blocks and the image sub-blocks which are not discarded in the pixel rate limit of the mosaic image.
Example 3
As shown in fig. 1, A, B, C is three image sub-blocks of image block ABC, the colors of the image sub-blocks representing the effective density of the corresponding image sub-blocks from light to deep as low as high.
In this example, the initial threshold is less than the effective information density of image sub-block A
The common boundary of the image sub-block A and the image sub-block B is a dividing line which penetrates through and is perpendicular to the boundary of the image block, but the effective information density of the image sub-block A and the effective information density of the image sub-block B are both larger than a threshold value and do not meet the cutting condition;
and therefore no cutting is performed. The cut is terminated and the threshold is not updated.
By the method, ineffective segmentation of the image block ABC can be reduced, and the coding rate is saved;
Example 4
As shown in fig. 1, A, B, C is three image sub-blocks of image block ABC, the colors of the image sub-blocks representing the effective density of the corresponding image sub-blocks from light to deep as low as high.
In this example, the initial threshold lies between the effective information densities of image sub-block a and image sub-block B.
The common boundary of the image sub-block A and the image sub-block B is a dividing line which penetrates through and is perpendicular to the boundary of the image block, the effective information density of the image sub-block A is smaller than a threshold value, the effective information density of the image sub-block B is larger than the threshold value, the cutting condition is met, the common boundary of the image sub-block B and the image sub-block C is a dividing line which penetrates through and is perpendicular to the boundary of the image block, but the effective information density of the image sub-block B and the effective information density of the image sub-block C are both larger than the threshold value, and the cutting condition is not met.
Therefore, the image block ABC is cut into the image block A and the image block BC by taking the common boundary of the image sub-block A and the image sub-block B as a dividing line, and the effective information density of the image block A is smaller than a threshold value, and no dividing line meeting the cutting condition exists, so that the image block ABC is discarded and not assembled into the spliced graph.
Removing image block ABC, adding image block BC, and recalculating a threshold value, wherein the updated threshold value is the minimum value of the effective information density of the image blocks and the image sub-blocks which are not discarded in the pixel rate limit of the mosaic image.
In this example, the updated threshold lies between the effective information densities of image sub-block B and image sub-block C.
Repeating the above process for the image block BC generated after cutting, wherein the common boundary of the image sub-block B and the image sub-block C is a dividing line which penetrates through and is perpendicular to the boundary of the image block, the effective information density of the image sub-block B is smaller than a threshold value, the effective information density of the image sub-block C is larger than the threshold value, and the cutting condition is met;
Therefore, the image block BC is cut into the image block B and the image block C by taking the common boundary of the image sub-block B and the image sub-block C as a dividing line, and the effective information density of the image block B is smaller than a threshold value and no dividing line meeting the cutting condition exists, so that the image block BC is discarded and not assembled into the spliced graph.
The cut is terminated, as shown in fig. 6, with discarded image blocks within the dashed box, and the final image block ABC is cut into image block a, image block B, and image block C, where image sub-a, image block B are discarded and not entered into the mosaic.
And removing the image block BC, adding the image block C, and recalculating a threshold value, wherein the updated threshold value is the minimum value of the effective information density of the image blocks and the image sub-blocks which are not discarded in the pixel rate limit of the mosaic image.
According to the method, after each cutting, the threshold value is updated according to the limitation of the pixel rate and the effective information density of the image sub-blocks which are not discarded, so that the threshold value is more accurate, the cutting is more sufficient, and the overall coding efficiency is increased.
Example 5
As shown in fig. 2, A, C, B is three image sub-blocks of the image block ACB, and the colors of the image sub-blocks from light to deep represent that the effective density of the corresponding image sub-blocks is lowest to high.
In this example, the initial threshold lies between the effective information densities of image sub-block B and image sub-block C.
The common boundary of the image sub-block A and the image sub-block C is a dividing line which penetrates through and is perpendicular to the boundary of the image block, the effective information density of the image sub-block A is smaller than a threshold value, and the effective information density of the image sub-block C is larger than the threshold value, so that a cutting condition is met;
The effective information density difference of the image sub-block A and the image sub-block C is larger, so that the common boundary of the image sub-block A and the image sub-block C is taken as an actual dividing line, the image block ACB is cut into the image block A and the image block CB, and the dividing line meeting the cutting condition is not existed because the effective information density of the image block A is smaller than a threshold value, and the image block A is discarded and not assembled into the spliced graph.
And removing the image block ACB, adding the image block CB, and recalculating a threshold value, wherein the updated threshold value is the minimum value of the effective information density of the image blocks and the image sub-blocks which are not discarded in the pixel rate limit of the mosaic image.
In this example, the updated threshold lies between the effective information densities of image sub-block C and image sub-block B.
Repeating the above process for the image block CB generated after cutting, wherein the common boundary of the image sub-block C and the image sub-block B is a dividing line which penetrates through and is perpendicular to the boundary of the image block, the effective information density of the image sub-block B is smaller than a threshold value, the effective information density of the image sub-block C is larger than the threshold value, and the cutting condition is met;
Therefore, the image block CB is cut into the image block C and the image block B by taking the common boundary of the image sub-block C and the image sub-block B as a dividing line, and the effective information density of the image block B is smaller than a threshold value and no dividing line meeting the cutting condition exists, so that the image block CB is discarded and not assembled into the spliced graph.
The cutting is terminated, as shown in fig. 7, the discarded image blocks are within the dashed frame, and the final image block ACB is cut into an image block a, an image block B, and an image block C, where the image block a, the image block B are discarded and not entered into the mosaic.
And removing the image block CB, adding the image sub-block C, and recalculating a threshold value, wherein the updated threshold value is the minimum value of the effective information density of the image blocks which are not discarded and the image sub-blocks within the pixel rate limit of the spliced image.
Example 6
As shown in fig. 2, A, C, B is three image sub-blocks of the image block ACB, and the colors of the image sub-blocks from light to deep represent that the effective density of the corresponding image sub-blocks is lowest to high.
In this example, the initial threshold lies between the effective information densities of image sub-block a and image sub-block B.
The common boundary of the image sub-block A and the image sub-block C is a dividing line which penetrates through and is perpendicular to the boundary of the image block, the effective information density of the image sub-block A is smaller than a threshold value, the effective information density of the image sub-block C is larger than the threshold value, the cutting condition is met, the common boundary of the image sub-block C and the image sub-block B is a dividing line which penetrates through and is perpendicular to the boundary of the image block, but the effective information density of the image sub-block C and the effective information density of the image sub-block B are both larger than the threshold value, and the cutting condition is not met.
Therefore, the image block ACB is cut into the image block A and the image block CB by taking the common boundary of the image sub-block A and the image sub-block C as a dividing line, and the effective information density of the image block A is smaller than a threshold value and no dividing line meeting the cutting condition exists, so that the image block A is discarded and not assembled into the spliced graph.
And removing the image block ACB, adding the image block CB, and recalculating a threshold value, wherein the updated threshold value is the minimum value of the effective information density of the image blocks and the image sub-blocks which are not discarded in the pixel rate limit of the mosaic image.
In this example, the updated threshold is located below the threshold of image sub-block B.
For an image block CB generated after cutting, repeating the process, wherein the common boundary of the image sub-block C and the image sub-block B is a dividing line which penetrates through and is perpendicular to the boundary of the image block, but the effective information density of the image sub-block C and the effective information density of the image sub-block B are both larger than a threshold value, and the cutting condition is not met;
The cut is terminated, as shown in fig. 8, with discarded image blocks within the dashed box, and the final image block ACB is cut into image block a and image block CB, where image block a is discarded and does not enter the mosaic.
Example 7
The embodiment of the invention provides a device for processing image blocks in immersive media coding
As shown in fig. 3, B, A, C is three image sub-blocks of the image block BAC, the colors of the image sub-blocks represent the effective density of the corresponding image sub-blocks from light to deep as low as high.
In this example, the initial threshold lies between the effective information densities of image sub-block a and image sub-block B.
And the effective information density calculating module is used for inputting the image sub-block A, the image sub-block B and the image sub-block C and outputting the effective information densities of the image sub-block A, the image sub-block B and the image sub-block C.
The threshold judging module is used for inputting effective information densities of an image sub-block A, an image sub-block B and an image sub-block C respectively, wherein the common boundary of the image sub-block B and the image sub-block A is a dividing line which penetrates through and is perpendicular to the boundary of the image block, the effective information density of the image sub-block B is larger than a threshold value, the effective information density of the image sub-block A is smaller than the threshold value, and a cutting condition is met;
And the dividing line decision module is used for inputting the effective information density difference value of the common boundary of the image sub-block B and the image sub-block A, the common boundary of the image sub-block A and the image sub-block C and the image sub-blocks at the two sides of the common boundary and the effective information density difference value, and outputting the effective information density difference value as an actual dividing line. Since the effective information density difference between the image sub-block a and the image sub-block C is larger, the common boundary between the image sub-block a and the image sub-block C is taken as the actual dividing line.
And the cutting module is used for inputting a common boundary of the image sub-block A and the image sub-block C of the parting line, cutting the decision and outputting the cut image block. The image block BAC is cut into an image block BA and an image block C;
The assembly and splicing decision module is used for inputting the image sub-block A with the effective information density smaller than the threshold value into the image block BA, and the image block BA has a dividing line meeting the cutting condition, so that the image block BA is not discarded.
And the threshold updating module inputs the effective information density and pixel rate limit of the image blocks BA and C which are not discarded and outputs the effective information density and pixel rate limit as an updated threshold. Removing image block BAC, adding image block BA and image block C, recalculating a threshold value, and updating the threshold value to be the minimum value of the effective information density of the image blocks which are not discarded in the pixel rate limit of the mosaic.
In this example, the updated threshold lies between the effective information densities of image sub-block a and image sub-block B.
Repeating the device flow for the image block BA generated after cutting, wherein the common boundary of the image sub-block B and the image sub-block A is a dividing line which penetrates through and is perpendicular to the boundary of the image block, the effective information density of the image sub-block B is larger than a threshold value, the effective information density of the image sub-block A is smaller than the threshold value, and the cutting condition is met;
And the image block BA is cut into the image block B and the image block A by taking the common boundary of the image sub-block B and the image sub-block A as a dividing line, and the effective information density of the image block A is smaller than a threshold value and no dividing line meeting the cutting condition exists, so that the image block BA is discarded and not assembled into the spliced graph.
The cut is terminated, as shown in fig. 9, with discarded image blocks within the dashed box, and the final image block BAC is cut into image block B, image block a, and image block C, where image block a is discarded and does not enter the mosaic.
And adding the image block B, and recalculating a threshold value, wherein the updated threshold value is the minimum value of the effective information density of the image blocks and the image sub-blocks which are not discarded in the pixel rate limit of the spliced image.
By the device, the segmentation of the granularity of the image sub-blocks can be realized, the image block A with low effective information density is accurately discarded, the image block with higher effective information density can be organized under the limit of the limited pixel rate in the multi-view image splicing process, the information utilization rate of a spliced image is improved, and the rendering quality is improved at a decoding end;
it should be noted that the above-mentioned embodiments are merely for illustrating the technical solution of the present invention, and not for limiting the same, and although the present invention has been described in detail with reference to the above-mentioned embodiments, it should be understood by those skilled in the art that the technical solution described in the above-mentioned embodiments may be modified or some technical features may be equivalently replaced, and these modifications or substitutions do not deviate the essence of the corresponding technical solution from the scope of the technical solution of the embodiments of the present invention.