HK1251970B - Decoder and method for generating high dynamic range images, and storage medium - Google Patents

Description

Decoder, method and storage medium for generating high dynamic range images

This application is a divisional application of the invention patent application with the application date of December 18, 2012, application number "201280066010.4", and invention name "Specified visual dynamic range encoding operations and parameters".

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Application No. 61/582,614, filed January 3, 2013, the disclosure of which is hereby incorporated by reference in its entirety.

Technical Field

The present application relates generally to video coding systems, and more particularly to systems for encoding, decoding, and presenting visual dynamic range images.

Background Art

Display technologies have been developed to support sending and presenting video content based on specific video formats. For example, MPEG video encoders and decoders can support video content encoded in the MPEG video format. Other video encoders and decoders can support video content encoded in different video formats.

Consumer devices, such as handheld devices, are typically installed or configured with a limited set of video encoding systems, each of which may support a specific video format from a limited set of video formats. Consequently, if a block of video content is not encoded and transmitted in accordance with the expected video format, the device may not be able to find a suitable video decoder to decode the video content and assist in presenting the video content. Even if the video content is presented, the presented video content may include an incorrect interpretation or representation of the received video content and produce visible artifacts of color and brightness values. WO2008/077272A1, WO2008/083521A1, WO2009/051692A2, and WO2010/105036A1 describe encoding video data in a scalable or layered manner.

The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any approach described in this section is prior art simply by virtue of its inclusion in this section. Similarly, unless otherwise indicated, it should not be assumed based on this section that a problem identified with respect to one or more approaches has been recognized in any prior art.

Summary of the Invention

According to one embodiment, a video encoding method is provided, comprising: receiving an input visual dynamic range image and an input base layer image associated with the input visual dynamic range image; generating a coding syntax including a plurality of syntax elements at at least a frame level for encoding-related operations; converting the input base layer image and the input visual dynamic range image into base layer data and enhancement layer data according to the coding syntax; wherein the enhancement layer data includes a residual value between the input visual dynamic range image and a predicted visual dynamic range image generated based on the base layer data; wherein the plurality of syntax elements specify one or more of the following operations: chroma resampling operation, inverse mapping operation, prediction operation based on non-overlapping areas, prediction operation based on overlapping areas, residual nonlinear quantization and dequantization operation, residual chroma resampling operation, residual spatial upsampling operation, data processing operation including interpolation, performing these operations to generate the base layer data and the enhancement layer data; encoding the coding syntax elements into reference processing unit data so that a corresponding decoder can use the same coding syntax for decoding-related operations; wherein the reference processing unit The data includes a current reference processing unit data unit; wherein the current reference processing unit data unit includes a reference processing unit data header and a reference processing unit data payload; in the current reference processing unit data unit, at least one of the multiple syntax elements in the coding syntax is indicated as being capable of being predicted based on a previous input visual dynamic range image and one or more previous reference processing unit data units of a previous input base layer image associated with the previous input visual dynamic range image; and the base layer data, the enhancement layer data and the reference processing unit data are output as a base layer signal, an enhancement layer signal and a reference processing unit signal; wherein the current reference processing unit data unit includes a flag indicating whether a frame-level syntax element of the previously sent reference processing unit data should be reused; wherein the reference processing unit data header includes a field indicating one or more visual dynamic range versions to which the reference processing unit data relates; the coding syntax included in the reference processing unit data enables reconstructing the input visual dynamic range image based on the base layer data and the enhancement layer data.

According to another embodiment, a video decoding method is provided, comprising: receiving base layer data, enhancement layer data and reference processing unit data in the form of a base layer signal, an enhancement layer signal and a reference processing unit signal, wherein the base layer data, the enhancement layer data and the reference processing unit data are associated with a common visual dynamic range source image; wherein the enhancement layer data comprises a residual value between the visual dynamic range source image and a predicted visual dynamic range image generated based on the base layer data; decoding the reference processing unit data into a coding syntax comprising a plurality of syntax elements at at least a frame level; wherein the plurality of syntax elements specify one or more of the following operations: a chroma resampling operation, an inverse mapping operation, a prediction operation based on a non-overlapping area, a prediction operation based on an overlapping area, a residual nonlinear quantization and dequantization operation, a residual chroma resampling operation, a residual spatial upsampling operation, a data processing operation including interpolation, performing these operations to generate the base layer data and the enhancement layer data; converting the base layer data and the enhancement layer data into a coding syntax comprising a plurality of syntax elements at at least a frame level; wherein the plurality of syntax elements specify one or more of the following operations: a chroma resampling operation, an inverse mapping operation, a prediction operation based on a non-overlapping area, a prediction operation based on an overlapping area, a residual nonlinear quantization and dequantization operation, a residual chroma resampling operation, a residual spatial upsampling operation, a data processing operation including interpolation, performing these operations to generate the base layer data and the enhancement layer data; converting the base layer data and the enhancement layer data into a coding syntax comprising a plurality of syntax elements at at least a frame level; data into a reconstructed visual dynamic range image; wherein the reference processing unit data includes a current reference processing unit data unit; wherein the current reference processing unit data unit includes a reference processing unit data header and a reference processing unit data payload; wherein the current reference processing unit data unit includes a flag indicating whether a frame-level syntax element of a previously sent reference processing unit data should be reused; wherein the reference processing unit data header includes a field indicating one or more visual dynamic range versions to which the reference processing unit data relates; wherein the coding syntax included in the reference processing unit data enables determination of a reconstructed visual dynamic range image based on the base layer data and the enhancement layer data; based on the current reference processing unit data unit, determining at least one syntax element of the multiple syntax elements of the coding syntax as being predictable based on one or more previous reference processing unit data units related to a previously reconstructed visual dynamic range image; and outputting the reconstructed visual dynamic range image.

The embodiment of the present invention also includes an encoder for executing the above encoding method.

The embodiment of the present invention also includes a decoder for executing the above decoding method.

The embodiment of the present invention also includes a system for executing the above method.

According to another embodiment, a video encoding system comprises: an encoder configured to perform the following operations: receive an input visual dynamic range image and an input base layer image associated with the input visual dynamic range image; generate a coding syntax including multiple syntax elements at at least frame level for encoding-related operations; convert the input visual dynamic range image and the input base layer image into base layer data and enhancement layer data according to the coding syntax; wherein the enhancement layer data includes a residual value between the input visual dynamic range image and a predicted visual dynamic range image generated based on the base layer data; wherein the multiple syntax elements specify one or more of the following operations: chroma resampling operation, inverse mapping operation, prediction operation based on non-overlapping area, prediction operation based on overlapping area, residual nonlinear quantization and dequantization operation, residual chroma resampling operation, residual spatial upsampling operation, data processing operation including interpolation, performing these operations to generate the base layer data and the enhancement layer data; encoding the coding syntax into reference processing unit data so that a corresponding decoder can use the same coding syntax for decoding-related operations; wherein The reference processing unit data includes a current reference processing unit data unit; wherein the current reference processing unit data unit includes a reference processing unit data header and a reference processing unit data payload; wherein the current reference processing unit data unit includes a flag indicating whether a frame-level syntax element of a previously sent reference processing unit data should be reused; wherein the reference processing unit data header includes a field indicating one or more visual dynamic range versions to which the reference processing unit data relates; the coding syntax included in the reference processing unit data enables the input visual dynamic range image to be reconstructed based on the base layer data and the enhancement layer data; in the current reference processing unit data unit, at least one of the multiple syntax elements in the coding syntax is indicated as being capable of being predicted based on a previous input visual dynamic range image and one or more previous reference processing unit data units of a previous input base layer image associated with the previous input visual dynamic range image; and the base layer data, the enhancement layer data and the reference processing unit data are output as a base layer signal, an enhancement layer signal and a reference processing unit signal.

According to another embodiment, a video decoding system comprises: a decoder configured to perform the following operations: receiving base layer data, enhancement layer data and reference processing unit data in the form of a base layer signal, an enhancement layer signal and a reference processing unit signal, wherein the base layer data, the enhancement layer data and the reference processing unit data are associated with a common visual dynamic range source image; wherein the enhancement layer data comprises a residual value between the visual dynamic range source image and a predicted visual dynamic range image generated based on the base layer data; decoding the reference processing unit data into a coding syntax comprising a plurality of syntax elements at at least a frame level; wherein the plurality of syntax elements specify one or more of the following operations: chroma resampling operation, inverse mapping operation, prediction operation based on non-overlapping areas, prediction operation based on overlapping areas, residual nonlinear quantization and dequantization operation, residual chroma resampling operation, residual spatial upsampling operation, data processing operation including interpolation, performing these operations to generate the base layer data and the enhancement layer data; converting the base layer ... the base layer data and the enhancement layer data being converted into a reconstructed visual dynamic range image; wherein the reference processing unit data comprises a current reference processing unit data unit; wherein the current reference processing unit data unit comprises a reference processing unit data header and a reference processing unit data payload; wherein the current reference processing unit data unit comprises a flag indicating whether a frame-level syntax element of a previously sent reference processing unit data should be reused; wherein the reference processing unit data header comprises a field indicating one or more visual dynamic range versions to which the reference processing unit data relates; wherein the coding syntax included in the reference processing unit data enables the reconstructed visual dynamic range image to be determined based on the base layer data and the enhancement layer data; according to the current reference processing unit data unit, at least one syntax element of the multiple syntax elements of the coding syntax is determined as being predictable based on one or more previous reference processing unit data units related to the previously reconstructed visual dynamic range image; and outputting the reconstructed visual dynamic range image.

According to one embodiment, a decoder for generating a high dynamic range image is provided, wherein the decoder includes a non-volatile memory and one or more processors, wherein generating an output image using the decoder includes: receiving reference processing unit (RPU) data and storing at least a portion of the RPU data in the non-volatile memory; extracting at least an RPU data header and an RPU data payload from the RPU data; receiving a base layer image; receiving an enhancement layer image; determining whether the decoder can use syntax elements from a previous RPU data payload; when it is determined that the decoder cannot use syntax elements from a previous RPU data payload, parsing the RPU data payload to extract syntax elements, and when it is determined that the decoder can use syntax elements from a previous RPU data payload, decoding a previous RPU Id value to predict syntax elements, wherein the syntax elements include a current RPU Id value, a mapping color space indicator, a chroma resampling filter indicator, a principal component mapping parameter, and a partition parameter; and combining the base layer image and the enhancement layer image based on the syntax elements to generate an output image. Parsing the RPU data payload includes: decoding the current RPU Id value; decoding the mapped color space indicator; decoding the chroma resampling filter indicator; decoding the pivot mapping parameters; and decoding the partition parameters.

According to one embodiment, a method for generating a high dynamic range image using a decoder is provided, wherein generating an output image using the decoder includes: receiving reference processing unit (RPU) data and storing at least a portion of the RPU data in a non-volatile memory; extracting at least an RPU data header and an RPU data payload from the RPU data; receiving a base layer image; receiving an enhancement layer image; determining whether the decoder can use syntax elements from a previous RPU data payload; when it is determined that the decoder cannot use syntax elements from a previous RPU data payload, parsing the RPU data payload to extract syntax elements, and when it is determined that the decoder can use syntax elements from a previous RPU data payload, decoding a previous RPU Id value to predict syntax elements, wherein the syntax elements include a current RPU Id value, a mapping color space indicator, a chroma resampling filter indicator, a principal component mapping parameter, and a partition parameter; and combining the base layer image and the enhancement layer image based on the syntax elements to generate an output image. Parsing the RPU data payload includes: decoding the current RPU Id value; decoding the mapped color space indicator; decoding the chroma resampling filter indicator; decoding the pivot mapping parameters; and decoding the partition parameters.

According to one embodiment, a non-transitory computer-readable storage medium storing computer-executable instructions is provided. The computer-executable instructions are used to execute the method according to the above embodiment using one or more processors.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is illustrated by way of example and not by way of limitation in the figures of the accompanying drawings in which like reference numerals refer to like elements, and in which:

1 and 2 illustrate a visual dynamic range (VDR) encoder that generates reference processing unit (RPU) data based on a coding syntax in accordance with one or more VDR specifications in an example embodiment;

FIG3 illustrates a NAL data unit including a NAL header and a raw byte sequence payload in an example embodiment;

Figure 4 shows the layout of the RPU data header. Figure 5 shows the parsing of the RPU data header in an example embodiment;

FIG6 illustrates the layout of an RPU data payload, and FIG7 to FIG9 illustrate RPU data payload decoding in an example embodiment;

FIG10 illustrates a VDR decoder parsing encoding syntax from RPU data in an example embodiment;

11A and 11B illustrate an example process flow according to an example embodiment of the present invention; and

FIG. 12 illustrates an example hardware platform upon which the computers or computing devices described herein may be implemented, according to an embodiment of the invention.

DETAILED DESCRIPTION

This paper describes an example embodiment that relates to using a layered VDR codec to encode, decode and represent visual dynamic range images. In the following description, for illustrative purposes, a large number of specific details are set forth in order to provide a thorough understanding of the present invention. However, it is apparent that the present invention can be practiced without these specific details. In other examples, in order to avoid unnecessarily closing, blurring or confusing the present invention, known structures and equipment will no longer be described in detail.

Example implementations are described herein according to the following outline:

1. General Overview

2. VDR encoder

3.RPU data unit

4. RPU data decoding—sequence level and/or frame level

5. RPU data decoding - partition level

6. RPU data decoding - chroma mapping

7. Additional Examples of RPU Data Decoding

8. Example Processing Flow

9. Implementation Mechanism - Hardware Overview

10. Equivalents, Extensions, Substitutions and Other Matters

1. General Overview

This overview provides a basic description of some aspects of the example embodiments of the present invention. It should be noted that this overview is not an extensive or exhaustive summary of aspects of the example embodiments. Furthermore, it should be noted that this overview is neither intended to be understood as identifying any particularly significant aspects or elements of the example embodiments, nor is it intended to be understood as specifically delineating any scope of the example embodiments, or generally describing any scope of the present invention. This overview merely provides some concepts related to the example embodiments in a concise and simplified form, and should be understood as a conceptual prelude to the more detailed description of the example embodiments that follows.

The technology described herein supports the transmission of reference processing data generated by different video coding systems using a common carrier and signaling it to downstream devices. As used herein, the term "common carrier" can refer to a common reference processing unit (RPU) data format configured to carry reference processing data generated based on any one of a variety of visual dynamic range (VDR) specifications. As used herein, the reference processing data carried in the common carrier provides a coding syntax including a plurality of syntax elements, wherein the syntax elements specify the operations and parameters used or to be used in the encoding and decoding of video data associated with the reference processing data. The syntax elements described herein can describe operations that are the same for some or all different VDR specifications or describe operations that are specific to only one or more of the different VDR specifications, rather than all different specifications. A common reference processing data decoding/parsing process can be implemented, for example, in a downstream (e.g., consumer) device to decode the coding syntax or the syntax elements therein, regardless of which VDR specification the coding syntax relates to. Therefore, downstream devices do not need to implement separate and different reference processing data decoding/parsing processes for each existing or new VDR specification that the consumer device is configured to or will be configured to support. Suppliers of downstream equipment may only need to focus on providing support for algorithms or operations for encoding or decoding media samples specified in existing or new VDR specifications. Because the technology described herein uses a common carrier to transmit or signal reference processing data associated with media samples generated according to different VDR specifications, the same reference processing data decoding/parsing process can be reused or substantially reused for all VDR specifications, including VDR specifications yet to be developed. As used herein, the term "media sample" refers to data that, when combined with the reference processing data, forms the VDR data described herein.

Multiple layers (or bitstreams) can be used to transmit VDR data (media samples and reference processing unit data) from an upstream device such as a VDR encoder to a downstream device. The VDR data carried in multiple layers can be used to support a wide range of display technologies, which may include but are not limited to any of backward compatibility display technologies and new high dynamic range (HDR) display technologies. As used herein, the term "VDR" or "visual dynamic range" may refer to a dynamic range wider than a standard dynamic range, and may include but is not limited to a wide dynamic range up to the instantaneous perceptible dynamic range and color gamut that human vision can instantaneously perceive. As used herein, the term "multi-layer" or "multiple layers" may refer to two or more bitstreams, including a base layer (BL), a reference processing unit (RPU) layer, and an enhancement layer (EL) that carry multiple video or image signals having one or more logical dependencies (video signals) with each other.

The base layer may carry BL data (base layer media samples) obtained by an upstream device from an SDR signal or mapped from input VDR data in a VDR signal. One or more enhancement layers (EL) may carry EL data (enhancement layer media samples) obtained at least in part from a VDR signal by an upstream device. In some embodiments, in order to exploit statistical redundancy between BL data and EL data, both of which are related to the same input VDR data, the EL data may be (redundantly) reduced to include residual values or differences between prediction values based on the BL video data and the input VDR video data. In some embodiments, the VDR encoder may be configured to apply an accurate prediction algorithm such that the residual values are reduced to zero; thus, the EL data may be used to save a reduced set of inter-layer reference pictures associated with the accurate prediction algorithm without saving unhelpful zero residual values. Inter-layer reference pictures for a residual-free video coding system may be generated not for a single VDR image, but for a set of related input VDR images.

In some embodiments, the RPU layer may carry reference processing data (or denoted as RPU data) generated by an upstream device. An upstream device, such as a VDR encoder, may use the RPU data to signal a coding syntax to a downstream device, such as a VDR decoder. The coding syntax enables the VDR decoder to reconstruct a VDR image based on the BL data and EL data in the BL layer and the EL layer.

Examples of syntax elements carried in RPU data may include, but are not limited to, any of the following: inter-layer prediction coefficients, residual nonlinear dequantization parameters, chroma resampling filter coefficients, color space transform indicators, or other VDR syntax elements (e.g., flags and descriptors of functions and/or operations performed by a VDR encoder to generate BL data and EL data and performed by a VDR decoder to decode BL data and EL data). Syntax elements in RPU data may be categorized as one of a sequence level, a frame level, a partition level, or a function/operation level.

Input VDR media content (e.g., HDR movie) can be subdivided into sequences (e.g., corresponding to a scene or a portion of a scene, etc.), frames (or images or pictures), or partitions (or portions of an image). Syntax elements at the sequence level, frame level, or partition level can be explicitly encoded in the current RPU data or predicted from previously sent RPU data at the corresponding sequence level, frame level, or partition level. Additionally, optionally, or alternatively, syntax elements can appear at more than one level of the sequence level, frame level, or partition level.

The techniques described herein, including a layered codec structure (BL layer, EL layer, and RPU layer), can be implemented by different VDR encoding systems. For example, a first VDR encoding system implementing some of these techniques can be a residual-based layered codec in which both the base layer and the enhancement layer use a chroma format (e.g., 4:2:0) and a low bit depth (e.g., 8 bits); a second VDR encoding system implementing some of these techniques can be a signal-based layered codec in which the enhancement layer uses a wider chroma format (e.g., 4:4:4) and a higher bit depth (e.g., 12 bits or more) than the chroma format (e.g., 4:2:0) and bit depth (e.g., 8 bits) of the base layer (4:2:0 8 bits). Specifically, the RPU data decoding techniques described herein can be implemented in a VDR encoding system that initially supports a set of one or more different VDR specifications and can be reused with little or no changes in the VDR encoding system when additional VDR specifications are subsequently supported.

The RPU layer coded bitstream or RPU data therein may be synchronized with the coded bitstreams of other layers, or BL data and EL data therein. For example, the RPU data and BL/EL data may be synchronized by picture display number according to the display order (e.g., as specified in H.264, picture_order_count).

The techniques described herein support a variety of operations for the following: inter-layer prediction, inverse mapping, chroma resampling, data processing such as interpolation in the boundary region of a partition, spatial scaling, non-linear quantization, etc. Some of the supported operations may be common to some or all of the different VDR specifications, while some other supported operations may be unique to one or more specifications rather than to all VDR specifications. For example, non-linear quantization/dequantization may be performed for VDR specifications with residual values, such as EL data.

The technology described herein supports video encoding and decoding driven by flexible coding syntax. The method enables parallel and continuous optimization of encoder and decoder designs, for example, using improved algorithms, implementation costs, speed, etc. Utilizing the redundancy between current RPU data and previously transmitted RPU data, the coding syntax can be effectively sent and signaled to the VDR decoder along with the layered VDR data via the VDR encoder. The coding syntax provides a roadmap (e.g., complete) for the VDR decoder to effectively perform decoding operations, such as in the inverse data stream.

In some embodiments, the mechanisms described herein form part of a media processing system including, but not limited to, any of the following: a handheld device, a game console, a television, a laptop computer, a netbook computer, a tablet computer, a cellular wireless telephone, an electronic book reader, a point-of-sale terminal, a desktop computer, a computer workstation, a computer kiosk, or various other types of terminals and media processing units.

Various modifications to the preferred embodiments and general principles and features described herein will be apparent to those skilled in the art. Therefore, the present disclosure is not intended to be limited to the embodiments shown, but is consistent with the broadest scope consistent with the principles and features described herein.

2. VDR encoder

A VDR encoder can generate BL data, EL data, and RPU data using a coding syntax that conforms to one of one or more different VDR specifications. These different VDR specifications can be marked or identified using different versions including different combinations of major and/or minor version numbers (other methods of identifying specific VDR specifications can be similarly used). As used herein, a VDR specification can provide specifications for syntax elements that can be included in the coding syntax, and the specification can be signaled from an upstream device, such as a VDR encoder, to a downstream device, such as a VDR decoder.

FIG1 illustrates a VDR encoder 102 that generates RPU data based on a coding syntax that conforms to one or more VDR specifications. In some embodiments, the VDR encoder 102 supports at least two VDR specifications, for example, each labeled with a first version (“1.0”) or a second version (“1.x”). The VDR encoder 102 can be configured to perform operations on BL data, EL data, RPU data, inter-layer prediction data, and intermediate media data according to a coding syntax that conforms to one or more VDR specifications supported by the VDR encoder 102. These different VDR specifications may include, but are not limited to, a first version that supports backward compatibility and a second version that does not support backward compatibility. As used herein, the term “backward compatibility” refers to whether the BL data includes an SDR image that is optimized for viewing on an SDR display. The VDR encoder 102 can be implemented using one or more computing devices.

In one embodiment, the VDR encoder 102 is configured to receive (input) a VDR signal 104 and obtain an input VDR image from the VDR signal 104. As used herein, an "input VDR image" may include wide or high dynamic range image data used to decode a VDR version of a source image, which may in turn be an original image captured by a high-end image acquisition device. The input VDR image may be a high bit depth (e.g., 10+ bits) image in an input color space that supports a high dynamic range color gamut. Examples of one or more VDR signals received or processed by the VDR encoding system described herein include, but are not limited to, any of the following: a 12-bit P3 D65 RGB 444 signal, a 12-bit Recommended (Rec.) 709 RGB 444 signal, a 12-bit DCDM X'Y'Z'444 signal, video data in a 16-bit TIFF file format, and the like.

In an example, each pixel represented in the input VDR image includes pixel values for all channels (e.g., red channel, green channel, and blue channel) defined for a color space (e.g., RGB color space). Each pixel may optionally and/or alternatively include upsampled pixel values or downsampled pixel values for one or more channels in the color space. It should be noted that in some embodiments, in addition to the three basic colors such as red, green, and blue, different basic colors may be used simultaneously in the color space described herein, for example, to support a wide color gamut; in these embodiments, the image data described herein includes additional pixel values for these different basic colors and may be processed simultaneously by the techniques described herein.

In one embodiment, the VDR encoder 102 may perform inter-layer prediction related operations in a mapped color space (YCbCr space, RGB space, or one of other color spaces). In some embodiments, if the input color space is different from the mapped color space, the input VDR image may be converted from the input color space to the mapped color space by a color space conversion unit.

1 , the VDR encoder 102 may be configured to receive (input) an SDR signal 108 and obtain BL data from the SDR signal 108. Examples of the one or more SDR signals received by the VDR encoding system described herein include, but are not limited to, any of the following: an 8-bit YCbCr signal, video data in an 8-bit YUV file format, and the like.

As used herein, "BL data" refers to low-bit-depth (e.g., 8-bit) image data that may or may not be optimized for viewing on an SDR display. As used herein, the term "low bit-depth" refers to image data quantized in a coding space with a low bit-depth; examples of low bit-depth include 8 bits, while the term "high bit-depth" refers to image data quantized in a coding space with a high bit-depth; examples of high bit-depth are 10 bits, 12 bits, or more. Specifically, the terms "low bit-depth" or "high bit-depth" do not refer to the least significant bit or the most significant bit of a pixel value.

In a first example, the BL data includes an SDR image optimized for viewing on an SDR display and may be associated with a first version of the VDR specification that supports backward compatibility. The SDR image may include color corrections performed by a colorist to make the SDR image appear as realistic as possible within a relatively narrow or standard dynamic range. For example, to create an image that appears realistic within a standard dynamic range, tonal information associated with some or all pixels in a source HDR image that produced the input VDR image may be changed or corrected in the SDR image.

In a second example, the VDR encoder 102 is configured to apply a VDR-SDR (e.g., tone) mapping to an input VDR image to obtain BL data, rather than obtaining the BL data from an input SDR signal similar to 108 of FIG. 1 . In this example, the BL data may not be optimized for viewing on an SDR display and may be associated with a second version of the VDR specification that does not support backward compatibility. The BL data may include a low-bit representation of the input VDR image. For example, the VDR-SDR mapping may be based on one or more of the following: global quantization, linear quantization, linear stretching, curve-based quantization, probability density function (PDF) optimized quantization, Roy-maximum quantization, partition-based quantization, perceptual quantization, cross-color channel/vector quantization, or other types of quantization. Additionally, optionally, or alternatively, for example, the VDR-SDR mapping may include zero or more of the following processes: denoising, frame alignment, color grading, etc. In this example, the BL data may not be optimized for rendering realistic-looking images within a standard dynamic range. Conversely, the BL data may be effectively combined with the EL data by downstream equipment to construct an output VDR image corresponding to the input VDR image obtained from the input VDR signal of FIG. 1 .

In one embodiment, the VDR encoder 102 or the base layer SDR encoder (116) therein is configured to encode an input SDR image that may be obtained from the SDR signal 108 or from a mapping operation performed on an input VDR image obtained from the VDR signal 104 into a base layer bitstream 128.

In one embodiment, the VDR encoder 102 employs a hybrid video coding model such as H.264/MPEG-4 AVC (IS14496-10), HEVC, MPEG-4 Part 2 (IS14496-2), MPEG-2 (IS11138-2), VP8, VC-1, and/or others. Media samples to be encoded in the base layer may be predicted based on neighboring samples in the same picture (using intra prediction) or samples from previously decoded pictures belonging to the same base layer (inter prediction). These decoded BL samples to be used for prediction may be stored or cached in the reference picture store 114 (of the base layer).

In one embodiment, the VDR encoder 102 is further configured to perform inter-layer prediction on media samples to be encoded in the enhancement layer based on the decoded BL samples. The decoded BL samples may be retrieved from a reference picture store 114 (which may be one or more memory buffers or other forms of memory space).

The VDR encoder 102 or the VDR RPU 110 therein can be configured to generate coding syntax for coding-related operations performed by the VDR encoder 102. These coding-related operations include operations performed to generate EL data to be transmitted in the enhancement layer bitstream 124. According to the techniques described herein, the coding syntax for coding-related operations is signaled to the VDR decoder by the VDR encoder so that the VDR decoder uses the same coding syntax for decoding-related operations. In some embodiments, the coding syntax signaled to the VDR decoder by the VDR encoder can specify one or more additional operations that are performed separately by the VDR decoder rather than by the VDR encoder. These additional operations include, but are not limited to, any operation in the display management operation.

These encoding syntaxes may include multiple syntax elements that conform to the specific VDR specification supported by the VDR encoder 102, and may include, but are not limited to, any of the following: inter-layer prediction coefficients, residual non-linear dequantization parameters, chroma resampling filter coefficients, color space transform indicators, or other VDR syntax elements (e.g., flags and descriptors of functions and/or operations performed by the VDR encoder to generate BL data and EL data and by the VDR decoder to decode the BL data and EL data).

In one embodiment, the RPU processing module 112 is configured to perform a series of sequence-level, frame-level, and partition-level operations (which may include, but are not limited to, those related to prediction) based on the coding syntax. For example, the RPU processing module 112 may perform operations specified by the coding syntax, such as, for example, inverse mapping of SDR-VDR mapping, chroma upsampling, one or more video data processing operations (e.g., filtering, interpolation, rescaling, etc.), or non-linear quantization (NLQ). Therefore, the prediction reference value may be generated by the RPU processing module 112.

In one embodiment, the VDR encoder 102 may perform one or more operations to generate a residual value (130) between VDR image data obtained from an input VDR image (or, if necessary, from the VDR signal 104 and the transformed color space) and a prediction reference value. The residual value may be different in the linear or logarithmic domain. In one embodiment, the VDR encoder 102 or a residual downsampling/resampling unit therein may be configured to perform one or more downsampling/resampling operations on the residual value (130) to generate a downsampled (e.g., 8-bit) residual value for further processing. In one embodiment, the VDR encoder 102 or a residual nonlinear quantizer (NLQ; 118) therein may be configured to perform one or more nonlinear quantization operations on the residual value (130) or the downsampled residual value and provide the nonlinearly quantized residual value to other units of the VDR encoder 102 for further processing.

In one embodiment, the VDR encoder 102 or the enhancement layer (8-bit/4:2:0 for illustration purposes only) encoder (120) therein is configured to encode the residual values (which may be non-linearly quantized and/or downsampled in some embodiments) as EL data into the enhancement layer bitstream 124.

In embodiments where the VDR encoder 102 or the enhancement layer (8-bit/4:2:0 for illustration purposes only) encoder (120) employs a hybrid video encoder model, the residual value may be predicted from neighboring residual value samples in the same picture (using intra prediction) or from residual value samples from previously decoded pictures belonging to the same enhancement layer (inter prediction). In one embodiment, the same layer EL samples used for prediction may be stored or cached in the reference picture store (for the enhancement layer) 122.

In one embodiment, the VDR encoder 102 or the VDR RPU 110 therein is configured to encode coding syntax as part of RPU data into the VDR RPU bitstream 126. The RPU data may include, but is not limited to, any of the following: SDR-VDR mapping parameters, polynomial parameters used by a prediction method applied to generate a predicted reference picture, NLQ parameters, parameters used by one or more video data processing operations performed by the VDR RPU (110). The VDR RPU 110 may set a flag or header field in an RPU data unit to indicate whether any syntax element in the coding syntax can be predicted based on previously sent RPU data for a previous sequence, previous frame, or previous partition.

One or both of the BL encoder (116) and the EL encoder (120) may be implemented using one or more of a variety of codecs, such as H.264/MPEG-4 AVC, HEVC, MPEG-2, VP8, VC-1, and/or others.

A corresponding VDR decoder (which implements the inverse data flow shown in FIG. 1 and supports the same VDR specification used by the VDR encoder 102 to generate the encoding syntax) can be used to decode the BL data stream, EL data stream, and RPU data stream generated by the VDR encoder 102 and generate a reconstructed version of the input VDR image.

2 shows a VDR encoder (202) that generates RPU data corresponding to one or more different VDR specifications. The VDR encoder 202 can be, but is not limited to, associated with a third version (e.g., denoted as "2.0") of the VDR specification that may be different from the first or second version implemented by the VDR encoder 102. The VDR encoder 202 can be implemented using one or more computing devices.

In an example embodiment, the VDR encoder 202 is configured to receive (input) a VDR signal 204 and obtain an input VDR image from the VDR signal 204. The input VDR image may include high bit depth (e.g., 10+ bits) image data in an input color space supporting a high dynamic range color gamut.

In some embodiments, if the input color space is different from the mapped color space in which the VDR encoder 202 performs the prediction operation, the input VDR image may be transformed from the input color space to the mapped color space by a color space conversion unit.

In one embodiment, as shown in FIG2 , the VDR encoder 202 is configured to receive (input) an SDR signal 208 and obtain BL data from the SDR signal 208. Alternatively, the VDR encoder 202 is configured to apply a VDR-SDR (e.g., tone) mapping to the input VDR image to obtain the BL data, rather than decoding the BL data from the SDR signal. The BL data may include low bit depth (e.g., 8-bit) image data that may or may not be optimized for viewing on an SDR display. The BL data associated with the VDR encoder 202 may or may not be similar to the BL data associated with the VDR encoder 102 as discussed above.

In one embodiment, the VDR encoder 202 or the base layer SDR encoder (216) therein is configured to encode BL data into a base layer bitstream 228, wherein the BL data may be obtained from the SDR signal 208 or obtained based on a mapping operation performed on an input VDR image obtained from the VDR signal 204.

In one embodiment, the media samples represented by the BL data may be predicted from neighboring samples in the same picture (using intra prediction) or from samples from previously decoded pictures belonging to the same base layer (inter prediction). These samples may be stored or cached in the reference picture store (base layer) 214.

In one embodiment, the VDR encoder 202 is configured to perform inter-layer prediction on high-bit-depth media samples associated with the enhancement layer based on the BL data samples. The BL data samples may be retrieved from a reference picture store 214 in one or more memory buffers or other forms of memory space. In some embodiments, the VDR decoder 202 is designed to perform one or more operations based on an accurate prediction algorithm using inter-layer reference pictures and decoded BL samples to not generate residual values for the high-bit-depth media samples (or, if generated, to zero residual values). Thus, the high-bit-depth media samples can be accurately predicted based at least in part on the inter-layer reference pictures and the decoded BL samples.

The VDR encoder 202 or the VDR RPU 210 therein may be configured to generate coding syntax for generating EL data (which may include inter-layer reference pictures) to be transmitted in the enhancement layer bitstream 224. The coding syntax may include a plurality of syntax elements that conform to the VDR specification supported by the VDR encoder 202, and may include, but is not limited to, any of the following: inter-layer prediction coefficients, chroma resampling filter coefficients, color space transform indicators, other VDR syntax elements (e.g., flags and descriptors of functions and/or operations performed by the VDR encoder to generate BL data, EL data, etc.), etc.

In one embodiment, the RPU processing module 212 is configured to perform a series of operations (which may include but are not limited to those related to prediction) based on the coding syntax. For example, the RPU processing module 212 may perform operations as specified by the coding syntax, such as inverse tone mapping such as SDR-VDR mapping, chroma upsampling, and one or more video data processing operations (e.g., filtering, interpolation, rescaling, etc.). In one embodiment, the RPU processing module 212 does not generate residual values (or all residual values are zero due to accurate prediction operations performed by the VDR encoder 202 or the RPU processing module 212 therein). In this embodiment, since the enhancement layer VDR encoder 220 processes pixel data, the RPU processing module 212 does not perform residual nonlinear quantization (NLQ) to generate EL data. Therefore, the coding syntax generated by the VDR RPU 210 may not have parameters related to residual nonlinear quantization (NLQ).

In one embodiment, the RPU processing module 212 is configured to generate inter-layer reference pictures based on the coding syntax. The inter-layer reference pictures described herein need not be generated for each input VDR image; an inter-layer reference picture may be generated for a sequence of one or more consecutive VDR images obtained from the VDR signal 204. Media samples from the inter-layer reference pictures may be stored or cached in the reference picture store 222 (for the enhancement layer).

In one embodiment, the VDR encoder 202 or the enhancement layer encoder (220) therein is configured to encode the output EL signal into an enhancement layer bitstream 224 based at least in part on inter-layer reference pictures obtained from the VDR signal 204 and/or the input VDR image.

In one embodiment, the VDR encoder 202 or the VDR RPU ( 210 ) therein is configured to encode the coding syntax as at least a portion of the RPU data into the VDR RPU bitstream 226 .

One or both of the base layer encoder (216) and the enhancement layer encoder (220) may be implemented using one or more of a variety of codecs such as H.264/MPEG-4 AVC, HEVC, MPEG-2, VP8, VC-1, and/or others.

A corresponding VDR decoder (which implements the inverse data flow shown in FIG. 2 and supports the same VDR specification used by the VDR encoder 202 to generate the encoding syntax) can be used to decode the bitstream generated by the third version of the VDR encoder 202 and generate a reconstructed version of the input VDR image.

Additionally, optionally, or alternatively, the techniques described herein may support VDR codecs (or encoding systems) corresponding to other versions of the VDR specification.

3.RPU data unit

In some embodiments, RPU data generated by an upstream device, such as a VDR encoder (e.g., 102 of FIG. 1 or 202 of FIG. 2 ), may be provided to a downstream device in a plurality of Network Abstraction Layer (NAL) data units. In one embodiment, as shown in FIG. 3 , a NAL data unit includes a NAL header and a Raw Byte Sequence Payload (RBSP). For illustrative purposes only, when the RBSP in the NAL data unit is used to encapsulate RPU data, the field "NAL-unit-type" in the NAL header may be set to 25 or other identification numbers different from those NAL types specified in the H.264/MPEG-4 AVC specification (IS 14496-10).

In some embodiments, the RPU data in the RBSP of the NAL data unit includes an RPU data header and an RPU data payload. The RPU data unit can be used as a common carrier for delivering RPU data from an upstream device to a downstream device, where the RPU data can be associated with any of a plurality of VDR specifications (e.g., different versions). The RPU data header may include header fields for identifying a codec or encoding system type (e.g., a 3D encoding system or a VDR encoder system) and a specific VDR specification among a plurality of different VDR specifications. The RPU data header may also include one or more high-level (e.g., sequence level or frame level) portions of the RPU data carried in the RPU data unit.

The RPU data payload can be used by an upstream device to send a descriptor (or syntactic description) of a set of flags, operations, and parameters that can be used to decode a multi-layer video signal and to reconstruct a VDR image using the decoded video signal to a downstream device. One or more flags, operations, and parameters described by the RPU data payload for reconstructing the VDR image can be related to inter-layer prediction. The flags, operations, and parameters used for inter-layer prediction described herein can be related to one or more of inverse mapping, chroma upsampling, and other functions such as display management. Additionally, optionally, or alternatively, one or more flags, operations, and parameters described by the RPU data payload for reconstructing the VDR image can be related to data processing attached to, or even in addition to, inter-layer prediction.

FIG4 illustrates the layout of an RPU data header in an example embodiment. In one embodiment, the RPU data header includes multiple header fields. For illustrative purposes only, the header fields may include, but are not limited to, any of the following: "rpu_type," "rpu_format," "vdr_rpu_profile," "vdr_rpu_level," "vdr sequence level information," "vdr frame level information," and the like.

The header field "rpu_type" can be used to identify whether the RPU data is related to a 3D codec (e.g., when rpu_type = 0 or 1) or a VDR codec (e.g., when rpu_type = 2). The header field "rpu_type" can be used to accommodate additional new video codecs yet to be developed. The header field "rpu_format" can be used to identify one or more VDR versions to which the RPU data is related. For illustrative purposes only, the most significant bits of the header field "rpu_format" can be used to distinguish major differences in VDR codecs, while the least significant bits of the same field can be used to distinguish minor changes in VDR codecs. For example, when the most significant bits (e.g., the top 3) of the header field "rpu_format" are 0, the RPU data is related to the VDR version 1.x process; on the other hand, when the most significant bits (e.g., the top 3) of the header field "rpu_format" are 1, the RPU data is related to the VDR version 2.0 process.

The VDR encoding system described herein may support one or more different RPU attributes. The header field "vdr_rpu_profile" may be used to identify the attributes to which the RPU data is associated. For example, a header field value of 0 indicates a baseline attribute specifying the mapped color space YCbCr, the mapped chroma format 4:2:0, the polynomial mapping method, and the globally unique mapped partition; a header field value of 1 indicates a primary attribute specifying all mapped color spaces, all mapped chroma formats, all mapping methods, and the globally unique mapped partition; and a header field value of 2 indicates a high attribute specifying all mapped color spaces, all mapped chroma formats, all mapping methods, and a locally mappable partition (global partition or local partition). In some embodiments, other possible values of the header field "rpu_profile" may be reserved for use by new attributes being developed or yet to be developed. The header field "rpu_level" may be used additionally and/or optionally to further distinguish the level of complexity of the RPU processing performed using the RPU data.

According to the techniques described herein, a coding syntax comprising one or more syntax elements conforming to the VDR specification may be sent/signaled by a VDR encoder to a VDR decoder in an RPU bitstream. Syntax elements may specify flags, operations, and parameters used in a VDR encoding operation and a corresponding VDR decoding operation. Parameters represented in syntax elements may have different coefficient types and may be specified as logical values, integer (fixed point) values, or floating point values with different precisions, lengths, or word lengths, etc.

Some syntax elements in the coding syntax can be classified as sequence-level information that remains unchanged for consecutive pictures in the entire sequence. Although it should be noted that the same syntax element can be used as a sequence-level or different levels in various coding syntaxes, examples of sequence-level information include, but are not limited to, any of the following syntax elements: "chroma_sample_loc_type", "vdr_color_primaries", "vdr_chroma_format_idc", etc. As shown in Figure 4, the sequence-level information is placed in the header field "vdr sequence level information", which can be a complex field and further includes a flag vdr_seq_info_present_flag to indicate whether any specific sequence-level information is encoded directly using one or more current RPU data units or whether the sequence-level information is predicted based on previous RPU data.

In some embodiments, for reasons of transmission efficiency, sequence-level information may not be sent by the VDR encoder to the VDR decoder for each picture (which may be interchangeably referred to as a frame in the context of this specification). Instead, the sequence-level parameters may be sent once for each consecutive frame of the sequence. However, for reasons of random access, error correction, and robustness, embodiments of the present invention do not exclude repeating the sequence-level parameters once, twice, etc. within the same sequence. In one example, in a sequence comprising 100 consecutive pictures, the sequence-level parameters may be repeated within the sequence after slices of 10 frames, 25 frames, 50 frames, etc. In another example, the sequence-level parameters may be repeated within the sequence for each instantaneous decoding refresh (IDR) picture, every two IDR pictures, etc.

Some syntax elements in the coding syntax can be classified as frame-level information that remains constant for the entire frame. In some embodiments, as shown in FIG4 , the frame-level information is placed in the header field "vdr frame-level information". In some embodiments, some or all of the frame-level information can be predicted based on the frame-level syntax elements sent in the previous RPU data unit.

For example, the inter-layer prediction coefficients may be the same or similar for a group of pictures (GOP), a scene, a sequence of frames, etc. Therefore, it is not necessary to repeat the frame-level information for each frame. For one or more current RPU data units having the same RPU identifier (Id), the VDR encoder may signal the RPU ID (or identifier) in the RPU data field "vdr_rpu_id" of the RPU data unit to the VDR decoder to indicate that the frame-level information is to be encoded directly in the current RPU data unit (therefore, the frame-level information can be directly retrieved without referring to previous RPU data identified by a different RPU ID).

In some embodiments, a flag "use_prev_vdr_rpu_flag" can be set in one or more current RPU data units to indicate to the VDR decoder that frame-level information in one or more previously transmitted RPU data units should be reused or used to predict frame-level information related to the one or more current RPU data units. The previously transmitted RPU data unit can be identified in the RPU data field "prev_vdr_rpu_id" in one or more current RPU data units. Thus, the transmission of predictable frame-level syntax elements in one or more current RPU data units can be avoided. In some embodiments, since the current RPU data unit does not directly carry coded frame-level syntax elements, the allocation of an RPU ID for the current RPU data unit can also be avoided. A maximum number of RPU IDs and their corresponding frame-level syntax elements can be used for prediction based on the cost-benefit trade-off between reduced data volume in bitstream transmission and increased memory usage at the VDR decoder.

In some embodiments, the techniques described herein support partitioning an image into one or more parts.Some syntax elements that may be used to specify coding syntax may be categorized as frame-level syntax elements, while some other syntax elements may be categorized as partition-level syntax elements.

4. RPU data decoding—sequence level and/or frame level

FIG5 illustrates an RPU decoding (or parsing) process that may be used to decode (or parse) sequence-level and/or frame-level syntax elements from an RPU data unit. The RPU decoding/parsing process may be configured to receive one or more current RPU data units and obtain at least some syntax elements from one or more RPU data headers of the one or more current RPU data units. First, the RPU decoding/parsing process may determine whether a flag "use_prev_vdr_rpu_flag" is present and, if so, what the value of the flag (syntax element) is.

If it is determined that the flag is set to 1 (or "yes"), the RPU decoding/parsing process proceeds to decode or parse the syntax element "prev_vdr_rpu_id" from the received RPU data unit, which points to the predictor (or previous) RPU ID associated with the previously transmitted syntax elements in one or more previously transmitted RPU data units. Based on the predictor RPU ID, the RPU decoding/parsing process can use the predictor RPU ID as a key to retrieve the previously transmitted syntax elements from the specified RPU data cache. The processing path with the "use_prev_vdr_flag" set to 1 (or "yes") can be used to predict some or all of the sequence-level, frame-level, and partition-level syntax elements of the coding syntax based on the previously transmitted syntax elements of the same level from the RPU data cache.

On the other hand, if it is determined that the flag "use_prev_vdr_rpu_flag" is 0 (or "No"), the RPU decoding/parsing process proceeds to decode or parse the syntax element "vdr_rpu_id" from the received RPU data unit, which is set to the current RPU ID assigned to the syntax element directly encoded in one or more current RPU data units. For purposes of illustration only, these syntax elements may include, but are not limited to, mapping_color_space, mapping_chroma_idc, chroma_resampling_filter_idc, num_pivots_minus2, pred_pivot_value[][], nlq_method_idx, nlq_num_pivots_minus2, nlq_pred_pivot_value[][], enable_residual_spatial_upsampling_flag, num_x_partition_minus1, num_y_partition_minus1, residual_resampling_filter_idc, overlapped_prediction_method, and the like. It should be noted that in various embodiments, different syntax elements and/or different names of syntax elements may be defined or used to implement the techniques described herein.

One or more of these syntax elements may be flags indicating the presence or absence of certain corresponding operations. For example, the flag "use_prev_vdr_rpu_flag" indicates the presence or absence of an operation to predict RPU data based on the cached syntax elements of the previous RPU ID. Similarly, the flag "enable_residual_spatial_upsampling_flag" may indicate whether a residual resampling filtering operation should be performed when reconstructing a VDR image based on the received BL data and EL data. The indicator "chroma_resampling_filter_idc" may indicate which chroma resampling filter should be used when reconstructing a VDR image based on the received BL data and EL data. In the RPU decoding/parsing process itself, each of these flags may also be used to determine whether a specific processing path should be taken.

RPU type and version information (which may indicate, for example, whether the RPU data is associated with the v1.x flow of the corresponding VDR specification and/or whether the VDR specification implements no residual EL data) may be used to determine some processing paths in the RPU decoding/parsing process shown in FIG. 5 .

Parameters such as the syntax elements "num_x_partition_minus 1" and "num_y_partition_minus 1" can also be used to determine some processing paths in the RPU decoding/parsing process. For example, if both syntax elements have a zero value, indicating a globally unique partition, then a processing path corresponding to the globally unique partition can be taken. On the other hand, as shown in FIG5 , if either or both of these syntax elements have a non-zero value, then a different processing path can be taken.

5. RPU data decoding - partition level

In some embodiments, partition-level syntax elements may be sent from a VDR encoder to a VDR decoder in one or more current RPU data units (e.g., one or more RPU payloads). FIG6 illustrates an RPU data payload decoding/parsing process that may be used to decode partition-level syntax elements from one or more current RPU data units in an exemplary embodiment. One or more partition-level syntax elements may be associated with coding syntax or may be used in coding syntax to specify inter-layer prediction related operations and/or other processing operations.

In some embodiments, the RPU data payload decoding/parsing process of FIG. 6 may be implemented as a function “vdr_rpu_data_payload()”, which may be called by the RPU decoding/parsing process of FIG. 5 , for example.

In some embodiments, several steps of FIG. 6 are repeated for each partition iterated along the x- and y-directions of the image frame. As shown in FIG. 6 , the function "rpu_data_mapping(X, Y)" may first be called for each partition to decode partition-level syntax elements common to multiple different VDR specifications. Subsequently, syntax elements more specific to a particular VDR specification may be decoded. The more specific syntax elements may be decoded based on other syntax elements or, for example, RPU information already decoded from one or more current RPU data units. For example, based on (1) the “rpu_format” field and (2) the syntax element “mapping_chroma_idc” (of the VDR specification of version number v1.x) or the syntax elements “vdr_chroma_format_idc” and “sdr_chroma_format_idc” (of the VDR specification of version number v2.x) decoded from one or more RPU data headers of one or more current RPU data units, the RPU data payload decoding/parsing process of Figure 6 can determine whether a chroma resampling operation should be performed on the received BL data and EL data that conform to the VDR specification of version “v1.x” or “v2.x”.

If the VDR specification is determined to be version v1.x and the flag "disable_residual_flag" is false, the RPU decoding/parsing function "rpu_data_nlq(x,y)" may be called for each partition. In addition, as shown in FIG6 , when both the x-partition index and the y-partition index are zero, other decoding/parsing functions such as the frame-level RPU decoding/parsing function "rpu_data_residual_resampling(x,y)" may also be called.

In some embodiments, the VDR specification implemented by the VDR encoding system supports chroma resampling, inverse mapping, prediction-based operations, including but not limited to: prediction based on overlapping areas, residual nonlinear quantization/dequantization, residual chroma resampling, spatial scaling, data processing (e.g., interpolation of partitioned boundary areas), etc.

For chroma resampling, the VDR specification may support both fixed filters and explicit 1D (2D separable) and 2D (non-separable) filters, or filters that use additional luminance or chroma channel information (cross-channel resampling filters). The syntax element "chroma_resampling_filter_idc" may be used to specify which of these filters is used as part of the encoding syntax. According to the techniques described herein, different chroma channels may use different filters. Additionally, optionally, or alternatively, the explicit filters may be symmetric or asymmetric. In some embodiments, one or more of the filters described herein may be designed to treat picture boundaries (image boundaries) as special cases in the filtering operation. For example, a filter may be simply padded to a picture boundary by repeating (as shown in FIG. 6 ) or mirroring. In some embodiments, one or more of the filters described herein may be designed to operate across different partitions, or to treat partition boundaries as picture boundaries in the same manner. Different chroma resampling filters may be used for different partitions of the same image. Additionally, optionally, or alternatively, a filter may be applied to all partitions of the complete image. Additionally, optionally, or alternatively, a particular type of filter, such as an explicit filter, may be specified for the entire image; however, different coefficients may be specified in the coding syntax for different partitions. For example, the syntax element "chroma_resampling_filter_idc" may be signaled at the frame level to indicate that a particular type of filter is used for the entire frame; however, different filter coefficients for different partitions within a frame may be signaled at the partition level. Additionally, optionally, or alternatively, the partition-level filter coefficients may be encoded directly, predicted from one or more previous partitions based on the current RPU ID, or predicted from one or more partitions in one or more RPU data units based on a previous RPUID. The coefficients described herein may be non-differentially encoded or differentially encoded (e.g., including differential values for values for different partitions, images, or chroma channels). The chroma resampling filter should also take the chroma sampling position into account.

For VDR layered codecs, inverse mapping can play an important role. The VDR specification described herein can support various inverse mapping methods. Examples of inverse mapping methods include, but are not limited to, any of the following: bit-shift, polynomial, MMR, SOP, 1D LUT, curve fitting, etc. The decoding/parsing function "rpu_data_mapping()" shown in Figure 8 can be used to decode (or parse) syntax elements related to the inverse mapping of each color component (luminance component or chrominance component) in the specified color space. The syntax element "syntax mapping_idc" can be used to indicate which inverse mapping method has been selected. Since different areas of an image can contain different visual content, the VDR specification can allow different partitions (e.g., different areas) to use different mapping methods. The dynamic range of each channel of an image can be divided into different segments (or blocks) and each dynamic range segment can use a different mapping method. In addition, each dynamic range in each different partition can use a different mapping method. This approach can be used for inverse mapping, where the intermediate dynamic range of the media content of the image is linear and can be processed using a linear mapping, while the dark and bright ranges are nonlinear and should be processed using a relatively complex mapping approach. In one embodiment, the syntax pivot_value is used to indicate the dynamic range segment. Additionally, optionally, or alternatively, the mapping pivot value can be differentially encoded (e.g., as indicated by the syntax element "pred_pivot_value") or directly encoded in a coding syntax that is signaled to a downstream VDR decoder for one or more current RPU data units.

In some embodiments, at least one of the plurality of partitions of the image may utilize a plurality of different dynamic range segments. In some embodiments, the maximum number of different dynamic range segments across all partitions of the image is determined or set below a limit. In some embodiments, while the dynamic ranges in different partitions may optionally differ, all partitions of the image maintain the same number of different dynamic range segments.

For linear mapping, the polynomial coefficients may be signaled to a downstream VDR decoder by the VDR encoder using one or more syntax elements in the encoding syntax. Alternatively, one or more syntax elements in the encoding syntax may be used to signal mapped pivot values for interpolating pixels in each dynamic range segment. The presence of the mapped pivot values may be signaled using a syntax element (or flag) "linear_interp_flag." In one example, some or all values of the data points in the 1D LUT may be signaled. In another example, at least some values in the 1D LUT may be created based on interpolation using the mapped pivot values signaled to the downstream VDR decoder.

The coefficients for a dynamic range map (e.g., tone map) for a partition may be encoded directly or, alternatively, predicted based on a mapped dynamic range segment in an adjacent partition obtained from the same RPU data unit. Additionally, alternatively, or alternatively, the coefficients for a partition may be predicted based on a mapped dynamic range segment in a partition of a mapped block obtained from a previous RPU data unit. For example, the coefficients for a partition may be predicted based on a mapped dynamic range segment in the same partition obtained from a previous RPU data unit.

The techniques described herein support the use of a mapping color space (e.g., indicated by the syntax element "mapping_color_space") that is different from the encoding color space (which can be signaled in sequence-level information, such as in the RPU data header). For example, the encoding color space can be YCbCr, while the mapping color space can be RGB. Other types of color spaces can be used as a choice of encoding space or mapping space. The mapping color space can be different for different partitions. Alternatively, the mapping color space can be the same for all partitions. The mapping method and metadata can be different for different channels of the mapping color space. Alternatively, the mapping method and metadata can be the same for all channels of the mapping color space. In embodiments where multiple partitions are used in an image, there may be discontinuities along partition boundaries. In one embodiment, the encoding syntax can be used to signal a boundary mapping method, which is performed by simply smoothing partition boundaries by weighted averaging of pixel values or color values and/or by blending partition boundaries using linear or nonlinear methods. In one embodiment, the syntax element "overlapped_prediction_method" in the encoding syntax can be used, at least in part, to signal the boundary mapping method.

6. RPU data decoding - chroma mapping

FIG7 illustrates an RPU data decoding (or parsing) operation (e.g., in the form of an rpu_data_chroma_resampling() function) that may be used to decode (or parse) syntax elements related to chroma resampling in an example embodiment. These syntax elements may, but are not necessarily, present at the partition level. The RPU data encoding operation may be implemented as a decode/parse function that may be called, for example, by the decode/parse process of FIG6 . In embodiments where the mapped color space is the same for the entire image, the syntax elements shown in FIG7 may optionally be present as frame-level syntax elements in the coding syntax.

In some embodiments, the multiple segments involved in the segment mapping operation may remain the same for all partitions.Some syntax elements related to interpolation may be presented in the coding syntax as frame-level syntax elements, while other syntax elements related to interpolation may be presented in the coding syntax as partition-level syntax elements.

As shown in Figure 7, the "rpu_data_chroma_resampling()" decoding/parsing function can decode multiple color components in the color space. For each color component, several steps can be repeated.

If the flag of the color component indicates to use the previous partition filter coefficients, the decoding/parsing function "rpu_data_chroma_resampling()" proceeds to obtain the predictor partition information of the color component. The predictor partition information may include partition filter coefficients obtained from the cached syntax elements of the previous RPU ID, or partition filter coefficients obtained from already decoded syntax elements of one or more other partitions of one or more current RPU data units.

On the other hand, if the flag for the color component indicates that the previous partition filter coefficients are not used, the decoding/parsing function "rpu_data_chroma_resampling()" proceeds to obtain the partition filter coefficients from one or more current RPU data units. These coefficients can be related to 2D explicit filters, 1D vertical explicit filters, 1D horizontal explicit filters, etc.

As with the coefficients used in dynamic range mapping, coefficients in chroma resampling or chroma mapping for a partition may alternatively be predicted based on similar coefficients in adjacent partitions obtained from the same one or more current RPU data units. Additionally, alternatively, or alternatively, coefficients for a partition may be predicted based on similar coefficients in partitions of a mapping block obtained from one or more previously transmitted RPU data units. For example, coefficients for a partition may be predicted based on similar coefficients in the same partition obtained from one or more previously transmitted RPU data units.

7. Additional Examples of RPU Data Decoding

FIG9 illustrates RPU data decoding (or parsing) operations (e.g., in the form of the rpu_data_nlq() function) for decoding (or parsing) syntax elements related to nonlinear quantization/dequantization at the partition level in an example embodiment. Specific VDR specifications may support nonlinear quantization/dequantization. Examples of nonlinear quantization/dequantization may include, but are not limited to, those based on linear blind spots, μ-law curves, Laplacian curves, S-shaped curves, and the like. The specific method of nonlinear quantization/dequantization may be signaled to the downstream VDR decoder in the syntax element "nlq_method_idc." In one embodiment, the same method may be used for all partitions of a picture (e.g., the syntax element "nlq_method_idc" may be signaled as part of the frame-level information in the RPU data header); however, the coefficients of the method may be the same or different for different partitions. The data range involved in nonlinear quantization/dequantization may be divided into multiple segments; different segments may have different coefficients for the same method.

Like other coefficients used in other operations, the coefficients in the nonlinear quantization/dequantization of a partition can be directly encoded or predicted based on the same coefficients in adjacent partitions obtained from the same RPU data unit. Additionally, optionally, or alternatively, the coefficients of the partition can be predicted based on similar coefficients in a partition of a mapped block obtained from a previous RPU data unit. For example, the coefficients of the partition can be predicted based on similar coefficients in the same partition obtained from a previous RPU data unit.

In some embodiments, the encoding syntax conforming to a particular VDR specification may specify chroma resampling and/or spatial upsampling (e.g., 1:2) to be performed on the residual data. In some embodiments, the operations performed on the residual data are handled in the encoding syntax in a manner similar to the operations related to the chroma resampling filter discussed above.

In some embodiments, different chroma formats are used for image data encoded in the BL signal and the EL signal. For example, the BL signal may use a different chroma format, different chroma sampling, and different bit depth than that used by the EL signal. Additionally, optionally, or alternatively, the BL signal and the EL signal may use different color spaces.

The techniques described herein support different processing orders in chroma resampling, color space conversion, and inverse mapping. In some embodiments, the VDR encoding system may support one, two, or more of a plurality of possible processing orders. One or more processing orders supported by the VDR encoding system may be considered optimal. For example, the encoding color space of both the BL data and the EL data in the output bitstream (e.g., BL bitstream 228 and EL bitstream 224 of FIG. 2 ) of the VDR encoder (e.g., 202 of FIG. 2 ) is YCbCr as specified by the VDR specification; the mapping color space may be RGB; the input SDR signal (e.g., 208 of FIG. 2 ) is YCbCr 4:2:0; the input VDR signal (e.g., 204 of FIG. 2 ) is RGB 4:4:4 12-bit. In this example, inter-layer reference data may be generated as follows. First, the VDR encoder performs chroma resampling from 4:2:0 to 4:4:4 on the BL data obtained from the input SDR signal. Next, a color transform from YCbCr to RGB can be performed on the BL data (now in a 4:4:4 chroma format). An inverse mapping can be performed on the BL data (now in a 4:4:4 chroma format and in the same mapped color space as in the mapped color space) in the mapped color space to generate inter-layer prediction values in the mapped color space. For the purpose of generating EL data, a color transform from RGB to YCbCr can be performed on the inter-layer prediction values.

8. Example Processing Flow

FIG10 illustrates a VDR decoder that decodes encoding syntax based on RPU data in an example embodiment. The encoding syntax may conform to a specific VDR specification, which may be, for example, version 1 (“1.0”) or version 2 (“1.x”) supported by the VDR encoder 102 of FIG1 . The VDR decoder may be configured to perform decoding operations on BL data, EL data, RPU data, inter-layer prediction data, and intermediate media data based on the encoding syntax. The VDR decoder of FIG10 may be implemented using one or more computing devices, custom and/or off-the-shelf hardware devices, programmable devices, any combination of the foregoing, and the like.

In some embodiments, the VDR decoder of FIG10 may implement one or more of the decoding/parsing processes shown in FIG5 to FIG9 to obtain the encoding syntax and the syntax elements therein. The VDR decoder of FIG10 may apply decoding operations to the BL data, EL data, and RPU data to construct an output VDR image corresponding to an input VDR image encoded, for example, by a VDR encoder (e.g., 102 of FIG1 ).

FIG11A illustrates an example process flow according to an example embodiment of the present invention. In some example embodiments, one or more computing devices or components may perform the process flow. In block 1102, a multi-layer VDR video encoder (e.g., 102 of FIG1 or 202 of FIG2 ) receives an input visual dynamic range (VDR) image and an input base layer (BL) image associated with the input VDR image.

In block 1104 , the multi-layer VDR video encoder generates a coding syntax comprising a plurality of syntax elements at a sequence level, a frame level, or a partition level.

In block 1106 , the multi-layer VDR video encoder converts the input BL image and the input VDR image into BL data and enhancement layer (EL) data according to a coding syntax.

In block 1108, the multi-layer VDR video encoder converts the coding syntax into reference processing unit (RPU) data.

In block 1110 , the multi-layer VDR video encoder outputs BL data, EL data, and RPU data as a BL signal, an EL signal, and an RPU signal.

In one embodiment, the multi-layer VDR video encoder is further configured to: generate one or more current RPU data units based at least in part on the encoding syntax; and determine a specific VDR specification to which the encoding syntax complies in the one or more current RPU data units.

In one embodiment, at least one of the one or more current RPU data units includes a data structure capable of supporting any one of a plurality of different VDR specifications.

In one embodiment, the multi-layer VDR video encoder is further configured to perform: indicating, in one or more current RPU data units, at least one syntax element of a plurality of syntax elements in a coding syntax that is predictable from one or more other partitions in the one or more current RPU data units.

In one embodiment, the multi-layer VDR video encoder is further configured to perform: indicating, in one or more current RPU data units, at least one syntax element of a plurality of syntax elements in a coding syntax that is predictable based on a previous input VDR image and one or more previous input BL images associated with the previous input VDR image.

In one embodiment, the input VDR picture and the previous input VDR picture belong to a sequence of input VDR pictures; the sequence of input VDR pictures shares a common set of syntax elements at sequence level.

In one embodiment, the input VDR image and the previous input VDR image belong to two different input VDR image sequences; a first sequence of the two different input VDR image sequences shares a first common set of sequence-level grammatical elements; and a second sequence of the two different input VDR image sequences shares a different second common set of sequence-level grammatical elements.

In one embodiment, at least one syntax element of the plurality of syntax elements may be used as a syntax element at two or more of the sequence level, the frame level, or the partition level.

In one embodiment, the BL data represents a standard dynamic range (SDR) image optimized for viewing on an SDR display. In one embodiment, the BL data does not represent a standard dynamic range (SDR) image optimized for viewing on an SDR display.

In one embodiment, the EL data comprises a residual value between an input VDR image and a predicted VDR image generated based on the BL data. In one embodiment, the EL data comprises an inter-layer reference picture of two or more input VDR images in a sequence of input VDR images; the two or more input VDR images comprise the input VDR image.

In one embodiment, the plurality of syntax elements include one or more parameters, coefficients, pivot values, flags indicating the presence or absence of an operation corresponding to the flag, or one or more types of metadata including display management metadata.

In one embodiment, the input VDR image includes image data encoded in an input color space; the EL data includes image data encoded in an output color space; the EL data is generated at least in part based on the mapping data; the mapping data is generated at least in part based on the BL data; and the mapping data includes mapped image data encoded in the mapped color space.

In one embodiment, at least two of the input color space, the output color space, and the mapped color space are different. In one embodiment, at least two of the input color space, the output color space, and the mapped color space are the same.

In one embodiment, the EL data includes image data encoded in a first chroma format and the BL data includes image data encoded in a second, different chroma format. In one embodiment, the EL data includes image data encoded in a chroma format and the BL data includes image data encoded in the same chroma format.

In one embodiment, the plurality of syntax elements signal one or more of the following operations: a chroma resampling operation, an inverse mapping operation, a non-overlapping region based prediction operation, an overlapping region based prediction operation, a residual non-linear quantization and dequantization operation, a residual chroma resampling operation, a spatial scaling operation, a data processing operation including interpolation, or a display management operation.

In one embodiment, the multi-layer VDR video encoder is further configured to perform: converting one or more input VDR images represented, received, transmitted or stored using one or more input video signals into one or more output VDR images represented, received, transmitted or stored using one or more output video signals.

In one embodiment, the input VDR image includes image data encoded in one of the following: a high dynamic range (HDR) image format, an RGB color space associated with the Academy Color Encoding Specification (ACES) standard of the Academy of Motion Picture Arts and Sciences (AMPAS), the P3 color space standard of the Digital Cinema Initiatives Alliance, the Reference Input Media Metrics/Reference Output Media Metrics (RIMM/ROMM) standards, an sRGB color space, an RGB color space, or a YCbCr color space.

FIG11B illustrates an example processing flow according to an example embodiment of the present invention. In some example embodiments, one or more computing devices or hardware components may perform the processing flow. In block 1152, a multi-layer video decoder (e.g., as shown in FIG10 ) receives base layer (BL) data, enhancement layer (EL) data, and reference processing unit (RPU) data as BL signals, EL data, and RPU data associated with a common visual dynamic range (VDR) source image.

In block 1154 , the multi-layer video decoder decodes the RPU data into coding syntax including a plurality of syntax elements at a sequence level, a frame level, or a partition level.

In block 1156 , the multi-layer video decoder converts the BL data and the EL data into a reconstructed VDR image according to the coding syntax.

In block 1158, the multi-layer video decoder outputs a reconstructed VDR image.

In one embodiment, the multi-layer video decoder is further configured to: determine a specific VDR specification to which the coding syntax complies from one or more current RPU data units; and obtain at least a portion of the coding syntax from the one or more current RPU data units.

In one embodiment, the multi-layer video decoder is further configured to perform: determining, from one or more current RPU data units, at least one syntax element of multiple syntax elements in the coding syntax that can be predicted based on one or more other partitions in the one or more current RPU data units.

In one embodiment, the multi-layer video decoder is further configured to perform: determining, from one or more current RPU data units, at least one syntax element of a plurality of syntax elements in a coding syntax that can be predicted based on one or more previous RPU data units related to a previously reconstructed VDR image.

In one embodiment, the reconstructed VDR image and the previously reconstructed VDR image belong to a sequence of reconstructed VDR images; the sequence of reconstructed VDR images shares a common set of syntax elements at the sequence level.

In one embodiment, the reconstructed VDR image and the previously reconstructed VDR image belong to two different reconstructed VDR image sequences; a first sequence in the two different reconstructed VDR image sequences shares a first common set of sequence-level grammatical elements; and a second sequence in the two different reconstructed VDR image sequences shares a different second common set of sequence-level grammatical elements.

In one embodiment, the EL data comprises inter-layer reference pictures of two or more reconstructed VDR images in a sequence of reconstructed VDR images, and the two or more reconstructed VDR images comprise the reconstructed VDR image.

In one embodiment, the reconstructed VDR image includes image data encoded in a first color space; the EL data includes image data encoded in a second color space; the reconstructed VDR image is generated at least in part based on mapping data obtained from the BL data; and the mapping data includes mapped image data encoded in a third color space.

In one embodiment, at least two of the first color space, the second color space, and the third color space are different. In one embodiment, at least two of the first color space, the second color space, and the third color space are the same.

In one embodiment, the multi-layer video decoder is further configured to perform: converting image data represented, received, transmitted or stored using one or more input video signals into one or more output VDR images represented, received, transmitted or stored using one or more output video signals.

In one embodiment, the reconstructed VDR image includes image data encoded in one of the following: a high dynamic range (HDR) image format, an RGB color space associated with the Academy Color Encoding Specification (ACES) standard of the Academy of Motion Picture Arts and Sciences (AMPAS), the P3 color space standard of the Digital Cinema Initiatives Alliance, the Reference Input Media Metrics/Reference Output Media Metrics (RIMM/ROMM) standards, an sRGB color space, an RGB color space, or a YCbCr color space.

In various exemplary embodiments, an encoder, a decoder, a system, an apparatus, or one or more other computing devices performs any or part of the methods described above.

9. Implementation Mechanism - Hardware Overview

According to one embodiment, the technology described herein can be realized by one or more special-purpose computing devices.Special-purpose computing devices can be hard-wired to perform technology, or can include digital electronic devices such as permanently being programmed to perform one or more application-specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs) of the present technology, or can include being programmed to perform one or more general-purpose hardware processors of the present technology according to the program instructions in firmware, memory, other storage devices or combination.This special-purpose computing device can also complete these technologies with customized hard-wired logic, ASIC or FPGA and customized programming combination.Special-purpose computing devices can be desktop computer systems, portable computer systems, handheld devices, network equipment or any other equipment that merges hard wiring and/or program logic to realize the present technology.

For example, Figure 12 is a block diagram illustrating a computer system 1200 upon which example embodiments of the present invention may be implemented. Computer system 1200 includes a bus 1202 or other communication mechanism for communicating information, and a hardware processor 1204 coupled with bus 1202 for processing information. Hardware processor 1204 may be, for example, a general-purpose microprocessor.

The computer system 1200 also includes a main memory 1206, such as a random access memory (RAM) or other dynamic storage device, coupled to the bus 1202 for storing information and instructions to be executed by the processor 1204. The main memory 1206 may also be used for storing temporary variables or other intermediate information during the execution of instructions to be executed by the processor 1204. Such instructions, when stored in a non-transitory storage medium accessible by the processor 1204, cause the computer system 1200 to appear as a special-purpose machine customized to perform the operations specified in the instructions.

Computer system 1200 also includes a read only memory (ROM) 1208 or other static storage device coupled to bus 1202 for storing static information and instructions for processor 1204. A storage device 1210, such as a magnetic or optical disk, is provided and coupled to bus 1202 for storing information and instructions.

The computer system 1200 may be coupled to a display 1212, such as a liquid crystal display, via the bus 1202 for displaying information to a computer user. An input device 1214, including alphanumeric and other keys, is coupled to the bus 1202 for communicating information and command selections to the processor 1204. Another type of user input device is a cursor control 1216, such as a mouse, trackball, or cursor direction keys, for communicating directional information and command selections to the processor 1204 and for controlling cursor movement on the display 1212. The input device typically has two degrees of freedom along two axes—a first axis (e.g., x) and a second axis (e.g., y)—that enables the device to specify a position in a plane.

Computer system 1200 can implement the techniques described herein using custom hard-wired logic, one or more ASICs or FPGAs, firmware and/or program logic that, in conjunction with the computer system, enables computer system 1200 or programs computer system 1200 as a special-purpose machine. According to one embodiment, computer system 1200 performs the techniques described herein in response to processor 1204 executing one or more sequences of one or more instructions contained in main memory 1206. These instructions may be read into main memory 1206 from other storage media, such as storage device 1210. Execution of the sequences of instructions contained in main memory 1206 causes processor 1204 to perform the process steps described herein. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions.

As used herein, the term "storage media" refers to any non-transitory medium that stores data and/or instructions that cause a machine to operate in a specific manner. Such storage media may include non-volatile media and/or volatile media. Non-volatile media include, for example, optical or magnetic disks such as storage device 1210. Volatile media include dynamic memory such as main memory 1206. Common forms of storage media include, for example, floppy disks, diskettes, hard disks, solid-state drives, magnetic tape, or any other magnetic data storage medium, CD-ROMs, any other optical data storage medium, any physical medium with a pattern of holes, RAM, PROM and EPROM, Flash EPROM, NVRAM, any other memory chip, or a cassette tape.

Storage media are distinct from, but can be used in conjunction with, transmission media. Transmission media participate in the transfer of information between storage media. For example, transmission media include coaxial cables, copper wire, and optical fiber, including the wires that comprise bus 1202. Transmission media can also take the form of sound or light waves, such as those generated during radio wave and infrared data communications.

Various forms of media may be involved in carrying one or more sequences of one or more instructions to the processor 1204 for execution. For example, the instructions may initially be carried on a disk or solid-state drive of a remote computer. The remote computer may load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. The modem of the local computer system 1200 may receive the data on the telephone line and convert the data into an infrared signal using an infrared transmitter. An infrared detector may receive the data carried in the infrared signal, and appropriate circuitry may place the data on the bus 1202. The bus 1202 carries the data to the main memory 1206, from which the processor 1204 retrieves and executes the instructions. Alternatively, the instructions received by the main memory 1206 may be stored on the storage device 1210 before or after being executed by the processor 1204.

Computer system 1200 also includes a communication interface 1218 coupled to bus 1202. Communication interface 1218 provides two-way data communication coupled to network link 1220, which is connected to local network 1222. For example, communication interface 1218 can be an integrated services digital network (ISDN) card, a cable modem, a satellite modem, or a modem for providing a data communication connection to a corresponding type of telephone line. As another example, communication interface 1218 can be a LAN card that provides a data communication connection to a compatible local area network (LAN). Wireless links can also be implemented. In any such implementation, communication interface 1218 sends and receives electrical, electromagnetic, or optical signals that carry digital data streams representing various types of information.

The network link 1220 typically provides data communication through one or more networks to other data devices. For example, the network link 1220 can provide a connection through the local network 1222 to a host computer 1224 or to data equipment operated by an Internet Service Provider (ISP) 1226. The ISP 1226, in turn, provides data communication services through the global packet data communication network now commonly referred to as the "Internet" 1228. Both the local network 1222 and the Internet 1228 use electrical, electromagnetic, or optical signals that carry digital data streams. The signals through the various networks and the signals on the network link 1220 and through the communication interface 1218 that carry the digital data to and from the computer system 1200 are example forms of transmission media.

Computer system 1200 can send messages and receive data, including program code, through the network, network link 1220, and communication interface 1218. In the Internet example, server 1230 can send the requested code for an application program through Internet 1228, ISP 1226, local network 1222, and communication interface 1218.

When received, the received code may be executed by processor 1204 and/or stored in storage device 1210 or other non-volatile storage for later execution.

10. Equivalents, Extensions, Substitutions and Other Matters

In the foregoing specification, example embodiments of the present invention have been described with reference to numerous specific details that vary from implementation to implementation. Therefore, the sole and exclusive indicator of what the invention is, and what the applicants intend it to be, is the set of claims that issue from this application, in the specific form in which the claims issue, including any subsequent amendments. Any definitions expressly set forth herein for terms included in the claims may govern the meaning of those terms as used in the claims. Therefore, no limitation, element, attribute, feature, advantage, or characteristic that is not expressly set forth in a claim should limit the scope of the claim in any way. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense.

In addition, the present disclosure also includes:

(1) A method comprising:

receiving an input visual dynamic range (VDR) image and an input base layer (BL) image associated with the input VDR image;

generating a coding syntax comprising a plurality of syntax elements at at least one of a sequence level, a frame level, or a partition level;

converting the input BL image and the input VDR image into BL data and enhancement layer (EL) data according to the coding syntax;

converting the coding syntax elements into reference processing unit (RPU) data; and

The BL data, the EL data, and the RPU data are output as a BL signal, an EL signal, and an RPU signal.

(2) The method according to (1), further comprising:

generating one or more current RPU data units based at least in part on the encoding syntax; and

A specific VDR specification to which the encoding syntax complies is determined in the one or more current RPU data units.

(3) The method according to (2), wherein at least one of the one or more current RPU data units includes a data structure capable of supporting any one of a plurality of different VDR specifications.

(4) The method according to (2) further includes: in the one or more current RPU data units, indicating at least one syntax element of the multiple syntax elements of the encoding syntax as capable of being predicted based on one or more other partitions of the one or more current RPU data units.

(5) The method according to (2) further includes: in the one or more current RPU data units, indicating at least one syntax element of the multiple syntax elements in the encoding syntax as being capable of being predicted based on one or more previous RPU data units of a previous input VDR image and a previous input BL image associated with the previous input VDR image.

(6) The method according to (5), wherein the input VDR image and the previous input VDR image belong to a sequence of input VDR images, and wherein the sequence of input VDR images shares a common set of sequence-level syntax elements.

(7) A method according to (5), wherein the input VDR image and the previous input VDR image belong to two different input VDR image sequences; wherein a first sequence of the two different input VDR image sequences shares a first common set of sequence-level grammatical elements; and wherein a second sequence of the two different input VDR image sequences shares a different second common set of sequence-level grammatical elements.

(8) The method according to (1), wherein at least one of the plurality of syntax elements can be used as a syntax element at two or more of a sequence level, a frame level, or a partition level.

(9) The method of (1), wherein the BL data represents a standard dynamic range (SDR) image optimized for viewing on an SDR display.

(10) The method of (1), wherein the BL data does not represent a standard dynamic range (SDR) image optimized for viewing on an SDR display.

(11) The method according to (1), wherein the EL data includes a residual value between the input VDR image and a predicted VDR image generated based on the BL data.

(12) The method according to (1), wherein the EL data includes inter-layer reference pictures of two or more input VDR images in an input VDR image sequence, and wherein the two or more input VDR images include the input VDR image.

(13) A method according to (1), wherein the multiple syntax elements include one or more of the following items: a parameter; a coefficient; a main element value; a flag indicating the presence or absence of an operation corresponding to the flag; or one or more types of metadata including display management metadata.

(14) A method according to (1), wherein the input VDR image includes image data encoded in an input color space, wherein the EL data includes image data encoded in an output color space, wherein the EL data is generated at least in part based on mapping data, wherein the mapping data is generated at least in part based on the BL data, and wherein the mapping data includes mapped image data encoded in a mapped color space.

(15) The method of (14), wherein at least two of the input color space, the output color space, the mapped color space, or the encoded color space are different.

(16) The method of (14), wherein at least two of the input color space, the output color space, the mapped color space, or the encoded color space are the same.

(17) The method of (1), wherein the EL data includes image data encoded in a first chroma format, and wherein the BL data includes image data encoded in a second, different chroma format.

(18) The method of (1), wherein the EL data includes image data encoded in a chroma format, and wherein the BL includes image data encoded in the same chroma format.

(19) A method according to (1), wherein the multiple syntax elements specify one or more of the following operations: a chroma resampling operation, an inverse mapping operation, a non-overlapping area based prediction operation, an overlapping area based prediction operation, a residual nonlinear quantization and dequantization operation, a residual chroma resampling operation, a residual spatial upsampling operation, a data processing operation including interpolation, or a display management operation.

(20) The method according to (1) further includes: converting one or more input VDR images represented, received, sent or stored using one or more input video signals into one or more output VDR images represented, received, sent or stored using one or more output video signals.

(21) The method of (1), wherein the input VDR image comprises image data encoded in one of the following formats: a high dynamic range (HDR) image format, an RGB color space associated with the Academy Color Encoding Specification (ACES) standard of the Academy of Motion Picture Arts and Sciences (AMPAS), the P3 color space standard of the Digital Cinema Initiatives Alliance, the Reference Input Media Metrics/Reference Output Media Metrics (RIMM/ROMM) standard, an sRGB color space, an RGB color space, or a YCbCr color space.

(22) A method comprising:

receiving, as a base layer (BL) signal, an enhancement layer (EL) signal, and a reference processing unit (RPU) signal, BL data, EL data, and RPU data associated with a common visual dynamic range (VDR) source image;

decoding the RPU data into a coding syntax comprising a plurality of syntax elements at at least one of a sequence level, a frame level, or a partition level;

converting the BL data and the EL data into a reconstructed VDR image according to the coding syntax; and

The reconstructed VDR image is output.

(23) The method according to (22), further comprising:

determining, based on one or more current RPU data units, a particular VDR specification to which the encoding syntax complies; and

At least a portion of the encoding syntax element is derived from the one or more current RPU data units.

(24) The method according to (23), wherein at least one of the one or more current RPU data units includes a data structure capable of supporting any one of a plurality of different VDR specifications.

(25) The method according to (23) further includes: determining, based on the one or more current RPU data units, at least one syntax element of the multiple syntax elements of the encoding syntax as being capable of being predicted based on one or more other partitions of the one or more current RPU data units.

(26) The method according to (23) further includes: determining, based on the one or more current RPU data units, at least one syntax element of the multiple syntax elements of the coding syntax as being capable of being predicted based on one or more previous RPU data units related to a previously reconstructed VDR image.

(27) The method of (26), wherein the reconstructed VDR image and the previously reconstructed VDR image belong to a sequence of reconstructed VDR images, and wherein the sequence of reconstructed VDR images shares a common set of sequence-level syntax elements.

(28) A method according to (26), wherein the reconstructed VDR image and the previously reconstructed VDR image belong to two different reconstructed VDR image sequences; wherein a first sequence in the two different reconstructed VDR image sequences shares a first common set of sequence-level syntactic elements; and wherein a second sequence in the two different reconstructed VDR image sequences shares a different second common set of sequence-level syntactic elements.

(29) The method according to (22), wherein at least one of the plurality of syntax elements can be used as a syntax element at two or more of a sequence level, a frame level, or a partition level.

(30) The method of (22), wherein the BL data represents a standard dynamic range (SDR) image optimized for viewing on an SDR display.

(31) The method of (22), wherein the BL data does not represent a standard dynamic range (SDR) image optimized for viewing on an SDR display.

(32) The method according to (22), wherein the EL data includes a residual value between a VDR image and a predicted VDR image generated based on the BL data.

(33) A method according to (22), wherein the EL data includes inter-layer reference pictures of two or more reconstructed VDR images in a sequence of reconstructed VDR images, and wherein the two or more reconstructed VDR images include the reconstructed VDR image.

(34) A method according to (22), wherein the multiple syntax elements include one or more of the following items: a parameter; a coefficient; a main element value; a flag indicating the presence or absence of an operation corresponding to the flag; or one or more types of metadata including display management metadata.

(35) A method according to (22), wherein the reconstructed VDR image includes image data encoded in a first color space, wherein the EL data includes image data encoded in a second color space, wherein the reconstructed VDR image is generated at least in part based on mapping data derived from the BL data, and wherein the mapping data includes mapped image data encoded in a third color space.

(36) The method according to (35), wherein at least two of the first color space, the second color space, and the third color space are different.

(37) The method according to (35), wherein at least two of the first color space, the second color space, and the third color space are the same.

(38) The method of (22), wherein the EL data includes image data encoded in a first chroma format, and wherein the BL data includes image data encoded in a second, different chroma format.

(39) The method according to (22), wherein the EL data includes image data encoded in a chroma format, and wherein the BL includes image data encoded in the same chroma format.

(40) A method according to (22), wherein the multiple syntax elements signal one or more of the following operations: chroma resampling operations, inverse mapping operations, non-overlapping area based prediction operations, overlapping area based prediction operations, residual nonlinear quantization and dequantization operations, residual chroma resampling operations, spatial scaling operations, data processing operations including interpolation, or display management operations.

(41) The method according to (22) also includes: converting image data represented, received, sent or stored using one or more input video signals into one or more output VDR images represented, received, sent or stored using one or more output video signals.

(42) The method of (22), wherein the reconstructed VDR image includes image data encoded in one of the following formats: a high dynamic range (HDR) image format, an RGB color space associated with the Academy Color Encoding Specification (ACES) standard of the Academy of Motion Picture Arts and Sciences (AMPAS), the P3 color space standard of the Digital Cinema Initiatives Alliance, the Reference Input Media Metrics/Reference Output Media Metrics (RIMM/ROMM) standard, an sRGB color space, an RGB color space, or a YCbCr color space.

(43) An encoder that performs any of the methods described in (1) to (21).

(44) A decoder that performs any of the methods described in (22) to (42).

(45) A system for performing any of the methods described in (1) to (42).

(46) A system comprising:

An encoder configured to:

receiving an input visual dynamic range (VDR) image and an input base layer (BL) image associated with the input VDR image;

generating a coding syntax comprising a plurality of syntax elements at a sequence level, a frame level, or a partition level;

converting the input VDR image and the input BL image into BL data and enhancement layer (EL) data according to the coding syntax;

converting the coding syntax into reference processing unit (RPU) data; and

outputting the BL data, the EL data and the RPU data as a BL signal, an EL signal and an RPU signal,

A decoder configured to:

receiving the BL data, the EL data, and the RPU data using the BL signal, the EL signal, and the RPU signal;

decoding the RPU data into the coding syntax comprising the plurality of syntax elements at a sequence level, a frame level, or a partition level;

converting the BL data and the EL data into a reconstructed VDR image according to the coding syntax; and

The reconstructed VDR image is output.

(47) A non-transitory computer-readable medium storing software instructions that, when executed by one or more processors, cause the steps of any of the methods described in (1) to (42) to be performed.

Claims (13)

1. A decoder for generating high dynamic range images, wherein the decoder includes non-transitory memory and one or more processors, wherein generating an output image using the decoder includes: Receive reference processing unit (RPU) data and store at least a portion of the RPU data in the non-transitory memory; At least the RPU data header and the RPU data payload are extracted from the RPU data; Receive the base layer image; Receive the enhancement layer image; Determine whether the decoder can use syntax elements from the previous RPU data payload; When it is determined that the decoder cannot use syntax elements from the previous RPU data payload, the RPU data payload is parsed to extract syntax elements. When it is determined that the decoder can use syntax elements from the previous RPU data payload, the previous RPU ID value is decoded to predict the syntax elements, wherein the syntax elements include the current RPU ID value, a mapped color space indicator, a chroma resampling filter indicator, principal mapping parameters, and partitioning parameters; and The base layer image and the enhancement layer image are combined based on the syntax elements to generate the output image. The parsing of the RPU data payload includes: Decode the current RPUId value; Decode the mapped color space indicator; Decode the chroma resampling filter indicator; Decode the principal mapping parameters; and The partition parameters are decoded. 2. The decoder according to claim 1, wherein the syntax elements include inter-layer prediction data and nonlinear quantizer (NLQ) data. 3. The decoder of claim 1, wherein determining whether the decoder can use syntax elements from a previous RPU data payload includes parsing the use_prev_vdr_rpu_flag flag, which, when set to 1, indicates that the syntax element is derived from a previously received RPU data payload, and when set to 0, indicates that the syntax element is explicitly derived. 4. The decoder according to claim 2, wherein the inter-layer prediction data is extracted according to the rpu_data_mapping() syntax structure, and the nonlinear quantizer NLQ data is extracted according to the rpu_data_nlq() syntax structure. 5. The decoder according to claim 4, wherein the rpu_data_mapping() syntax structure further includes: The dynamic range of the output image is divided into multiple principal component indices for segments, where, For one or more primary indices, the rpu_data_mapping() syntax structure includes the corresponding mapping index indicating the inverse mapping method; and wherein, For one or more mapping methods, the rpu_data_mapping() syntax structure includes the corresponding mapping parameters. 6. The decoder according to claim 5, wherein the inverse mapping method includes one of the following: a polynomial prediction method, an MMR prediction method, or a lookup table prediction method. 7. The decoder according to claim 4, wherein the rpu_data_nlq() syntax structure further includes: The dynamic range of the output image is divided into multiple principal component indices for segments, where, For one or more primary indices, the rpu_data_nlq() syntax structure includes the corresponding non-linear quantization method; and For one or more nonlinear quantization methods, the rpu_data_nlq() syntax structure includes the corresponding nonlinear quantization parameters. 8. The decoder according to claim 7, wherein the nonlinear quantization method includes one of the following: linear dead zone method, μ-law method, Laplace operator method, or S-shaped method. 9. The decoder according to claim 8, wherein, for the linear dead-time method, the rpu_data_nlq() syntax structure further includes the decoder output level value and the decoder residual input maximum value. 10. The decoder according to claim 1, wherein the RPU data header comprises: An RPU type flag indicating the encoding type, wherein the encoding type includes either 3D image encoding or high dynamic range image encoding; An RPU format field indicating the version format of the RPU data payload; RPU attribute fields; and RPU level field. 11. The decoder of claim 10, wherein the RPU type flag includes an rpu_type flag, and when rpu_type = 2, the encoding type includes high dynamic range image encoding. 12. A method for generating a high dynamic range image using a decoder, wherein generating an output image using the decoder comprises: Receive reference processing unit (RPU) data and store at least a portion of the RPU data in a non-transitory memory; At least the RPU data header and the RPU data payload are extracted from the RPU data; Receive the base layer image; Receive the enhancement layer image; Determine whether the decoder can use syntax elements from the previous RPU data payload; When it is determined that the decoder cannot use syntax elements from the previous RPU data payload, the RPU data payload is parsed to extract syntax elements. When it is determined that the decoder can use syntax elements from the previous RPU data payload, the previous RPU ID value is decoded to predict the syntax elements, wherein the syntax elements include the current RPU ID value, a mapped color space indicator, a chroma resampling filter indicator, principal mapping parameters, and partitioning parameters; and The base layer image and the enhancement layer image are combined based on the syntax elements to generate the output image. The parsing of the RPU data payload includes: Decode the current RPUId value; Decode the mapped color space indicator; Decode the chroma resampling filter indicator; Decode the principal mapping parameters; and The partition parameters are decoded. 13. A non-transitory computer-readable storage medium storing a computer program that can be executed by a processor to perform the following steps: Receive reference processing unit (RPU) data and store at least a portion of the RPU data in a non-transitory memory; At least the RPU data header and the RPU data payload are extracted from the RPU data; Receive the base layer image; Receive the enhancement layer image; Determine whether the decoder can use syntax elements from the previous RPU data payload; When it is determined that the decoder cannot use syntax elements from the previous RPU data payload, the RPU data payload is parsed to extract syntax elements. When it is determined that the decoder can use syntax elements from the previous RPU data payload, the previous RPU ID value is decoded to predict the syntax elements, wherein the syntax elements include the current RPU ID value, a mapped color space indicator, a chroma resampling filter indicator, principal mapping parameters, and partitioning parameters; and The base layer image and the enhancement layer image are combined based on the syntax elements to generate the output image. The parsing of the RPU data payload includes: Decode the current RPUId value; Decode the mapped color space indicator; Decode the chroma resampling filter indicator; Decode the principal mapping parameters; and The partition parameters are decoded.