RU2830974C1 - Signalling oversampling of reference image with indication of constant window size in video bit stream - Google Patents

Abstract

FIELD: video encoding and decoding.

SUBSTANCE: invention relates to encoding and decoding video, and more specifically to signalling the size of an image or parts of an image, which can vary from image to image. Disclosed is a method of decoding an encoded video bitstream, which includes: obtaining a first flag indicating whether a constant image size is used in an encoded video sequence including a current image; based on a first flag indicating that a constant image size is used, decoding the current image without performing resampling of the reference image; based on the first flag indicating that the constant size of the image is not used, obtaining a second flag indicating whether the size of the matching window is signalled; based on the second flag indicating that the matching window size is signalled, obtaining the matching window size, determining the resampling coefficient between the current image and the reference image based on the size of the matching window and performing resampling of the reference image on the current image using the resampling coefficient. First flag is signalled in the sequence parameter set (SPS) and indicates whether the SPS matching window parameters are signalled in the SPS.

EFFECT: high efficiency of encoding and decoding a video bit stream.

17 cl, 10 dwg, 1 tbl

Description

CROSS-REFERENCE TO RELATED APPLICATION

[1] This application claims the benefit of U.S. Provisional Patent Application No. 62/903,601, filed September 20, 2019, and U.S. Provisional Patent Application No. 17/009,979, filed September 2, 2020, which are incorporated herein in their entireties.

Field of technology to which the invention relates

[1] The disclosed subject matter relates to video encoding and decoding, and more particularly to signaling the size of an image or portions of an image that may vary from image to image or from portion of an image to portion of an image.

State of the art

[2] Video encoding and decoding using inter-frame prediction with motion compensation is known. Uncompressed digital video may consist of a series of images, each image having a spatial dimension of, for example, 1920 x 1080 luma samples and associated chroma samples. The series of images may have a fixed or variable picture rate (also known informally as a frame rate), for example, 60 images per second or 60 Hz. Uncompressed video has significant bitrate requirements. For example, 1080p60 4:2:0 video at 8 bits per sample (1920x1080 luma sample resolution at 60 Hz frame rate) requires a bandwidth of close to 1.5 Gbps. An hour of such video requires over 600 GB of disk space.

[3] One of the goals of video encoding and decoding can be to reduce the redundancy of the input video signal through compression. Compression can help reduce the above-mentioned bandwidth or storage space requirements, in some cases by two orders of magnitude or more. Both lossless and lossy compression, as well as a combination of both, can be used. Lossless compression refers to techniques in which an exact copy of the original signal can be reconstructed from the compressed source signal. With lossy compression, the reconstructed signal may not be identical to the original signal, but the distortion between the original and reconstructed signals is small enough to make the reconstructed signal suitable for the intended application. In the case of video, lossy compression is widely used. The degree of distortion tolerable depends on the application; for example, higher distortion can be tolerated by users of certain consumer streaming applications than by users of broadcast television applications. The achievable compression ratio can reflect the following: a higher allowable/tolerable distortion can lead to a higher compression ratio.

[4] A video encoder and video decoder may use techniques from several broad categories, including, for example, motion compensation, transform, quantization, and entropy coding, some of which will be presented below.

[5] Historically, video encoders and decoders have tended to operate with a given picture size, which in most cases was defined and remained constant for a coded video sequence (CVS), group of pictures (GOP), or similar multi-picture time frame. For example, in MPEG-2, the system design is known to vary the horizontal resolution (and hence the picture size) depending on factors such as scene activity, but only for I-pictures, and hence typically for GOPs. Resampling of reference pictures to use different resolutions in CVS is known, for example, from ITU-T Rec. H.263, Annex P. However, here the picture size is not changed, only the reference pictures are resampled, with the result that potentially only parts of the picture canvas may be used (in the case of downsampling) or only parts of the scene may be captured (in the case of upsampling). Furthermore, H.263 Annex Q allows resampling of an individual macroblock by a factor of two (in each dimension), up or down. Again, the image size remains the same. The macroblock size is fixed in H.263 and therefore does not need to be transmitted.

[6] Changing the image size of predicted images has become more common in modern video coding. For example, VP9 allows resampling the reference image and changing the resolution of the entire image. Similarly, some proposals made for VVC (including, e.g., Hendry, et. al., “On adaptive resolution change (ARC) for VVC,” JVT paper JVET-M0135-v1, January 9-19, 2019, incorporated herein in its entirety) allow resampling entire reference images to a different - higher or lower - resolution. This paper proposes to encode different resolution candidates in a sequence parameter set and reference them using syntax elements for each image in the picture parameter set.

Disclosure of the essence of the invention

[7] In one embodiment, a method is provided for decoding an encoded video bitstream using at least one processor, wherein the method includes receiving a first flag indicating whether a constant picture size is used in an encoded video sequence including a current picture; based on the first flag indicating that a constant picture size is used, decoding the current picture without performing resampling of the reference picture; based on the first flag indicating that a constant picture size is not used, receiving a second flag indicating whether a correspondence window size is signaled; based on the second flag indicating that the correspondence window size is signaled, receiving the correspondence window size, determining an resampling factor between the current picture and the reference picture based on the correspondence window size, and performing resampling of the reference picture on the current picture using the resampling factor.

[8] In one embodiment, a device is provided for decoding an encoded video bitstream, wherein the device includes at least one memory element configured to store program code; and at least one processor configured to read the program code and operate in accordance with instructions of the program code, wherein the program code includes: a first acquisition code configured to cause at least one processor to receive a first flag indicating whether a constant picture size is used in an encoded video sequence including a current picture; a decoding code configured to cause at least one processor, based on the first flag indicating that a constant picture size is used, to decode the current picture without performing resampling of the reference picture; a second acquisition code configured to cause at least one processor, based on the first flag indicating that a constant picture size is not used, to receive a second flag indicating whether a match window size is signaled; and execution code configured to cause at least one processor, based on a second flag indicating that a match window size is signaled, to obtain a match window size, determine an oversampling factor between the current image and the reference image based on the match window size, and oversample the reference image on the current image using the oversampling factor.

[9] In one embodiment, a non-transitory computer-readable medium is provided that stores instructions including: one or more instructions that, when executed by one or more processors of a device for decoding an encoded video bitstream, cause the one or more processors to: receive a first flag indicating whether a constant picture size is used in an encoded video sequence including a current picture; based on the first flag indicating that a constant picture size is used, decode the current picture without performing resampling of the reference picture; based on the first flag indicating that a constant picture size is not used, receive a second flag indicating whether a match window size is signaled; based on the second flag indicating that the match window size is signaled, receive the match window size, determine an oversampling factor between the current picture and the reference picture based on the match window size, and perform resampling of the reference picture on the current picture using the oversampling factor.

Brief description of the drawings

[10] Additional features, nature and various advantages of the disclosed subject matter will be more apparent from the following detailed description and the accompanying drawings, in which:

[11] Fig. 1 shows a schematic illustration of a simplified block diagram of a communication system according to an embodiment.

[12] Fig. 2 shows a schematic illustration of a simplified block diagram of a communication system according to an embodiment.

[13] Fig. 3 shows a schematic illustration of a simplified block diagram of a decoder according to an embodiment.

[14] Fig. 4 shows a schematic illustration of a simplified block diagram of an encoder according to an embodiment.

[15] Fig. 5 shows a schematic illustration of ARC/RPR parameter signaling options according to an embodiment.

[16] Figs. 6A-6B show schematic illustrations of examples of syntax tables according to an embodiment.

[17] Fig. 7 shows a schematic illustration of transmitting an image size and a matching window in an SPS in accordance with embodiments.

[18] Fig. 8 shows a schematic illustration of transmitting an image size and a matching window in a PPS in accordance with embodiments.

[19] Fig. 9 shows a flow chart of an exemplary process for decoding an encoded video bitstream according to an embodiment.

[20] Fig. 10 shows a schematic illustration of a computer system according to an embodiment.

Implementation of the invention

[21] Fig. 1 illustrates a simplified block diagram of a communication system (100) according to an embodiment of the present disclosure. The system (100) may include at least two terminals (110 and 120) connected to each other via a network (150). For unidirectional data transmission, a first terminal (110) may encode video data at a local location for transmission to another terminal (120) via the network (150). A second terminal (120) may receive the encoded video data of the other terminal from the network (150), decode the encoded data, and display the reconstructed video data. Unidirectional data transmission may be common in multimedia service applications and the like.

[22] Fig. 1 shows a second pair of terminals (130, 140) that support bidirectional transmission of encoded video, which may occur, for example, during a video conference. For bidirectional data transmission, each terminal (130, 140) may encode video data captured at a local location for transmission to the other terminal via a network (150). Each terminal (130, 140) may also receive encoded video data transmitted by the other terminal, may decode the encoded data, and may display the reconstructed video data on a local display device.

[23] In Fig. 1, the terminals (110-140) may be depicted as servers, personal computers and smartphones, but the principles of the present disclosure may not be limited thereto. Embodiments of the present disclosure find application in laptop computers, tablet computers, media players and/or specialized videoconferencing equipment. The network (150) represents any number of networks that transmit encoded video data between the terminals (110-140), including, for example, wired and/or wireless communication networks. The communication network (150) may exchange data in circuit-switched and/or packet-switched channels. Representative networks include telecommunication networks, local area networks, wide area networks and/or the Internet. For the purposes of the present discussion, the architecture and topology of the network (150) may be immaterial to the operation of the present disclosure unless explained below herein.

[24] Fig. 2 illustrates, as an example of application for the disclosed subject matter, the placement of a video encoder and a video decoder in a streaming environment. The disclosed subject matter may be equally applicable to other video-enabled applications, including, for example, videoconferencing, digital television, storage of compressed video on digital media, including CD, DVD, memory card, etc., and so on.

[25] The streaming system may include a capture subsystem (213), which may include a video source (201), such as a digital camera, creating, for example, a stream (202) of uncompressed video samples. This sample stream (202), depicted with a thick line to emphasize the large amount of data compared to encoded video bitstreams, may be processed by an encoder (203) connected to the camera (201). The encoder (203) may include hardware, software, or a combination of both to include or implement aspects of the disclosed subject matter, as described in more detail below. The encoded video bitstream (204), depicted with a thin line to emphasize the smaller amount of data compared to the sample stream, may be stored on the streaming server (205) for future use. One or more streaming clients (206, 208) may access a streaming server (205) to retrieve copies (207, 209) of an encoded video bitstream (204). The client (206) may include a video decoder (210) that decodes an incoming copy of the encoded video bitstream (207) and produces an outgoing stream (211) of video samples that may be displayed on a display (212) or other visualization device (not shown). In some streaming systems, the video bitstreams (204, 207, 209) may be encoded in accordance with certain video encoding/compression standards. Examples of these standards include ITU-T Recommendation H.265. A video encoding standard informally known as universal video coding or VVC is under development. The disclosed subject matter may be used in the context of VVC.

[26] Fig. 3 may be a functional block diagram of a video decoder (210) according to an embodiment of the present disclosure.

[27] The receiver (310) may receive one or more coded video sequences to be decoded by the decoder (210); in the same or another embodiment, one coded video sequence at a time, where the decoding of each coded video sequence is independent of the other coded video sequences. The coded video sequence may be received from a channel (312), which may be a hardware/software connection to a storage device in which the coded video data is stored. The receiver (310) may receive the coded video data with other data, such as coded audio data and/or auxiliary data streams, which may be sent to their respective using entities (not shown). The receiver (310) may separate the coded video sequence from the other data. To combat jitter in the network, a buffer memory (315) can be connected between the receiver (310) and the entropy decoder/parser (320) (hereinafter referred to as the "parser"). When the receiver (310) receives data from a storage/forwarding device with sufficient bandwidth and controllability or from an isosynchronous network, the buffer memory (315) may not be needed or may be small. For use in high-quality packet networks, such as the Internet, a buffer (315) may be needed, which may be relatively large and may preferably be of an adaptive size.

[28] The video decoder (210) may include a parser (320) for recovering symbols (321) from the entropy encoded video sequence. Categories of these symbols include information used to control the operation of the decoder (210) and potentially information for controlling a rendering device, such as a display (212), which is not an integral part of the decoder but may be connected to it, as was shown in Fig. 3. The control information for the rendering device(s) may be in the form of additional enhancement information (SEI messages) or fragments of a video suitability information (VUI) parameter set (not shown). The parser (320) may parse/entropy decode the received encoded video sequence. The coding of the coded video sequence may be performed in accordance with a video coding technology or standard and may follow principles well known to those skilled in the art, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, and so on. The parser (320) may extract from the coded video sequence a set of subgroup parameters for at least one of the pixel subgroups in the video decoder based on at least one parameter corresponding to the group. The subgroups may include groups of pictures (GOPs), pictures, tiles, slices, bricks, macroblocks, coding tree units (CTUs), coding units (CUs), blocks, transform units (TUs), prediction units (PUs), and so on. A tile may indicate a rectangular area of CUs/CTUs in a specific column and row of tiles in a picture. A brick may indicate a rectangular area of CUs/CTU rows within a specific tile. A slice may refer to one or more image bricks contained in a NAL unit. A sub-image may refer to a rectangular region of one or more slices in an image. The entropy decoder/parser may also extract information from the encoded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, and so on.

[29] The parser (320) may perform an entropy decoding/parsing operation on the video sequence received from the buffer (315) to produce symbols (321).

[30] The symbol recovery (321) may include a plurality of different modules depending on the type of the encoded video image or its parts (such as: outer and inner image, outer and inner block) and other factors. Which modules are involved and how can be controlled using subgroup control information that has been parsed from the encoded video sequence using the parser (320). The flow of such subgroup control information between the parser (320) and the plurality of modules below is not shown for clarity.

[31] In addition to the functional blocks already mentioned, the decoder 210 may be conceptually subdivided into a number of functional modules, as described below. In a practical implementation operating under commercial constraints, many of these modules closely interact with each other and may be, at least partially, integrated into each other. However, for the purposes of describing the disclosed subject matter, the conceptual subdivision into functional modules given below is appropriate.

[32] The first module is the scaling/inverse transform module (351). The scaling/inverse transform module (351) receives the quantized transform coefficient, as well as control information, including which transform to use, the block size, the quantization coefficient, the quantization scaling matrices, and so on, in the form of symbol(s) (321) from the parser (320). It can output blocks containing sample values, which can be input to the aggregator (355).

[33] In some cases, the output samples of the scaling/inverse transform module (351) may relate to an intra-coded block; that is, to a block that does not use prediction information from previously reconstructed images, but may use prediction information from previously reconstructed portions of the current image. Such prediction information may be provided by the intra-prediction module (352). In some cases, the intra-prediction module (352) generates a block of the same size and shape as the block being reconstructed, using surrounding already reconstructed information extracted from the current (partially reconstructed) image (358). The aggregator (355), in some cases, adds for each sample the prediction information generated by the intra-prediction module (352) to the information of the output samples provided by the scaling/inverse transform module (351).

[34] In other cases, the output samples of the scaling/inverse transform module (351) may relate to an inter-coded and potentially motion-compensated block. In such a case, the motion-compensated prediction module (353) may access the reference picture memory (357) to retrieve the samples used for prediction. After motion compensation of the selected samples in accordance with the symbols (321) related to the block, these samples may be added by the aggregator (355) to the output of the scaling/inverse transform module (here called residual samples or residual signal) to generate information about the output samples. The addresses in the reference picture memory from which the motion-compensated prediction module selects the prediction samples may be controlled by motion vectors available to the motion-compensated prediction module in the form of symbols (321), which may have, for example, X, Y and reference picture components. Motion compensation may also involve interpolation of sample values retrieved from reference image memory, when using precise sub-sample motion vectors, motion vector prediction mechanisms, and so on.

[35] The output samples of the aggregator (355) may be subjected to various in-loop filtering methods in the in-loop filter module (356). The video compression technologies may include in-loop filtering technologies that are controlled by parameters included in the bitstream of the encoded video and provided to the in-loop filter module (356) as symbols (321) from the parser (320), but may also respond to metainformation obtained during decoding of previous (in decoding order) portions of the encoded image or encoded video sequence, as well as respond to previously reconstructed and filtered by in-loop filtering sample values.

[36] The output signal of the loop filter module (356) may be a stream of samples that may be output to the visualization device (212) and also stored in a reference image memory for use in future inter-frame prediction.

[37] Certain coded images after full reconstruction may be used as reference images for future prediction. After a coded image is fully reconstructed and the coded image has been identified as a reference image (e.g., by the parser (320)), the current reference image (358) may become part of the reference image buffer (357), and fresh memory of the current images may be reallocated before starting to reconstruct the next coded image.

[38] The video decoder 210 may perform decoding operations according to a predetermined video compression technology, which may be documented in a standard, such as ITU-T Rec. H.265. The coded video sequence may conform to the syntax defined by the video compression technology or standard used, in the sense that it conforms to the syntax of the video compression technology or standard as specified in the video compression technology document or standard, and in particular in their profile document. Also, for conformance, it may be necessary that the complexity of the coded video sequence be within the limits defined by the level of the video compression technology or standard. In some cases, the levels limit the maximum image size, the maximum frame rate, the maximum reconstruction sampling rate (measured, for example, in megasamples per second), the maximum size of the reference image, and so on. The limits set by the levels may in some cases be further limited by the specifications of a hypothetical reference decoder (HRD) and metadata for managing the HRD buffer transmitted in the coded video sequence.

[39] In an embodiment, the receiver (310) may receive additional (redundant) data with the encoded video. The additional data may be included as part of the encoded video sequence(s). The additional data may be used by the video decoder (210) to correctly decode the data and/or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant images, forward error correction codes, and so on.

[40] Fig. 4 may be a functional block diagram of a video encoder (203) according to an embodiment of the present disclosure.

[41] The encoder (203) may receive video samples from a video source (201) (which is not part of the encoder), which may capture video image(s) for encoding by the encoder (203).

[42] The video source (201) may provide a source video sequence for encoding by the encoder (203) in the form of a digital stream of video samples, which may have any suitable bit depth (e.g.: 8 bits, 10 bits, 12 bits, ...), any color space (e.g. BT.601 Y CrCB, RGB, ...) and any suitable sample structure (e.g. Y CrCb 4:2:0, Y CrCb 4:4:4). In a multimedia serving system, the video source (201) may be a storage device storing pre-prepared video. In a videoconferencing system, the video source (203) may be a camera that captures local image information in the form of a video sequence. The video data may be provided as a plurality of individual images that convey motion when viewed sequentially. The images themselves may be organized as a spatial array of pixels, where each pixel may contain one or more samples, depending on the sample structure used, the color space, etc. A person skilled in the art can easily understand the relationship between pixels and samples. The description below is focused on samples.

[43] According to an embodiment, the encoder (203) can encode and compress the images of the original video sequence into the encoded video sequence (443) in real time or with any other time constraints, as required by the application. Ensuring an appropriate encoding rate is one of the functions of the controller (450). The controller controls other functional modules, as described below, and is functionally connected to these modules. The connection is not shown for clarity. The parameters set by the controller can include parameters related to rate control (picture skip, quantizer, lambda value of rate-distortion optimization methods, ...), image size, group of pictures (GOP) layout, maximum motion vector search range, and so on. A person skilled in the art can easily determine other functions of the controller (450), as they can relate to a video encoder (203) optimized for a particular system design.

[44] Some video encoders operate in what one skilled in the art will readily recognize as a "coding loop". As a simplified description, the coding loop may consist of a coding portion of the encoder (430) (hereinafter "source encoder") (responsible for generating symbols based on the input picture to be encoded and the reference picture(s)), and a (local) decoder (433) embedded in the encoder (203) that reconstructs the symbols to generate sample data that a (remote) decoder can also generate (since any compression between the symbols and the encoded video bitstream is lossless in the video compression technologies discussed in the disclosed subject matter). This reconstructed sample stream is input into a reference picture memory (434). Since decoding the symbol stream produces bit-accurate results regardless of the location of the decoder (local or remote), the contents of the reference picture buffer are also bit-accurate between the local encoder and the remote encoder. In other words, the prediction part of the encoder "sees" as reference picture samples exactly the same sample values that the decoder would "see" if the prediction was used during decoding. This fundamental principle of reference picture synchronicity (and the resulting drift if synchronicity cannot be maintained, e.g. due to channel errors) is well known to those skilled in the art.

[45] The operation of the "local" decoder (433) may be the same as that of the "remote" decoder (210), which has already been described in detail above in the description of Fig. 3. Referring briefly also to Fig. 4, however, since the symbols are available and the encoding/decoding of the symbols into the coded video sequence by the entropy encoder (445) and the parser (320) can be performed without loss, the entropy decoding parts of the decoder (210), including the channel (312), the receiver (310), the buffer (315) and the parser (320), cannot be completely implemented in the local decoder (433).

[46] At this point, it can be observed that any decoding technology other than parsing/entropy decoding that is present in the decoder must also necessarily be present in a substantially identical functional form in the corresponding encoder. For this reason, the disclosed subject matter focuses on the operation of the decoder. The description of the encoding technologies can be shortened, since they are the inverse of the fully described decoding technologies. Only in some areas is a more detailed description required, which is given below.

[47] As part of its operation, the source encoder (430) may perform motion-compensated predictive coding, which encodes an input frame with prediction by reference to one or more previously encoded frames from a video sequence, which have been designated as "reference frames." In this way, the coding engine (432) encodes differences between blocks of pixels of the input frame and blocks of pixels of the reference frame(s), which may be selected as the prediction reference(s) for the input frame.

[48] The local video decoder (433) may decode the encoded video data of frames, which may be designated as reference frames, based on the symbols generated by the source encoder (430). The operations of the encoding engine (432) may preferably be lossy processes. When the encoded video data may be decoded in the video decoder (not shown in Fig. 4), the reconstructed video sequence may typically be a copy of the original video sequence with some errors. The local video decoder (433) replicates the decoding processes that may be performed by the video decoder on the reference frames, and may cause the reconstructed reference frames to be stored in the reference picture cache (434). In this way, the encoder (203) may locally store copies of the reconstructed reference frames, which have common content, as reconstructed reference frames, which will be received by the video decoder at the far end (in the absence of transmission errors).

[49] The predictor (435) may perform predictive searches for the encoding engine (432). That is, for a new image to be encoded, the predictor (435) may search the reference image memory (434) for sample data (as candidates for blocks of reference pixels) or for certain metadata, such as motion vectors of reference images, block shapes, and so on, that may serve as suitable prediction references for new images. The predictor (435) may operate on a block-by-pixel sample basis to find appropriate prediction references. In some cases, as determined by the search results obtained by the predictor (435), the input image may have prediction references taken from a plurality of reference images stored in the reference image memory (434).

[50] The controller (450) may control the encoding operations of the video encoder (430), including, for example, setting parameters and subgroup parameters used to encode video data.

[51] The output signals of all the above-mentioned functional modules may be subjected to entropy encoding in an entropy encoder (445). The entropy encoder converts the symbols generated by the various functional modules into an encoded video sequence by lossless compression of the symbols according to technologies known to those skilled in the art, such as Huffman encoding, variable length encoding, arithmetic encoding, and so on.

[52] The transmitter (440) may buffer the encoded video sequence(s) created by the entropy encoder (445) to prepare it for transmission via the communication channel (460), which may be a hardware/software connection to a storage device that will store the encoded video data. The transmitter (440) may provide merging of the encoded video data from the video encoder (430) with other data to be transmitted, such as encoded audio data and/or auxiliary data streams (sources not shown).

[53] The controller (450) may control the operation of the encoder (203). During encoding, the controller (450) may assign each encoded image to a specific encoded image type, which may affect the encoding methods that may be applied to the corresponding image. For example, images may often be assigned to one of the following frame types:

[54] An intra-picture (I-picture) may be one that can be encoded and decoded without using any other frame in the sequence as a prediction source. Some video codecs allow different types of intra-pictures, including, for example, pictures with independent decoder updating. A person skilled in the art is familiar with these variants of I-pictures and their corresponding applications and features.

[55] A predictive image (P-image) may be one that can be encoded and decoded using intra prediction or inter prediction using at most one motion vector and a reference index to predict the sample values of each block.

[56] A bidirectionally predicted image (B-image) may be one that can be encoded and decoded using intra prediction or inter prediction using at most two motion vectors and reference indices to predict the sample values of each block. Similarly, multi-predicted images may use more than two reference images and associated metadata to reconstruct a single block.

[57] The source pictures may typically be spatially partitioned into a plurality of blocks of samples (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and coded on a block-by-block basis. The blocks may be predictively coded with reference to other (already coded) blocks, as determined by the coding assignment applied to the corresponding pictures of the blocks. For example, blocks of I-pictures may be coded without prediction, or they may be predictively coded with reference to already coded blocks of the same picture (spatial prediction or intra prediction). Pixel blocks of P-pictures may be coded without prediction, by means of spatial prediction, or by means of temporal prediction with reference to one previously coded reference picture. Blocks of B-pictures may be coded without prediction, by means of spatial prediction, or by means of temporal prediction with reference to one or two previously coded reference pictures.

[58] The video encoder (203) may perform encoding operations in accordance with a predetermined video encoding technology or standard, such as ITU-T Recommendation H.265. In its operation, the video encoder (203) may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancy in the input video sequence. The encoded video data may therefore conform to a syntax specified by the video encoding technology or standard used.

[59] In an embodiment, the transmitter (440) may transmit additional data with the encoded video. The video encoder (430) may include such data as part of the encoded video sequence. The additional data may comprise temporal/spatial/SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, supplemental enhancement information (SEI) messages, fragments of a video suitability information (VUI) parameter set, and so on.

[60] Recently, some attention has been drawn to compressed domain aggregation or extraction of multiple semantically independent image parts into a single video image. In particular, in the context of, for example, 360 coding or certain surveillance applications, multiple semantically independent source images (e.g., the surface of six cubes of a cube-projected 360 scene or individual camera inputs in the case of a multi-camera surveillance setup) may require separate adaptive resolution settings to cope with the different activity of each scene at a given time. In other words, encoders may choose to use different resampling factors for different semantically independent images that make up the entire 360-degree or surveillance scene at a given time. When combined into a single image, there is a need to perform resampling of the reference image and signal the availability of adaptive resolution coding for the coded image parts.

[61] Below are some terms that are referred to in the remainder of this description.

[62] A sub-picture may refer to, in some cases, a rectangular arrangement of samples, blocks, macroblocks, coding units, or similar objects that are semantically grouped and that can be independently coded at a different resolution. One or more sub-pictures may form a picture. One or more coded sub-pictures may form a coded picture. One or more sub-pictures may be assembled into a picture, and one or more sub-pictures may be extracted from the picture. In certain environments, one or more coded sub-pictures may be assembled in a compressed domain without transcoding to the sample level into a coded picture, and in the same or other cases, one or more coded sub-pictures may be extracted from the coded picture in the compressed domain.

[63] In the following, reference picture resampling (RPR) or adaptive resolution resampling (ARC) may refer to mechanisms that allow changing the resolution of a picture or sub-picture in a coded video sequence, for example by resampling a reference picture. In the following, RPR/ARC parameters refer to control information necessary to perform adaptive resolution resampling, which may include, for example, filter parameters, scaling factors, output and/or reference picture resolutions, various control flags, and so on.

[64] In embodiments, encoding and decoding may be performed for a single semantically independent encoded video image. Before describing the consequences of encoding/decoding multiple sub-images with independent RPR/ARC parameters and their implied additional complexity, options for transmitting RPR/ARC parameters should be described.

[65] FIG. 5 shows several embodiments of signaling RPR/ARC parameters. As noted for each of the embodiments, they may have certain advantages and certain disadvantages in terms of coding efficiency, complexity, and architecture. A video coding standard or technology may select one or more of these embodiments or embodiments known in the art for transmitting RPR/ARC parameters. The embodiments may not be mutually exclusive and may possibly be interchangeable depending on the needs of the application, the standards used, or the choice of encoder.

[66] RPR/ARC parameter classes may include:

[67] upsampling/downsampling factors, separate or combined in the X and Y dimensions

[68] upsampling/downsampling factors with added time dimension, indicating increase/decrease in size at a constant rate for a given number of images

[69] Either of the above two options may include encoding one or more presumably short syntax elements that may point to a table containing the coefficient(s).

[70] The resolution in the X or Y dimension in sample units, blocks, macroblocks, coding units (CUs), or any other suitable granularity of the input picture, output picture, reference picture, coded picture, in combination or separately. If there is more than one resolution (e.g. one for the input picture, one for the reference picture), then in some cases one set of values may be derived from the other set of values. This may be conveyed, for example, by using flags. See below for a more detailed example.

[71] "warp" coordinates similar to those used in Annex P of H.263, again with a suitable level of granularity as described above. Annex P of H.263 defines one efficient way of encoding such warp coordinates, but other, potentially more efficient ways may also be under development. For example, the variable-length reversible Huffman coding of the warp coordinates according to Annex P could be replaced by a binary coding of a suitable length, where the length of the binary codeword could, for example, be derived from the maximum image size, possibly multiplied by a certain factor and offset by a certain value to account for "warp" beyond the boundaries of the maximum image size.

[72] Upsampling and/or downsampling filter parameters. In embodiments, there may be only one upsampling and/or downsampling filter. However, in embodiments, it may be desirable to provide greater flexibility in the filter design, and this may require communicating the filter parameters. Such parameters may be selected using an index into a list of possible filter designs, the filter may be fully specified (e.g., using a list of filter coefficients using suitable entropy encoding techniques), the filter may be implicitly selected using appropriate upsampling/downsampling ratios, which in turn are communicated in accordance with any of the mechanisms mentioned above, and so on.

[73] The description below assumes encoding a finite set of upsampling/downsampling factors (the same factor that is to be used in the X and Y dimensions) specified by a codeword. This codeword may be encoded with variable length, for example using the Exponential Golomb code common to certain syntax elements in video coding specifications such as H.264 and H.265. One suitable mapping of values to upsampling/downsampling factors might, for example, correspond to Table 1:

TABLE 1

Code word Exponential Golomb code Source/Target Resolution 0 1 1/1 1 010 1/1.5 (50% increase) 2 011 1.5/1 (50% reduction) 3 00100 1/2 (100% increase) 4 00101 2/1 (100% reduction)

[74] Many similar mappings can be developed according to the needs of the application and the capabilities of the upsampling and downsampling mechanisms available in the video compression technology or standard. The table can be expanded to include more values. The values can also be represented by entropy coding mechanisms other than Exponential Golomb codes, for example using binary coding. This can have certain advantages when the resampling factors are of interest outside the video processing mechanisms themselves (primarily the encoder and decoder), for example by MANE. It should be noted that for situations where resolution changes are not required, the Exponential Golomb code can be chosen, which is short; in the table above it is only one bit. This can have an advantage in coding efficiency compared to using binary codes for the most common case.

[75] The number of entries in the table, as well as their semantics, may be fully or partially customizable. For example, the basic structure of the table may be conveyed in a "high" parameter set, such as a sequence or a decoder parameter set. In embodiments, one or more such tables may be defined in a video coding technology or standard and may be selected, for example, by a decoder or a sequence parameter set.

[76] Below is a description of how the upsampling/downsampling factor (ARC information), encoded as described above, may be incorporated into a video coding technology or standard syntax. Similar considerations may apply to one or more codewords controlling upsampling/downsampling filters. See below for discussion of when relatively large amounts of data are required for the filter or other data structures.

[77] As shown in Fig. 5, H.263 Annex P includes ARC information (502) in the form of four warp coordinates in the picture header (501), in particular in the H.263 PLUSPTYPE header extension (503). This may be a reasonable design choice when a) there is an available picture header and b) frequent changes in the ARC information are expected. However, the overhead of using H.263 style transmission may be quite high, and the scaling factors may not be relative to picture boundaries, since the picture header may be temporary.

[78] In the same or another embodiment, the passing of ARC parameters may correspond to a detailed example as shown in Figs. 6A-6B. Figs. 6A-6B depict syntax diagrams in a representation type using a notation that roughly corresponds to C-style programming, such as has been used in video coding standards since at least 1993. Lines in bold indicate syntax elements present in the bitstream, lines without bold often indicate control flow or variable setting.

[79] As shown in Fig. 6A, the tile group header (601) may conventionally contain, as an exemplary syntactic structure of a header applicable to a (possibly rectangular) portion of an image, an Exponential Golomb coded variable-length syntax element dec_pic_size_idx (602) (shown in bold). The presence of this syntax element in the tile group header may be constrained by the use of adaptive resolution (603) - here the flag value is not shown in bold, meaning that the flag is present in the bitstream at the point where it occurs in the syntax diagram. The fact whether adaptive resolution is used for this image or its portions may be conveyed in any high-level syntactic structure inside or outside the bitstream. In the example shown, it is conveyed in the sequence parameter set, as shown below.

[80] Fig. 6B also shows a selection of a sequence parameter set (610). The first syntax element shown is adaptive_pic_resolution_change_flag (611). When true, this flag may indicate the use of an adaptive resolution, which in turn may require certain control information. In the example, such control information is conditionally present based on the value of the flag based on the if() statement in the parameter set (612) and the tile group header (601).

[81] When adaptive resolution is used, the output resolution in units of samples (613) is encoded in this example. Item number 613 refers to both output_pic_width_in_luma_samples and output_pic_height_in_luma_samples, which together may define the output image resolution. Certain restrictions on either value may be defined elsewhere in the video coding technology or standard. For example, a level definition may limit the number of total output samples that may be the product of the values of these two syntax elements. Also, certain video coding technologies or standards, or external technologies or standards such as system standards, may limit the range of the numbering (e.g., one or both dimensions must be divisible by a power of 2) or the ratio of the width to the height (e.g., the width and height must be in a ratio such as 4:3 or 16:9). Such restrictions may be introduced to facilitate hardware implementations or for other reasons, and are well known in the art.

[82] In some applications, it may be appropriate for the encoder to instruct the decoder to use a specific reference picture size, rather than implicitly assuming that this size is the output picture size. In this example, the reference_pic_size_present_flag syntax element (614) conveys the conditional presence of the dimensions (615) of the reference picture (again, the position number refers to both the width and the height).

[83] Finally, a table of possible widths and heights of the decoded image is shown. Such a table can be expressed, for example, by specifying a table (num_dec_pic_size_in_luma_samples_minus1) (616). "Minus1" can refer to the interpretation of the value of this syntax element. For example, if the encoded value is zero, there is one table entry. If the value is five, there are six table entries. Then, for each "row" in the table, the width and height of the decoded image are included in the syntax (617).

[84] The table entries (617) provided can be indexed using the dec_pic_size_idx (602) syntax element in the tile group header, thereby allowing different decoded sizes - essentially scaling factors - for each tile group.

[85] Some video coding technologies or standards, such as VP9, support spatial scalability by implementing certain forms of reference image resampling (transmitted in a completely different manner than in the disclosed subject matter) in combination with temporal scalability to provide spatial scalability. In particular, certain reference images may be upsampled using ARC-style techniques to a higher resolution to form the basis of a spatial enhancement layer. These upsampled images may be enhanced using normal high-resolution prediction mechanisms to add detail.

[86] The embodiments discussed herein may be used in such an environment. In some cases, in the same or another embodiment, a value in a NAL unit header, such as a temporal identifier field, may be used to indicate not only a temporal layer, but also a spatial layer. This may have certain advantages for certain system configurations; for example, the existing Selected Forwarding Units (SFU) architecture, created and optimized for selected temporal layers forwarded based on the temporal identifier value of a NAL unit header, may be used without modification for scalable environments. In order to make this possible, there may be a requirement for a mapping between the coded picture size and the temporal layer indicated by the temporal identifier field in the NAL unit header.

[87] Recently, some attention has been drawn to compressed domain aggregation or extraction of multiple semantically independent image parts into a single video image. In particular, in the context of, for example, 360 coding or certain surveillance applications, multiple semantically independent source images (e.g., the surface of six cubes of a cube-projected 360 scene or individual camera inputs in the case of a multi-camera surveillance setup) may require separate adaptive resolution settings to cope with the different activity of each scene at a given time. In other words, encoders may choose to use different resampling factors for different semantically independent images that make up the entire 360-degree or surveillance scene at a given time. When combined into a single image, there is a need to perform resampling of the reference image and signal the availability of adaptive resolution coding for the coded image parts.

[88] In embodiments, not all samples of the reconstructed image are intended for output. The encoder may specify a rectangular portion of the image intended for output using a correspondence window. The correspondence window may be described or designated, for example, by an offset to the left and right of the edges of the image, as determined by the size of the image. Certain use cases may be identified where correspondence windows may be important, including stretched scanning, spatial assembly of views in Multiview systems, or 360 systems, where correspondence windows may point to one of several surfaces of a cube map for output.

[89] Since not all applications require the use of a match window, and since match window parameters may require a certain number of bits in the bitstream and therefore, when not used, may harm coding efficiency, the availability of such parameters may be restricted by a flag.

[90] In embodiments, the size of the matching window may be signaled in a picture parameter set (PPS). The matching window parameters, which may indicate the size of the matching window, may be used to calculate the resampling factor when the size of the matching window of the reference picture differs from the size of the window of the current picture. The decoder may need to recognize the size of the matching window of each picture to determine whether the resampling process is required. When the resampling factor is not equal to 1, the size of the output picture is not constant in the coded video sequence (CVS), and special processing and post-processing of the output pictures may be used, such as up/down scaling for display.

[91] In embodiments, a flag that indicates whether the decoded/output image has the same size and the resampling factor is 1 in the coded video sequence (CVS)/bitstream may be signaled in a high-level parameter set such as a decoding parameter set (DPS), a video parameter set (VPS), or a sequence parameter set (SPS). The flag may be used to negotiate a session for video streaming or to configure a decoder and display settings.

[92] Referring to Fig. 7, the constant_pic_size_flag (704) flag equal to 1 may indicate that the sizes of the images in the CVS are the same. However, the constant_pic_size_flag equal to 0 may indicate that the image sizes of the images in the CVS may or may not be the same. If the value of constant_pic_size_flag is 1, the sps_conformance_window_flag (705) flag may be present in the SPS (701). However, the sps_conformance_window_flag equal to 1 may indicate that the conformance crop window offset parameters follow in an appropriate position, such as later in the SPS. However, the sps_conformance_window_flag equal to 0 may indicate that the conformance crop window offset parameters are missing.

[93] In embodiments, sps_conf_win_left_offset (706), sps_conf_win_right_offset (707), sps_conf_win_top_offset (708), and sps_conf_win_bottom_offset (709) may point to image samples in the CVS that are output from the image decoding process as a rectangular region specified in the image coordinates for output.

[94] In embodiments where the syntax elements sps_conf_win_left_offset, sps_conf_win_right_offset, sps_conf_win_top_offset, and sps_conf_win_bottom_offset are not present, the values of sps_conf_win_left_offset, sps_conf_win_right_offset, sps_conf_win_top_offset, and sps_conf_win_bottom_offset may be inferred to be equal to 0.

[95] In embodiments, as shown in Fig. 8, pic_width_in_luma_samples (802) may define the width of each decoded picture based on the PPS (801) in units of luma samples. In embodiments, pic_width_in_luma_samples may not be equal to 0, may be an integer multiple of Max(8, MinCbSizeY), and may be less than or equal to pic_width_max_in_luma_samples. If absent, the value of pic_width_in_luma_samples may be derived as equal to pic_width_max_in_luma_samples. Wherein, pic_height_in_luma_samples (803) may indicate the height of each decoded picture based on the PPS in units of luma samples. In this case, pic_height_in_luma_samples in some cases may not be equal to 0, but may be an integer multiple of Max(8, MinCbSizeY), and may be less than or equal to pic_height_max_in_luma_samples. If absent, the value of pic_height_in_luma_samples may be inferred as equal to pic_height_max_in_luma_samples.

[96] In embodiments, as still shown in Fig. 8, conformance_window_flag (804) equal to 1 may indicate that the parameters of the conformity cropping window offset follow in a suitable place, for example, following in the PPS (801). In this case, conformance_window_flag equal to 0 may indicate that the parameters of the conformity cropping window offset are absent. In this case, conf_win_left_offset (805), conf_win_right_offset (806), conf_win_top_offset (807) and conf_win_bottom_offset (808) may indicate the samples of images based on the PPS, which are output from the decoding process, as a rectangular area specified in the coordinates of the image for output.

[97] In the same embodiment, when the syntax elements conf_win_left_offset, conf_win_right_offset, conf_win_top_offset, and conf_win_bottom_offset are absent, the values of conf_win_left_offset, conf_win_right_offset, conf_win_top_offset, and conf_win_bottom_offset may be inferred to be equal to the values of sps_conf_win_left_offset, sps_conf_win_right_offset, sps_conf_win_top_offset, and sps_conf_win_bottom_offset, respectively.

[98] In embodiments, the matching crop window may comprise brightness samples with horizontal image coordinates from SubWidthC * conf_win_left_offset to pic_width_in_luma_samples - (SubWidthC * conf_win_right_offset + 1) and vertical image coordinates from SubHeightC * conf_win_top_offset to pic_height_in_luma_samples - (SubHeightC * conf_win_bottom_offset + 1), inclusive.

[99] The value of SubWidthC * (conf_win_left_offset + conf_win_right_offset) may be less than pic_width_in_luma_samples, and the value of SubHeightC * (conf_win_top_offset + conf_win_bottom_offset) may be less than pic_height_in_luma_samples.

[100] The variables PicOutputWidthL and PicOutputHeightL can be obtained as shown in Equation 1 and Equation 2 below:

PicOutputWidthL = pic_width_in_luma_samples - SubWidthC * (conf_win_right_offset + conf_win_left_offset) (Equation 1)

PicOutputHeightL = pic_height_in_luma_samples - SubHeightC * (conf_win_bottom_offset + conf_win_top_offset) (Equation 2)

[101] In embodiments, the fractional interpolation process with resampling of the reference image may be processed as follows.

[102] The input data for this process may be a luma location (xSb, ySb) indicating the upper left sample of the current coding sub-unit relative to the upper left luma sample of the current picture, a variable sbWidth indicating the width of the current coding sub-unit, a variable sbHeight indicating the height of the current coding sub-unit, a motion vector offset mvOffset, a refined motion vector refMvLX, a selected array of reference picture samples refPicLX, an index of the half-sample interpolation filter hpelIfIdx, a bidirectional optical flow flag bdofFlag, and a variable cIdx indicating the index of the color component of the current block.

[103] The output of this process might be: an array predSamplesLX (sbWidth + brdExtSize) x (sbHeight + brdExtSize) of prediction sample values.

[104] The prediction block boundary extension size brdExtSize can be obtained as shown in Equation 3 below:

brdExtSize = (bdofFlag || (inter_affine_flag[xSb][ySb] && sps_affine_prof_enabled_flag)) ? 2:0 (Equation 3)

[105] The variable fRefWidth can be set equal to PicOutputWidthL of the reference image in luminance samples. The variable fRefHeight can be set equal to PicOutputHeightL of the reference image in luminance samples. The motion vector mvLX can be set equal to (refMvLX - mvOffset).

[106] If cIdx is 0, the following may apply:

The scaling factors and their fixed-point representations can be defined according to Equation 4 and Equation 5 below:

hori_scale_fp = ((fRefWidth<<14)+(PicOutputWidthL

>>1))/PicOutputWidthL (Equation 4)

vert_scale_fp = ((fRefHeight<<14)+(PicOutputHeightL

>>1))/PicOutputHeightL (Equation 5)

Let (xIntL,yIntL) be the luminance location specified in integer sample units, and (xFracL,yFracL) may be the offset specified in 1/16 sample units. These variables can be used in this section to specify fractional sample locations within the refPicLX arrays of reference samples.

The upper left coordinate of the bounding box for filling with the reference sample (xSbIntL,ySbIntL) can be set to (xSb+(mvLX[0]>>4),ySb+(mvLX[1]>>4)).

For each luminance sample location (xL=0..sbWidth - 1 + brdExtSize, yL=0..sbHeight-1+brdExtSize) within the predicted luminance samples array predSamplesLX, the corresponding predicted luminance sample value predSamplesLX [xL] [yL] is obtained as follows:

Let (refxSbL, refySbL) and (refxL,refyL) be the luminance locations pointed to by the motion vector (refMvLX, refMvLX), given in units of 1/16 counts. The variables refxSbL, refxL, refySbL, and refyL can be obtained as shown in equations 6-9 below:

refxSbL=((xSb<<4)+refMvLX[0])*hori_scale_fp (Equation 6)

refxL=((Sign(refxSb)*((Abs(refxSb)+128)>>8)
+ xL*((hori_scale_fp+8)>>4))+32)>>6(Equation 7)

refySbL=((ySb <<4)+refMvLX[1])* vert_scale_fp (Equation 8)

refyL=((Sign(refySb)*((Abs(refySb)+128)>>8)+yL*
((vert_scale_fp+8)>>4))+32)>>6(Equation 9)

The variables xIntL, yIntL,xFracL and yFracL can be obtained as shown in equations 10-13 below:

xIntL=refxL>>4 (Equation 10)

yIntL=refyL>>4 (Equation 11)

xFracL=refxL & 15 (Equation 12)

yFracL=refyL & 15 (Equation 13)

If bdofFlag is TRUE or (sps_affine_prof_enabled_flag is TRUE and inter_affine_flag[xSb][ySb] is TRUE) and one or more of the following conditions are true, the predicted luma sample value predSamplesLX[xL][yL]may be obtained by invoking the integer luma sample extraction process as specified in the relevant clause of the video coding specification, with (xIntL+(xFracL>>3)-1),yIntL+(yFracL>>3)-1) and refPicLX as input.

1. xL is equal to 0.

2. xL is equal to sbWidth+1.

3. yL is equal to 0.

4. yL is equal to sbHeight + 1.

Otherwise, the predicted luminance sample value predSamplesLX[xL][yL] can be obtained by invoking the 8-way luminance sample interpolation filtering process as specified in the corresponding clause of the video coding specification with (xIntL - (brdExtSize > 0 ? 1:0),yIntL - (brdExtSize>0?1:0)), (xFracL, yFracL), (xSbIntL, ySbIntL), refPicLX, hpelIfIdx, sbWidth, sbHeight, and (xSb, ySb) as inputs.

[107] Otherwise (cIdx is not 0), the following may apply:

1. Let (xIntC, yIntC) be the chroma location specified in integer sample units, and (xFracC, yFracC) be the offset specified in 1/32 sample units. These variables can be used in this section to specify common fractional sample locations within the refPicLX arrays of reference samples.

2. The top left coordinate of the bounding box to fill with the reference count (xSbIntC, ySbIntC) is set to ((xSb / SubWidthC) + (mvLX[0] >> 5), (ySb / SubHeightC) + (mvLX[1]) >> 5)).

3. For each chroma sample location (xC = 0..sbWidth - 1, yC = 0 ..sbHeight - 1) within the predicted chroma samples arrays predSamplesLX, the corresponding predicted chroma sample value predSamplesLX[xC][yC] can be obtained as follows:

Let (refxSbC, refySbC) and (refxC,refyC) be the chromaticity locations pointed to by the motion vector (mvLX[0], mvLX[1]), specified in units of 1/32 of a sample. The variables refxSbC, refySbC, refxC, and refyC can be obtained as shown in equations 14-17 below:

refxSbC =((xSb / SubWidthC << 5) + mvLX[0]) * hori_scale_fp (Equation 14)

refxC = ((Sign (refxSbC)* ((Abs (refxSbC ) + 256) >> 9)
+ xC * ((hori_scale_fp + 8) >> 4)) + 16) >> 5 (Equation 15)

refySbC =( (ySb / SubHeightC<< 5) + mvLX[1]) * vert_scale_fp (Equation 16)

refyC = ((Sign (refySbC)* ((Abs (refySbC ) + 256) >> 9)
+ yC * ((vert_scale_fp + 8) >> 4)) + 16) >> 5 (Equation 17)

The variables xIntC, yIntC, xFracC and yFracC can be obtained as shown in Equations 18-21 below:

xIntC = refxC>> 5 (Equation 18)

yIntC = refyC>> 5 (Equation 19)

xFracC = refyC&31 (Equation 20)

yFracC = refyC&31 (Equation 21)

[108] The predicted sample value predSamplesLX[xC][yC] can be obtained by invoking the above process with (xIntC, yIntC), (xFracC, yFracC), (xSbIntC, ySbIntC), sbWidth, sbHeight, and refPicLX as inputs.

[109] Fig. 9 shows an example flow chart of a process 900 for decoding an encoded video bitstream. In some implementations, one or more of the process blocks of Fig. 9 may be performed by decoder 210. In some implementations, one or more of the process blocks of Fig. 9 may be performed by another device or group of devices separate from or including decoder 210, such as encoder 203.

[110] As shown in Fig. 9, process 900 may include receiving a first flag indicating whether a constant picture size is used in an encoded video sequence including the current picture (step 910).

[111] As further shown in Fig. 9, process 900 may include determining from a first flag whether a constant image size is used (step 920).

[112] As further shown in Fig. 9, the process 900 may include, based on a first flag indicating that a constant picture size is used (YES in step 920), decoding the current picture without performing resampling of the reference picture (step 930).

[113] As further shown in Fig. 9, based on the first flag indicating that a constant image size is not used (NO in block 920), process 900 may proceed to block 940, block 950, block 960, and block 970.

[114] As further shown in Fig. 9, process 900 may include receiving a second flag indicating whether the compliance window size is signaled (step 940).

[115] As further shown in Fig. 9, the process 900 may include, based on a second flag indicating that a correspondence window size is signaled, obtaining the correspondence window size (block 950), determining a resampling factor between the current picture and the reference picture based on the correspondence window size (block 960), and performing resampling of the reference picture for the current picture using the specified resampling factor (block 970).

[116] In one embodiment, the size of the matching window may be signaled as at least one offset distance from the boundary of the current image.

[117] In one embodiment, the first flag may be signaled in a sequence parameter set (SPS), and the second flag may be signaled in one of the SPS and a picture parameter set (PPS).

[118] In one embodiment, the second flag may be signaled in the SPS and may indicate whether the SPS compliance window parameters are signaled in the SPS.

[119] In one embodiment, based on the second flag indicating that the SPS conformance window parameters are signaled in the SPS, the conformance window size may be obtained based on the SPS conformance window parameters.

[120 In one embodiment, based on the first flag indicating that the image size is not constant, the process 900 may include receiving a third flag indicating whether the PPS matching window parameters are signaled in the PPS.

[121] In one embodiment, based on the second flag indicating that the SPS conformance window parameters are signaled in the SPS and the third flag indicating that the PPS conformance window parameters are not signaled in the PPS, the conformance window size may be obtained based on the SPS conformance window parameters.

[122] In one embodiment, based on the second flag indicating that the SPS conformance window parameters are not signaled in the SPS and the third flag indicating that the PPS conformance window parameters are signaled in the PPS, the conformance window size may be obtained based on the PPS conformance window parameters.

[123] Although Fig. 9 shows exemplary blocks of process 900, in some implementations, process 900 may include additional blocks, fewer blocks, different blocks, or blocks arranged differently than those shown in Fig. 9. Additionally or alternatively, two or more blocks of process 900 may be executed in parallel.

[124] Furthermore, the proposed methods may be implemented by processing circuits (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored on a non-transitory machine-readable medium to implement one or more of the proposed methods.

[125] The methods described above may be implemented as computer software using machine-readable instructions and physically stored on one or more machine-readable media. For example, Fig. 10 shows a computer system 1000 suitable for implementing certain embodiments of the disclosed subject matter.

[126] Computer software may be encoded using any suitable machine code or computer language that can be subject to assembly, compilation, linking, or similar mechanisms to produce code containing instructions that can be executed directly or through interpretation, microcode execution, etc., by computer central processing units (CPUs), graphics processing units (GPUs), etc.

[127] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, etc.

[128] The components shown in Fig. 10 for computer system 1000 are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of computer software implementing embodiments of the present disclosure. Nor should the configuration of components be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary embodiment of computer system 1000.

[129] The computer system 1000 may include certain human-machine interface input devices. Such a human-machine interface input device may respond to input from one or more human users through, for example, tactile input (e.g., keystrokes, swipes, control glove movements), audio input (e.g., voice, hand claps), visual input (e.g., gestures), olfactory input (not shown). Human-machine interface devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as sound (e.g., speech, music, ambient sound), images (e.g., scanned images, photographic images obtained from a still image camera), video (e.g., two-dimensional video, three-dimensional video, including stereoscopic video).

[130] Human interface input devices may include one or more of (only one of each): keyboard 1001, mouse 1002, trackpad 1003, touch screen 1010 and corresponding graphics adapter 1050, control glove, joystick 1005, microphone 1006, scanner 1007, camera 1008.

[131] The computer system 1000 may also include certain human-machine interface output devices. Such human-machine interface output devices may stimulate the senses of one or more human users through, for example, tactile output, sound, light, and smell/taste. Such human-machine interface output devices may include haptic output devices (e.g., haptic feedback from a touch screen (1010), a control glove (1004), or a joystick (1005), but there may also be haptic feedback devices that do not serve as input devices), audio output devices (e.g., speakers (1009), headphones (not shown)), image output devices (e.g., screens (1010), including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capability, each with or without haptic feedback capability - some of which may be capable of outputting two-dimensional visual output or more than three-dimensional output by such means as stereographic output; virtual reality glasses (not shown), holographic displays and smoke tanks (not shown)) and printers (not shown).

[132] The computer system (1000) may also include human accessible storage devices and associated media such as optical media including CD/DVD ROM/RW (1020) with CD/DVD or similar media (1021), flash drive (1022), removable hard disk or solid state drive (1023), previously developed magnetic media such as tape and floppy disk (not shown), specialized ROM/ASIC/PLD based devices such as security keys (not shown), etc.

[133] Those skilled in the art will also appreciate that the term "machine-readable medium" as used in connection with the subject matter disclosed herein does not include transmission media, carrier waves, or other time signals.

[134] The computer system 1000 may also include an interface(s) to one or more communication networks (1155). The networks may be, for example, wireless, wired, optical. In addition, the networks may be local, wide area, metropolitan, vehicular and industrial, real-time, delay-tolerant, etc. Examples of networks include local area networks such as Ethernet, wireless local area networks (LANs), cellular networks including global systems for mobile communications (GSM), third generation (3G), fourth generation (4G), fifth generation (5G), Long Term Evolution (LTE) networks, etc., wired television or wireless wide area digital networks including cable television, satellite television and terrestrial broadcast television, vehicular and industrial, including CAN bus, and so on. Some networks typically require external network interface adapters (1154) that are connected to certain general-purpose data ports or peripheral buses (1149) (e.g., universal serial bus (USB) ports of computer system 1000); others are typically integrated into the core of computer system 1000 by connecting to a system bus, as described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). For example, network 1055 can be connected to peripheral bus 1049 using network interface 1054. Using any of these networks, computer system 1000 can communicate with other entities. Such communication can be unidirectional, for receiving only (e.g., broadcast television), unidirectional for sending only (e.g., a CAN bus to certain devices with a CAN bus), or bidirectional, such as for other computer systems using a local or wide area digital network. Each of these networks and network interfaces (1154) can use specific protocols and protocol stacks, as described above.

[135] The above-mentioned human-machine interface devices, human accessible storage devices and network interfaces may be connected to the core 1040 of the computer system 1000.

[136] The core (1040) may include one or more central processing units (CPUs) (1041), graphics processing units (GPUs) (1042), specialized programmable processors in the form of field programmable gate arrays (FPGAs) (1043), hardware accelerators (1044) for specific tasks, etc. These devices, along with read-only memory (ROM) 1045, random access memory (RAM) 1046, internal mass storage such as internal hard drives that are not accessible to the user, solid-state drives (SSDs), etc. 1047, may be connected via a system bus 1048. In some computer systems, the system bus 1048 may be accessible as one or more physical slots to provide the ability to expand with additional CPUs, GPUs, etc. Peripheral devices may be connected either directly to the system bus 1048 cores, or via the 1049 peripheral bus. The peripheral bus architecture includes the connection of peripheral components (PCI), USB, etc.

[137] The CPU 1041, the GPU 1042, the FPGA 1043, and the accelerators 1044 may execute certain instructions that, in combination, may constitute the aforementioned computer code. This computer code may be stored in the ROM 1045 or the RAM 1046. Transient data may also be stored in the RAM 1046, while persistent data may be stored, for example, in the internal mass storage 1047. Fast storage and retrieval to any of the storage devices may be achieved through the use of a cache memory, which may be tightly coupled to one or more of the CPU 1041, the GPU 1042, the mass storage 1047, the ROM 1045, the RAM 1046, and the like.

[138] The computer-readable medium may contain computer code for performing various computer-implemented operations. The media and computer code may be specially designed and constructed for the purposes of this disclosure, or they may be well known and available to those skilled in the art of computer software.

[139] By way of example, and not limitation, a computer system having architecture 1000, and in particular core 1040, may provide functionality by having a processor(s) (including CPUs, GPUs, FPGAs, accelerators, etc.) executing software embodied on one or more tangible, computer-readable media. Such computer-readable media may be media associated with a user-accessible mass storage device as described above, as well as certain memory devices of core 1040 that are non-transitory in nature, such as internal mass storage device 1047 or ROM 1045. Software implementing various embodiments of the present disclosure may be stored in such devices and executed by core 1040. The computer-readable medium may include one or more memory devices or chips according to particular needs. The software may cause the core 1040, and in particular the processors therein (including the CPU, GPU, FPGA, etc.), to perform certain processes or certain portions of certain processes described herein, including defining data structures stored in the RAM 1046 and modifying such data structures in accordance with the processes defined by the software. In addition or as an alternative, the computer system may provide functionality as a result of logic embedded in hardware circuitry or otherwise embodied in circuitry (e.g., accelerator 1044) that may operate in place of or in conjunction with software to perform certain processes or certain portions of certain processes described herein. Reference to software may include logic and vice versa, where applicable. Reference to a computer-readable medium may include circuitry (e.g., an integrated circuit (IC)) storing software for execution, circuitry embodying logic for execution, or both, where applicable. The present disclosure includes any suitable combination of hardware and software.

[140] Although this disclosure describes several exemplary embodiments, there are modifications, permutations, and various substitution equivalents that fall within the scope of the disclosure. It will thus be appreciated that those skilled in the art will be able to devise numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the disclosure and are thus within its spirit and scope of protection.

Claims (29)

1. A method for encoding a video bitstream using at least one processor, wherein the method comprises: based on the determination that a constant picture size is not used to encode a video sequence that includes the current picture, setting a first flag indicating whether the size of the matching window is signaled; and based on the first flag indicating that the match window size is being signaled: setting the size of the matching window, and signaling the resampling factor between the current image and the reference image based on the size of the matching window. 2. The method according to claim 1, wherein the size of the correspondence window is signaled as at least one offset distance from the boundary of the current image. 3. The method according to claim 1, wherein the first flag is signaled in one of a sequence parameter set (SPS) and a picture parameter set (PPS). 4. The method according to claim 3, wherein the first flag is signaled in the SPS and indicates whether the SPS conformance window parameters are signaled in the SPS. 5. The method according to claim 4, wherein based on the first flag indicating that the SPS conformance window parameters are signaled in the SPS, the conformance window size is set based on the SPS conformance window parameters. 6. The method of claim 4, further comprising setting a second flag indicating whether the PPS correspondence window parameters are signaled in the PPS based on a determination that the image size is not constant. 7. The method according to claim 6, wherein, based on the first flag indicating that the SPS conformance window parameters are signaled in the SPS and the second flag indicating that the PPS conformance window parameters are not signaled in the PPS, the conformance window size is determined based on the SPS conformance window parameters. 8. The method according to claim 6, wherein based on the first flag indicating that the SPS conformance window parameters are not signaled in the SPS and the second flag indicating that the PPS conformance window parameters are signaled in the PPS, the conformance window size is determined based on the PPS conformance window parameters. 9. A device for encoding a video bitstream, the device comprising: at least one memory element configured to store program code; and at least one processor configured to read program code and operate in accordance with the instructions of the program code, wherein the program code includes: a first setting code configured to cause at least one processor, based on a determination that a constant picture size is not used to encode a video sequence including the current picture, to set a first flag indicating whether a match window size is signaled; and execution code configured to cause at least one processor, based on a first flag indicating that a match window size is signaled, to set a match window size, and to signal an oversampling factor between the current image and the reference image based on the match window size. 10. The device of claim 9, wherein the size of the matching window is signaled as at least one offset distance from the boundary of the current image. 11. The device of claim 9, wherein the first flag is signaled in one of a sequence parameter set (SPS) and a picture parameter set (PPS). 12. The device of claim 11, wherein the first flag is signaled in the SPS and indicates whether the SPS compliance window parameters are signaled in the SPS. 13. The device according to claim 12, wherein based on the first flag indicating that the SPS conformance window parameters are signaled in the SPS, the conformance window size is set based on the SPS conformance window parameters. 14. The device of claim 12, wherein the program code further includes a second setting code configured to cause at least one processor to set a second flag indicating whether the PPS correspondence window parameters are signaled in the PPS based on the determination that the image size is not constant. 15. The device of claim 14, wherein, based on the first flag indicating that the SPS conformance window parameters are signaled in the SPS and the second flag indicating that the PPS conformance window parameters are not signaled in the PPS, the conformance window size is set based on the SPS conformance window parameters. 16. The device according to claim 14, wherein based on the first flag indicating that the SPS conformance window parameters are not signaled in the SPS and the second flag indicating that the PPS conformance window parameters are signaled in the PPS, the conformance window size is set based on the PPS conformance window parameters. 17. A non-transitory machine-readable medium storing a bit stream that may be encoded by: based on determining that a constant picture size is not used to encode a video sequence that includes the current picture, setting a first flag indicating whether the size of the matching window is signaled; and based on the first flag indicating that the match window size is being signaled: setting the size of the matching window, and signaling the resampling factor between the current image and the reference image based on the size of the matching window.