WO2025038308A1

WO2025038308A1 - Direct frame prediction with residual coding

Info

Publication number: WO2025038308A1
Application number: PCT/US2024/040688
Authority: WO
Inventors: Yue Chen; Debargha Mukherjee; Lester LU; Rachel BARKER; Urvang Joshi; Mohammed Golam Sarwer; Jianle Chen; Onur Guleryuz
Original assignee: Google LLC
Current assignee: Google LLC
Priority date: 2023-08-14
Filing date: 2024-08-02
Publication date: 2025-02-20
Anticipated expiration: 2026-02-14

Abstract

Coding blocks of a current frame is disclosed. A frame-level prediction mode is coded in a header of the current frame. First blocks of the current frame are coded using the frame-level prediction mode. No residual blocks are associated with any of the first blocks. One or more second blocks of the current frame are coded. Coding the one or more second blocks includes coding respective residual blocks for the one or more second blocks.

Description

DIRECT FRAME PREDICTION WITH RESIDUAL CODING

BACKGROUND

[0001] Digital video streams may represent video using a sequence of frames or still images. Digital video can be used for various applications including, for example, video conferencing, high definition video entertainment, video advertisements, or sharing of usergenerated videos. A digital video stream can contain a large amount of data and consume a significant amount of computing or communication resources of a computing device for processing, transmission, or storage of the video data. Various approaches have been proposed to reduce the amount of data in video streams, including encoding or decoding techniques.

SUMMARY

[0002] One general aspect includes a method for coding blocks of a current frame. The method includes coding a frame-level prediction mode in a header of the current frame; coding first blocks of the current frame using the frame-level prediction mode, where no residual blocks are associated with any of the first blocks; and coding one or more second blocks of the current frame, where coding the one or more second blocks may include: coding respective residual blocks for the one or more second blocks.

[0003] Implementations may include one or more of the following aspects.

[0004] The method where the current frame may be partitioned into fixed-size blocks.

[0005] A partitioning of at least one of the one or more second blocks may be identified based on a quad-tree partitioning.

[0006] Coding the one or more second blocks of the current frame may include coding the one or more second blocks of the current frame based on the frame-level prediction mode. [0007] Coding the one or more second blocks of the current frame may include coding one of the one or more second blocks based on a prediction mode coded in a header of the one of the one or more second blocks.

[0008] The method may include identifying respective locations of the one or more second blocks.

[0009] Identifying the respective locations of the one or more second blocks may include inferring the respective locations of the one or more second blocks based on the frame-level prediction mode.

[0010] Identifying the respective locations of the one or more second blocks may include decoding, from a compressed bitstream, the respective locations of the one or more second blocks.

[0011] The frame-level prediction mode can be a global-motion prediction mode and the method may include coding parameters of the global-motion prediction mode.

[0012] The method may include inferring locations of the one or more second blocks based on the frame-level prediction mode.

[0013] The frame-level prediction mode can be based on temporal interpolated picture.

[0014] At least some of the one or more second blocks of the current frame can be coded using the frame-level prediction mode.

[0015] The method may include applying a deblocking filter to at least one of the one or more second blocks.

[0016] Coding the one or more second blocks may include coding the one or more second blocks based on a fixed transform type.

[0017] Coding the one or more second blocks may include identifying a transform type, from a set of available transform types, for coding at least one of the one or more second blocks based on a fixed transform type.

[0018] Coding the one or more second blocks may include coding the one or more second blocks based on a predefined transform size.

[0019] The predefined transform size can be one of a smallest transform block size, a largest transform block size, or a fixed transform block size.

[0020] Coding the one or more second blocks may include identifying, for a block of the one or more second blocks, signaled transform block sizes for coding the block.

[0021] Implementations of the described techniques may include hardware, a method or process, or computer software on a computer-accessible medium.

[0022] An implementation may be a device that includes a processor that is configured to perform at least some of the previous aspects.

[0023] An implementation may be a device that includes a memory and a processor where the processor is configured to execute instructions stored in the memory to perform at least some of the previous aspects.

[0024] An implementation may be a non-transitory computer-readable storage medium that includes executable instructions that, when executed by a processor, facilitate performance of operations that perform at least some of the previous aspects. [0025] An implementation may be a non-transitory computer-readable storage medium having stored thereon an encoded bitstream, wherein the encoded bitstream is configured for decoding by at least some of the previous aspects.

[0026] An implementation may be a non-transitory computer-readable storage medium having stored thereon an encoded bitstream generated by an encoder performing at least some of the previous aspects.

[0027] These and other aspects of the present disclosure are disclosed in the following detailed description of the embodiments, the appended claims and the accompanying figures. It will be appreciated that aspects can be implemented in any convenient form. For example, aspects may be implemented by appropriate computer programs which may be carried on appropriate carrier media which may be tangible carrier media (e.g., disks) or intangible carrier media (e.g., communications signals). Aspects may also be implemented using suitable apparatus which may take the form of programmable computers running computer programs arranged to implement the methods and/or techniques disclosed herein. Aspects can be combined such that features described in the context of one aspect may be implemented in another aspect.

BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The description herein makes reference to the accompanying drawings described below, wherein like reference numerals refer to like parts throughout the several views. [0029] FIG. l is a schematic of a video encoding and decoding system.

[0030] FIG. 2 is a block diagram of an example of a computing device that can implement a transmitting station or a receiving station.

[0031] FIG. 3 is a diagram of a typical video stream to be encoded and subsequently decoded.

[0032] FIG. 4 is a block diagram of an encoder according to implementations of this disclosure.

[0033] FIG. 5 is a block diagram of a decoder according to implementations of this disclosure.

[0034] FIG. 6 is an illustration of examples of portions of a video frame.

[0035] FIG. 7 is an illustration of a frame-level prediction mode that is based on temporal interpolated picture.

[0036] FIGS. 8A-D are illustrations of frame-level prediction modes that are global motion models. [0037] FIG. 9 is a flowchart of a technique for coding blocks of a current frame based on a frame-level prediction mode.

DETAILED DESCRIPTION

[0038] Video compression schemes may include breaking respective images, or video frames, into smaller portions, such as video blocks, and generating an encoded bitstream using techniques to limit the information included for respective video blocks thereof. The encoded bitstream can be decoded to re-create the source images from the limited information. Encoding or decoding a video block can include predicting motion within that video block, such as with respect to one or more other video blocks in the same video frame or in a different video frame.

[0039] Encoding a video stream, or a portion thereof, such as a frame or a block, can include using temporal in the video stream to improve coding efficiency. For example, a current block of a video stream may be encoded based on identifying a difference (residual) between the previously coded pixel values, or between a combination of previously coded pixel values, and those in the current block.

[0040] Encoding using spatial similarities can be known as intra prediction. Intra prediction can attempt to predict the pixel values of a block of a frame of a video stream using pixels peripheral to the block; that is, using pixels that are in the same frame as the block but that are outside the block. A prediction block resulting from intra prediction is referred to herein as an intra predictor. Intra prediction can be performed along a direction of prediction where each direction can correspond to an intra prediction mode. The intra prediction mode can be signalled by an encoder to a decoder. A prediction block resulting from intra prediction is referred to herein as intra predictor.

[0041] Inter prediction is performed using a motion vector. A motion vector used to generate a prediction block refers to a frame other than a current frame, i.e., a reference frame. Reference frames can be located before or after the current frame in the sequence of the video stream. Some codecs use up to eight reference frames, which can be stored in a frame buffer. The motion vector can refer to (i.e., use) one of the reference frames of the frame buffer. For a current block being encoded or decoded (i.e., a coding block), the motion vector describes a vertical offset and a horizontal offset in the reference frame of a collocated reference block.

[0042] Video coding as described above typically requires that coding mode information be associated with at least many of the blocks of a frame therewith consuming a substantial number of bits in a bitstream. That is, coding mode information for a block may be included in block header of the block. Coding modes, such as the SKIP, MERGE, or DIRECT modes have been developed (e.g., implemented by codecs) to reduce the number of bits required for coding block-level coding mode information. Different codecs may have different implementations and semantics for such coding modes.

[0043] To illustrate, and without limitations, if a current block is coded with the MERGE mode, then the motion vectors of all the neighboring blocks as well as the residual error for the current block may be transmitted from an encoder to a decoder (such as via a compressed bitstream); if a current block is coded with the DIRECT mode, only the motion vector for the current block may be transmitted and the decoder may estimate the residual error from the motion vector and other decoded blocks in the same frame; and, if a current block is coded with the SKIP mode, no motion information and no residual information is transmitted from the encoder to the decoder for the current block. But, again, different codecs may have different semantics for these modes.

[0044] While these modes reduce the bit requirements associated with block-level coding mode information, at least certain block-level coding mode information is still required. To illustrate, a block header may minimally include a flag (e.g., skip) indicating whether residual information is associated with the block.

[0045] To further reduce the bits required for coding mode information, direct framelevel prediction modes have been developed. A direct frame-level prediction mode may generate a prediction frame according to a single frame-level prediction algorithm. Said another way, each block of a frame that is coded using a direct frame-level prediction mode is coded using that prediction mode indicated for the frame. Furthermore, a defining characteristic of direct frame-level prediction modes is that no residual information is transmitted for the blocks of the frame. As such, the prediction block itself of a current block becomes the reconstructed block of the current block.

[0046] A direct frame-level prediction mode can be any prediction mode, technique, or algorithm that does not require transmitting (e.g., including) block-level prediction information in a compressed bitstream. Two examples of frame-level prediction modes are described with respect to FIGS. 7 and 8A-8D. FIG. 7 describes a frame-level prediction mode that is based on a temporal interpolated picture (TIP). The frame-level prediction mode described with respect to FIG. 7 is referred to as the “TIP mode.” FIGS. 8A-8D describe a frame-level prediction mode that is based on a global motion model (GMM) and a reference frame. The frame-level prediction mode described with respect to FIGS. 8A-8D is referred to as the “GMM mode.” However, the teachings herein are not limited to these two frame-level prediction modes. It is also noted that the same concepts described herein can be used with tiles or segments of a frame, where each tile or segment can be associated with a different frame-level prediction mode.

[0047] Frame-level prediction modes have the advantage of extremely low overhead compared to traditional block-based prediction. This is because the syntax that signals the frame-level prediction model is included only at the frame level, rather than for each individual block. However, using a direct frame-level prediction mode for a frame may compromise adaptability to local (e.g., block-level) content variability that the frame model would miss.

[0048] Direct frame prediction with residual coding, according to implementations of this disclosure, presents a residual coding framework designed to compensate for the information lost in the direct frame-level prediction.

[0049] For ease of reference, a block of a current frame with which a residue (i.e., a residual block) is associated is referred to herein as a “residue-based block;” and a block of the current frame with which no residue is associated is referred to herein as a “residue-less block.” That is, the compressed bitstream would include a residual block (e.g., transform coefficients) for a residue-based block; but would not include a residual block (e.g., transform coefficients) for a residue-less block. One or more of coding block partitioning (e.g., further into subblocks), transform type(s), or block (filter) restoration may be associated with a residue-based block, as further described herein.

[0050] A frame may be first partitioned into coding blocks (or, simply, blocks). A block can either have associated therewith a residual block (included in a compressed bitstream) or not. The block partitioning can be into variable block sizes (such as via quadtree partition). Alternatively, and to minimize signaling overhead, certain fixed block sizes can be used. For at least some of the blocks, one or more syntax elements may be used to indicate whether at least one of transform coding or loop restoration are activated (e.g., enabled or performed) for the block. As further described herein, whether transform coding is activated for a block may be signaled at the block level (e.g., in the block header) or may be derived based on other data. Similarly, and as further described herein, whether loop restoration is activated for a block may be signaled at the block level (e.g., in the block header) or may be derived based on other data.

[0051] That a block is a residue-based block can either be explicitly signaled (such as by an encoder) or derived (such as by a decoder) from (e.g., inferred based on) the frame-level prediction model. Residual coding (i.e., for residue-based blocks) can be used to cover areas of a current frame that are not well represented by the frame-level prediction model (such as borders that are out of reach of a global warping model, as further described below). If a coding block has residual coding activated, transform coding may also be applied, which can have variable transform sizes or derived transform sizes (e.g., max supported size for the coding block), and based on the same transform kernel set for regular coded frames or a reduced subset thereof. In-loop restoration may also be performed to mitigate artifacts (e.g., blocking artifacts) between transform blocks. It is also possible to apply additional restoration filters to enhance the reconstruction.

[0052] Further details of techniques for direct frame prediction with residual coding are described herein with initial reference to a system in which they can be implemented. FIG. 1 is a schematic of a video encoding and decoding system 100. A transmitting station 102 can be, for example, a computer having an internal configuration of hardware such as that described in FIG. 2. However, other implementations of the transmitting station 102 are possible. For example, the processing of the transmitting station 102 can be distributed among multiple devices.

[0053] A network 104 can connect the transmitting station 102 and a receiving station 106 for encoding and decoding of the video stream. Specifically, the video stream can be encoded in the transmitting station 102, and the encoded video stream can be decoded in the receiving station 106. The network 104 can be, for example, the Internet. The network 104 can also be a local area network (LAN), wide area network (WAN), virtual private network (VPN), cellular telephone network, or any other means of transferring the video stream from the transmitting station 102 to, in this example, the receiving station 106.

[0054] The receiving station 106, in one example, can be a computer having an internal configuration of hardware such as that described in FIG. 2. However, other suitable implementations of the receiving station 106 are possible. For example, the processing of the receiving station 106 can be distributed among multiple devices.

[0055] Other implementations of the video encoding and decoding system 100 are possible. For example, an implementation can omit the network 104. In another implementation, a video stream can be encoded and then stored for transmission at a later time to the receiving station 106 or any other device having memory. In one implementation, the receiving station 106 receives (e.g., via the network 104, a computer bus, and/or some communication pathway) the encoded video stream and stores the video stream for later decoding. In an example implementation, a real-time transport protocol (RTP) is used for transmission of the encoded video over the network 104. In another implementation, a transport protocol other than RTP may be used (e.g., a Hypertext Transfer Protocol-based (HTTP -based) video streaming protocol).

[0056] When used in a video conferencing system, for example, the transmitting station 102 and/or the receiving station 106 may include the ability to both encode and decode a video stream as described below. For example, the receiving station 106 could be a video conference participant who receives an encoded video bitstream from a video conference server (e.g., the transmitting station 102) to decode and view and further encodes and transmits his or her own video bitstream to the video conference server for decoding and viewing by other participants.

[0057] FIG. 2 is a block diagram of an example of a computing device 200 that can implement a transmitting station or a receiving station. For example, the computing device 200 can implement one or both of the transmitting station 102 and the receiving station 106 of FIG. 1. The computing device 200 can be in the form of a computing system including multiple computing devices, or in the form of one computing device, for example, a mobile phone, a tablet computer, a laptop computer, a notebook computer, a desktop computer, and the like.

[0058] A processor 202 in the computing device 200 can be a conventional central processing unit. Alternatively, the processor 202 can be another type of device, or multiple devices, capable of manipulating or processing information now existing or hereafter developed. For example, although the disclosed implementations can be practiced with one processor as shown (e.g., the processor 202), advantages in speed and efficiency can be achieved by using more than one processor.

[0059] A memory 204 in computing device 200 can be a read only memory (ROM) device or a random access memory (RAM) device in an implementation. However, other suitable types of storage device can be used as the memory 204. The memory 204 can include code and data 206 that is accessed by the processor 202 using a bus 212. The memory 204 can further include an operating system 208 and application programs 210, the application programs 210 including at least one program that permits the processor 202 to perform the techniques described herein. For example, the application programs 210 can include applications 1 through N, which further include a video coding application that performs the techniques described herein. The computing device 200 can also include a secondary storage 214, which can, for example, be a memory card used with a mobile computing device. Because the video communication sessions may contain a significant amount of information, they can be stored in whole or in part in the secondary storage 214 and loaded into the memory 204 as needed for processing.

[0060] The computing device 200 can also include one or more output devices, such as a display 218. The display 218 may be, in one example, a touch sensitive display that combines a display with a touch sensitive element that is operable to sense touch inputs. The display 218 can be coupled to the processor 202 via the bus 212. Other output devices that permit a user to program or otherwise use the computing device 200 can be provided in addition to or as an alternative to the display 218. When the output device is or includes a display, the display can be implemented in various ways, including by a liquid crystal display (LCD), a cathode-ray tube (CRT) display, or a light emitting diode (LED) display, such as an organic LED (OLED) display.

[0061] The computing device 200 can also include or be in communication with an image-sensing device 220, for example, a camera, or any other image-sensing device 220 now existing or hereafter developed that can sense an image such as the image of a user operating the computing device 200. The image-sensing device 220 can be positioned such that it is directed toward the user operating the computing device 200. In an example, the position and optical axis of the image-sensing device 220 can be configured such that the field of vision includes an area that is directly adjacent to the display 218 and from which the display 218 is visible.

[0062] The computing device 200 can also include or be in communication with a soundsensing device 222, for example, a microphone, or any other sound-sensing device now existing or hereafter developed that can sense sounds near the computing device 200. The sound-sensing device 222 can be positioned such that it is directed toward the user operating the computing device 200 and can be configured to receive sounds, for example, speech or other utterances, made by the user while the user operates the computing device 200.

[0063] Although FIG. 2 depicts the processor 202 and the memory 204 of the computing device 200 as being integrated into one unit, other configurations can be utilized. The operations of the processor 202 can be distributed across multiple machines (wherein individual machines can have one or more processors) that can be coupled directly or across a local area or other network. The memory 204 can be distributed across multiple machines such as a network-based memory or memory in multiple machines performing the operations of the computing device 200. Although depicted here as one bus, the bus 212 of the computing device 200 can be composed of multiple buses. Further, the secondary storage 214 can be directly coupled to the other components of the computing device 200 or can be accessed via a network and can comprise an integrated unit such as a memory card or multiple units such as multiple memory cards. The computing device 200 can thus be implemented in a wide variety of configurations.

[0064] FIG. 3 is a diagram of an example of a video stream 300 to be encoded and subsequently decoded. The video stream 300 includes a video sequence 302. At the next level, the video sequence 302 includes a number of adjacent frames 304. While three frames are depicted as the adjacent frames 304, the video sequence 302 can include any number of adjacent frames 304. The adjacent frames 304 can then be further subdivided into individual frames, for example, a frame 306. At the next level, the frame 306 can be divided into a series of planes or segments 308. The segments 308 can be subsets of frames that permit parallel processing, for example. The segments 308 can also be subsets of frames that can separate the video data into separate colors. For example, a frame 306 of color video data can include a luminance plane and two chrominance planes. The segments 308 may be sampled at different resolutions.

[0065] Whether or not the frame 306 is divided into segments 308, the frame 306 may be further subdivided into blocks 310, which can contain data corresponding to, for example, 16x16 pixels in the frame 306. The blocks 310 can also be arranged to include data from one or more segments 308 of pixel data. The blocks 310 can also be of any other suitable size such as 4x4 pixels, 8x8 pixels, 16x8 pixels, 8x16 pixels, 16x16 pixels, or larger. Unless otherwise noted, the terms block and macroblock are used interchangeably herein.

[0066] FIG. 4 is a block diagram of an encoder 400 according to implementations of this disclosure. The encoder 400 can be implemented, as described above, in the transmitting station 102, such as by providing a computer software program stored in memory, for example, the memory 204. The computer software program can include machine instructions that, when executed by a processor such as the processor 202, cause the transmitting station 102 to encode video data in the manner described in FIG. 4. The encoder 400 can also be implemented as specialized hardware included in, for example, the transmitting station 102. In one particularly desirable implementation, the encoder 400 is a hardware encoder.

[0067] The encoder 400 has the following stages to perform the various functions in a forward path (shown by the solid connection lines) to produce an encoded or compressed bitstream 420 using the video stream 300 as input: an intra/inter prediction stage 402, a transform stage 404, a quantization stage 406, and an entropy encoding stage 408. The encoder 400 may also include a reconstruction path (shown by the dotted connection lines) to reconstruct a frame for encoding of future blocks. In FIG. 4, the encoder 400 has the following stages to perform the various functions in the reconstruction path: a dequantization stage 410, an inverse transform stage 412, a reconstruction stage 414, and a loop filtering stage 416. Other structural variations of the encoder 400 can be used to encode the video stream 300.

[0068] When the video stream 300 is presented for encoding, respective adjacent frames 304, such as the frame 306, can be processed in units of blocks. At the intra/inter prediction stage 402, respective blocks can be encoded using intra-frame prediction (also called intraprediction) or inter-frame prediction (also called inter-prediction). In any case, a prediction block can be formed. In the case of intra-prediction, a prediction block may be formed from samples in the current frame that have been previously encoded and reconstructed. In the case of inter-prediction, a prediction block may be formed from samples in one or more previously constructed reference frames.

[0069] Next, the prediction block can be subtracted from the current block at the intra/inter prediction stage 402 to produce a residual block (also called a residual). The transform stage 404 transforms the residual into transform coefficients in, for example, the frequency domain using block-based transforms. The quantization stage 406 converts the transform coefficients into discrete quantum values, which are referred to as quantized transform coefficients, using a quantizer value or a quantization level. For example, the transform coefficients may be divided by the quantizer value and truncated.

[0070] The quantized transform coefficients are then entropy encoded by the entropy encoding stage 408. The entropy-encoded coefficients, together with other information used to decode the block (which may include, for example, syntax elements such as used to indicate the type of prediction used, transform type, motion vectors, a quantizer value, or the like), are then output to the compressed bitstream 420. The compressed bitstream 420 can be formatted using various techniques, such as variable length coding (VLC) or arithmetic coding. The compressed bitstream 420 can also be referred to as an encoded video stream or encoded video bitstream, and the terms will be used interchangeably herein.

[0071] The reconstruction path (shown by the dotted connection lines) can be used to ensure that the encoder 400 and a decoder 500 (described below with respect to FIG. 5) use the same reference frames to decode the compressed bitstream 420. The reconstruction path performs functions that are similar to functions that take place during the decoding process (described below with respect to FIG. 5), including dequantizing the quantized transform coefficients at the dequantization stage 410 and inverse transforming the dequantized transform coefficients at the inverse transform stage 412 to produce a derivative residual block (also called a derivative residual). At the reconstruction stage 414, the prediction block that was predicted at the intra/inter prediction stage 402 can be added to the derivative residual to create a reconstructed block. The loop filtering stage 416 can be applied to the reconstructed block to reduce distortion such as blocking artifacts.

[0072] Other variations of the encoder 400 can be used to encode the compressed bitstream 420. In some implementations, a non-transform based encoder can quantize the residual signal directly without the transform stage 404 for certain blocks or frames. In some implementations, an encoder can have the quantization stage 406 and the dequantization stage 410 combined in a common stage.

[0073] FIG. 5 is a block diagram of a decoder 500 according to implementations of this disclosure. The decoder 500 can be implemented in the receiving station 106, for example, by providing a computer software program stored in the memory 204. The computer software program can include machine instructions that, when executed by a processor such as the processor 202, cause the receiving station 106 to decode video data in the manner described in FIG. 5. The decoder 500 can also be implemented in hardware included in, for example, the transmitting station 102 or the receiving station 106.

[0074] The decoder 500, similar to the reconstruction path of the encoder 400 discussed above, includes in one example the following stages to perform various functions to produce an output video stream 516 from the compressed bitstream 420: an entropy decoding stage 502, a dequantization stage 504, an inverse transform stage 506, an intra/inter prediction stage 508, a reconstruction stage 510, a loop filtering stage 512, and a post filter stage 514. Other structural variations of the decoder 500 can be used to decode the compressed bitstream 420.

[0075] When the compressed bitstream 420 is presented for decoding, the data elements within the compressed bitstream 420 can be decoded by the entropy decoding stage 502 to produce a set of quantized transform coefficients. The dequantization stage 504 dequantizes the quantized transform coefficients (e.g., by multiplying the quantized transform coefficients by the quantizer value), and the inverse transform stage 506 inverse transforms the dequantized transform coefficients to produce a derivative residual that can be identical to that created by the inverse transform stage 412 in the encoder 400. Using header information decoded from the compressed bitstream 420, the decoder 500 can use the intra/inter prediction stage 508 to create the same prediction block as was created in the encoder 400 (e.g., at the intra/inter prediction stage 402). [0076] At the reconstruction stage 510, the prediction block can be added to the derivative residual to create a reconstructed block. The loop filtering stage 512 can be applied to the reconstructed block to reduce blocking artifacts. Examples of filters which may be applied at the loop filtering stage 512 include, without limitation, a deblocking filter, a directional enhancement filter, and a loop restoration filter. Other filtering can be applied to the reconstructed block. In this example, the post filter stage 514 is applied to the reconstructed block to reduce blocking distortion, and the result is output as the output video stream 516. The output video stream 516 can also be referred to as a decoded video stream, and the terms will be used interchangeably herein.

[0077] Other variations of the decoder 500 can be used to decode the compressed bitstream 420. In some implementations, the decoder 500 can produce the output video stream 516 without the post filter stage 514.

[0078] FIG. 6 is an illustration of examples of portions of a video frame 600, which may, for example, be the frame 306 shown in FIG. 3. The video frame 600 includes a number of 64x64 blocks 610, such as four 64x64 blocks 610 in two rows and two columns in a matrix or Cartesian plane, as shown. Each 64x64 block 610 may include up to four 32x32 blocks 620. Each 32x32 block 620 may include up to four 16x 16 blocks 630. Each 16x 16 block 630 may include up to four 8x8 blocks 640. Each 8x8 block 640 may include up to four 4x4 blocks 950. Each 4x4 block 950 may include 16 pixels, which may be represented in four rows and four columns in each respective block in the Cartesian plane or matrix. In some implementations, the video frame 600 may include blocks larger than 64x64 and/or smaller than 4x4. Subject to features within the video frame 600 and/or other criteria, the video frame 600 may be partitioned into various block arrangements.

[0079] The pixels may include information representing an image captured in the video frame 600, such as luminance information, color information, and location information. In some implementations, a block, such as a 16x 16 pixel block as shown, may include a luminance block 660, which may include luminance pixels 662; and two chrominance blocks 670, 680, such as a U or Cb chrominance block 670, and a V or Cr chrominance block 680. The chrominance blocks 670, 680 may include chrominance pixels 690. For example, the luminance block 660 may include 16x 16 luminance pixels 662 and each chrominance block 670, 680 may include 8x8 chrominance pixels 690 as shown. Although one arrangement of blocks is shown, any arrangement may be used. Although FIG. 6 shows NxN blocks, in some implementations, NxM blocks may be used, wherein N and M are different numbers. For example, 32x64 blocks, 64x32 blocks, 16x32 blocks, 32x 16 blocks, or any other size blocks may be used. In some implementations, N*2N blocks, 2N*N blocks, or a combination thereof, may be used.

[0080] In some implementations, coding the video frame 600 may include ordered blocklevel coding. Ordered block-level coding may include coding blocks of the video frame 600 in an order, such as raster-scan order, wherein blocks may be identified and processed starting with a block in the upper left corner of the video frame 600, or portion of the video frame 600, and proceeding along rows from left to right and from the top row to the bottom row, identifying each block in turn for processing. For example, the 64^64 block in the top row and left column of the video frame 600 may be the first block coded and the 64x64 block immediately to the right of the first block may be the second block coded. The second row from the top may be the second row coded, such that the 64x64 block in the left column of the second row may be coded after the 64x64 block in the rightmost column of the first row. [0081] In some implementations, coding a block of the video frame 600 may include using quad-tree coding, which may include coding smaller block units within a block in raster-scan order. For example, the 64x64 block shown in the bottom left corner of the portion of the video frame 600 may be coded using quad-tree coding wherein the top left 32x32 block may be coded, then the top right 32x32 block may be coded, then the bottom left 32x32 block may be coded, and then the bottom right 32x32 block may be coded. Each 32x32 block may be coded using quad-tree coding wherein the top left 16x 16 block may be coded, then the top right 16x 16 block may be coded, then the bottom left 16x 16 block may be coded, and then the bottom right 16x 16 block may be coded. Each 16x 16 block may be coded using quad-tree coding wherein the top left 8x8 block may be coded, then the top right 8x8 block may be coded, then the bottom left 8x8 block may be coded, and then the bottom right 8x8 block may be coded. Each 8x8 block may be coded using quad-tree coding wherein the top left 4x4 block may be coded, then the top right 4x4 block may be coded, then the bottom left 4x4 block may be coded, and then the bottom right 4x4 block may be coded. In some implementations, 8x8 blocks may be omitted for a 16x 16 block, and the 16x 16 block may be coded using quad-tree coding wherein the top left 4x4 block may be coded, then the other 4x4 blocks in the 16x 16 block may be coded in raster-scan order.

[0082] In some implementations, coding the video frame 600 may include encoding the information included in the original version of the image or video frame by, for example, omitting some of the information from that original version of the image or video frame from a corresponding encoded image or encoded video frame. For example, the coding may include reducing spectral redundancy, reducing spatial redundancy, or a combination thereof. Reducing spectral redundancy may include using a color model based on a luminance component (Y) and two chrominance components (U and V or Cb and Cr), which may be referred to as the YUV or YCbCr color model, or color space. Using the YUV color model may include using a relatively large amount of information to represent the luminance component of a portion of the video frame 600, and using a relatively small amount of information to represent each corresponding chrominance component for the portion of the video frame 600. For example, a portion of the video frame 600 may be represented by a high-resolution luminance component, which may include a 16x 16 block of pixels, and by two lower resolution chrominance components, each of which represents the portion of the image as an 8x8 block of pixels. A pixel may indicate a value, for example, a value in the range from 0 to 255, and may be stored or transmitted using, for example, eight bits. Although this disclosure is described in reference to the YUV color model, another color model may be used. Reducing spatial redundancy may include transforming a block into the frequency domain using, for example, a discrete cosine transform. For example, a unit of an encoder may perform a discrete cosine transform using transform coefficient values based on spatial frequency.

[0083] Although described herein with reference to matrix or Cartesian representation of the video frame 600 for clarity, the video frame 600 may be stored, transmitted, processed, or a combination thereof, in a data structure such that pixel values may be efficiently represented for the video frame 600. For example, the video frame 600 may be stored, transmitted, processed, or any combination thereof, in a two-dimensional data structure such as a matrix as shown, or in a one-dimensional data structure, such as a vector array. Furthermore, although described herein as showing a chrominance subsampled image where U and V have half the resolution of Y, the video frame 600 may have different configurations for the color channels thereof. For example, referring still to the YUV color space, full resolution may be used for all color channels of the video frame 600. In another example, a color space other than the YUV color space may be used to represent the resolution of color channels of the video frame 600.

[0084] FIG. 7 is an illustration of a frame-level prediction mode that is based on temporal interpolated picture (TIP). A frame-level prediction mode that is based on using a TIP is referred to herein as a TIP mode. A TIP reference frame is a reference frame generated by interpolating reference blocks from a forward reference frame and a backward reference frame. The TIP reference frame may be generated based on a motion field determined according to the frame-level non-linear motion offset, for example, by applying the frame- level non-linear motion offset against an initial motion field determined for the video frame. The updated motion field resulting from that process may then be used to generate the TIP reference frame, which may then be used as a spatially and temporally co-located reference frame to the video frame for predicting the video frame during inter-prediction.

[0085] A current frame 700 represents a frame under prediction using the TIP mode for example, during encoding (e.g., at the intra/inter prediction stage 402) or decoding (e.g., at the intra/inter prediction stage 510). A TIP reference frame 702 is generated using a motion field based on a backward reference frame 704 and a forward reference frame 706. For example, where the current frame 700 is denoted as Fi, the backward reference frame 704 can be denoted as Fi-i and the forward reference frame 706 can be denoted at Fi+i. A temporal motion vector predictor 708 represents a motion vector predictor pointing from the forward reference frame 706 to the backward reference frame 704. Amotion vector 710 pointing from the current frame 700 to the TIP reference frame 702 represents the motion vector which may be used with the TIP reference frame 702 to predict the motion within one or more blocks of the current frame 700. For example, the temporal motion vector predictor 708 may be a motion vector predictor for the motion vector 710.

[0086] That blocks of the current frame 700 are to be coded using the TIP reference frame 702 can be indicated in a header of the current frame 700. For example, an encoder, such as the encoder 400 of FIG. 4, may encode in the header of the current frame 700 included in a compressed bitstream, such as the compressed bitstream 420 of FIG. 4, and a decoder, such as the decoder 500 of FIG. 5, may decode from the header, one or more syntax elements indicating that blocks of the current frame 700 are coded using TIP mode (e.g., using the TIP reference frame 702).

[0087] As mentioned above, that a block of the current frame 700 is coded using the frame-level prediction mode (e.g., using the TIP reference frame 702 in this case) can mean that no residuals are coded for the block. However, according to implementations of this disclosure, at least some (e.g., few) blocks (i.e., residue-based blocks) of the current frame 700 are coded with residuals. That is, the encoder may include respective residuals for the residue-based blocks in the compressed bitstream. Including residuals (e.g., residual blocks) in a compressed bitstream can be as described with respect to FIG. 4.

[0088] In an example, the encoder may encode (and the decoder may decode) explicit indications of the blocks (i.e., the residue-based blocks) associated with residual blocks in the compressed bitstream. The indications of such blocks may be coded in the header of the current frame 700. In an example, the respective headers of at least some of the residue-based blocks may include additional mode information, such as which respective transform types to be applied, whether deblocking filters are to be applied post reconstructions, and/or whether other filters are to be applied. In an example, the respective headers may also include respective partitioning information into prediction subblocks or transform subblocks.

[0089] In an example, which blocks are associated with residuals can be inferred and need not be explicitly coded. Calculating the motion field as described herein can result in some blocks of the current frame 700 not being associated with motion vectors. That is, no projection operations may pass through some blocks of the current frame. Blocks that are not intersected by motion vectors may be referred to herein as orphan blocks. Said another way, the projection operations described herein may result in “holes” in the motion field. The orphan blocks can be residue-based blocks.

[0090] Motion information associated with each of the orphan blocks may be obtained from respective neighboring blocks of the orphan blocks. In one implementation, the motion vector of a nearest spatial neighbor to an orphan block in the current frame 700 is assigned to the orphan block. In another implementation, a multi-hypothesis technique to fill the holes in the motion field can be used. The multi-hypothesis technique can be used to identify motion vectors for orphan blocks. Instead of directly re-using the nearest available motion vectors, at least some (e.g., all) of the candidate motion vectors of the neighboring blocks can be evaluated. For each candidate motion vector, the candidate motion vector can be scaled according to the distance of reference frames of the current block in both directions. The scaled motion vectors can be used to identify (e.g., retrieve or fetch) two reference blocks, referred to as REF_BL0CKQ and REF_BLOCK1. Amongst the candidate motion vectors, the candidate motion vector that results in the minimal difference between REF_BL0CKQ and REF_BLOCK e.g. abs(REF_BLOCKo - REF_BLOCK1) can be used to fill the missing hole in the motion field.

[0091] FIGS. 8A-D are illustrations of frame-level prediction modes that are global motion models (GMMs). A frame-level prediction mode that is based on using a GMM is referred to herein as a GMM mode.

[0092] As is known, not all motion within a frame can be described and/or efficiently described using translational motion models with respect to a reference frame. For example, some motion may include scaling, shearing, or rotating motion, either alone or with translational motion. Such motion can be attributed, for example, to camera motion and is applicable to all, or at least many, blocks of a frame. As such, the motion is “global” to a frame.

[0093] As mentioned, and further described below, the global motion can itself be a translational motion. As such, predicting blocks of a current frame using a translational global motion model can result in better performance (e.g., improved compression) than using local translational motion at the block level (i.e., regular motion compensation). In encoding blocks using inter prediction, the global motion may be used to produce a reference block in a reference frame. Global motion may be represented by a “parameterized motion model” or “motion model.”

[0094] The encoder may include, such as in the frame header of a current frame, the parameters of the GMM and an indication of a reference frame to be used in association with the GMM. The decoder may decode such parameters and indication of the reference frame. In another example, the decoder may include a codebook of geometric transformations and one or more syntax elements included in the frame header of the current frame can indicate which of the geometric transformations is to be applied. In an example, a geometric transformation may be according to an affine model. Affine transformation is a linear transform between the coordinates of two spaces that is determined by six affine coefficients. While the affine transformation may include translational motion, it can also encompass scaling, rotation and shearing. Therefore, an affine motion model is able to capture more complex motion than the conventional translational model.

[0095] The affine transformation model can project a pixel at (x, y) of the current block to a prediction pixel at (%', y ') in a reference frame through formula (1). In formula (1), the tuple (c, ) corresponds to a translational action; the parameters a and e can be used to control the scaling factors in the vertical and horizontal axes, and in conjunction with the parameters b and d decide (e.g., determine, set, etc.) a rotation angle.

[0096] FIGS. 8A-D depict different motion model types used to project pixels of a block of a frame to a warped patch within a reference frame according to the GMM associated with the frame as a whole. The warped patch can be used to generate a prediction block for encoding or decoding that block. A parameterized motion model indicates how the pixels of a block are to be scaled, rotated, or otherwise moved when projected into the reference frame. Data indicative of pixel projections can be used to identify parameterized motion models corresponding to a respective motion model. The number and function of the parameters of a parameterized motion model depend upon the specific projection used.

[0097] In FIG. 8 A, pixels of a block 802A are projected to a warped patch 804A of a frame 800 A using a homographic motion model. A homographic motion model uses eight parameters to project the pixels of the block 802A to the warped patch 804A. A homographic motion is not bound by a linear transformation between the coordinates of two spaces. As such, the eight parameters that define a homographic motion model can be used to project pixels of the block 802A to a quadrilateral patch (e.g., the warped patch 804A) within the frame 800A. Homographic motion models thus support translation, rotation, scaling, changes in aspect ratio, shearing, and other non-parallelogram warping. A homographic motion between two spaces is defined as follows: a*X+b*Y+c , d*X+e*Y+f x = - ; and v = - . g*X+h*Y+l ^J g*X+h*Y+l

[0098] In these equations, (x, y) and (X, Y) are coordinates of two spaces, namely, a projected position of a pixel within the frame 800 A and an original position of a pixel within the block 802A, respectively. Further, a, b, c, d, e, f, g, and h are the homographic parameters and are real numbers representing a relationship between positions of respective pixels within the frame 800A and the block 802A. Of these parameters, a represents a fixed scale factor along the x-axis with the scale of the y-axis remaining unchanged, b represents a scale factor along the x-axis proportional to the y-distance to a center point of the block, c represents a translation along the x-axis, d represents a scale factor along the y-axis proportional to the x- distance to the center point of the block, e represents a fixed scale factor along the y-axis with the scale of the x-axis remaining unchanged, f represents a translation along the y-axis, g represents a proportional scale of factors of the x- and y-axes according to a function of the x- axis, and h represents a proportional scale of factors of the x- and y-axes according to a function of the y-axis.

[0099] In FIG. 8B, pixels of a block 802B are projected to a warped patch 804B of a frame 800B using an affine motion model. An affine motion model uses six parameters to project the pixels of the block 802B to the warped patch 804B. An affine motion is a linear transformation between the coordinates of two spaces defined by the six parameters. As such, the six parameters that define an affine motion model can be used to project pixels of the block 802B to a parallelogram patch (e.g., the warped patch 804B) within the frame 800B. Affine motion models thus support translation, rotation, scale, changes in aspect ratio, and shearing. The affine projection between two spaces is defined as follows: x = a * X + b * Y + c; and y = d * X + e * Y + f.

[0100] In these equations, (x, y) and (X, Y) are coordinates of two spaces, namely, a projected position of a pixel within the frame 800B and an original position of a pixel within the block 802B, respectively. Also, a, b, c, d, e, and f are affine parameters and are real numbers representing a relationship between positions of respective pixels within the frame 800B and the block 802B. Of these, a and d represent rotational or scaling factors along the x-axis, b and e represent rotational or scaling factors along the y-axis, and c and f respectively represent translation along the x- and y-axes.

[0101] In FIG. 8C, pixels of a block 802C are projected to a warped patch 804C of a frame 800C using a similarity motion model. A similarity motion model uses four parameters to project the pixels of the block 802C to the warped patch 804C. A similarity motion is a linear transformation between the coordinates of two spaces defined by the four parameters. For example, the four parameters can be a translation along the x-axis, a translation along the y-axis, a rotation value, and a zoom value. As such, the four parameters that define a similarity motion model can be used to project pixels of the block 802C to a square patch (e.g., the warped patch 804C) within the frame 800C. Similarity motion models thus support square-to-square transformation with rotation and zoom.

[0102] In FIG. 8D, pixels of a block 802D are projected to a warped patch 804D of a frame 800D using a translational motion model. A translational motion model uses two parameters to project the pixels of the block 802D to the warped patch 804D. A translational motion is a linear transformation between the coordinates of two spaces defined by the two parameters. For example, the two parameters can be a translation along the x-axis and a translation along the y-axis. As such, the two parameters that define a translational motion model can be used to project pixels of the block 802D to a square patch (e.g., the warped patch 804D) within the frame 800D.

[0103] As mentioned above with respect to FIG. 7, in an example, a compressed bitstream can include an explicit indication of blocks (i.e., residue-based blocks) for which residuals are encoded in the compressed bitstream. In another example, the residue-based blocks can be inferred.

[0104] In an example, all boundary blocks can be inferred to have residual blocks associated therewith (e.g., encoded therefor). That is, all blocks that are along the left, top, right, and bottom of the current frame can be inferred to have associated therewith residual blocks. In an example, at least a subset of the boundary blocks can be determined to have residual blocks associated therewith. In an example, a decoder may analyze the parameters of the GMM to infer the blocks of the current frame associated with residuals. For example, if a prediction block for a current block is determined to include at least some pixels that are outside of the reference frame, then the current block is determined to be associated with a residual block (i.e., the current block is determined to be a residue-based block).

[0105] FIG. 9 is a flowchart of a technique 900 for coding blocks of a current frame based on a frame-level prediction mode. The technique 900 can be implemented, for example, as a software program that may be executed by computing devices such as transmitting station 102 or receiving station 106. The software program can include machine- readable instructions that may be stored in a memory such as the memory 204 or the secondary storage 214, and that, when executed by a processor, such as the processor 202, may cause the computing device to perform the technique 900. The technique 900 may be implemented in whole or in part in the intra/inter prediction stage 402 of the encoder 400 of FIG. 4 and/or the intra/inter prediction stage 508 of the decoder 500 of FIG. 5. The technique 900 can be implemented using specialized hardware or firmware. Multiple processors, memories, or both, may be used. When the technique 900 is implemented by an encoder, the term “coding” includes encoding, such as in the compressed bitstream; and when the technique 900 is implemented by a decoder, the term “coding” includes decoding, such as from the compressed bitstream.

[0106] While not specifically shown in FIG. 9, the current frame may be partitioned into blocks of uniform size. That is, the current frame can be partitioned into fixed-size blocks. The uniform block size can be 32^32, 64x64, 128x 128, or some other size. In an example, the size of the blocks may be coded in the header of the current frame. To illustrate, and without limitations, the header may include a syntax element indicating the uniform block size. A two-bit syntax element may be used to indicate the block size. To illustrate, and without limitations, 00 may indicate a block size of 32x32, 01 may indicate a block size of 64x64, and 10 may indicate a block size of 128x 128. The decoder may decode the syntax element to determine the uniform block size. In another example, the block size may be predetermined (e.g., pre-configured) at the encoder and decoder. In an example, a residue-based block may be further partitioned based on a quad-tree partitioning. For example, a residuebased block may be partitioned into multiple transform blocks.

[0107] At 902, a frame-level prediction mode is coded. For example, the encoder may determine, such as based on a rate-distortion analysis, that the current frame is to be encoded based on a frame-level prediction mode. The encoder may determine that a substantial number of blocks of the current frame are to be encoded using the same frame-level prediction mode. As such, the encoder may encode, such as in a header of the current frame, an indication of the frame-level prediction mode. The decoder may decode the indication of the frame-level prediction mode from the header of the current frame. To illustrate, and without limitations, a flag may indicate whether the blocks of the current frame are coded using a frame-level prediction mode; and a second flag may indicate whether the mode is the TIP mode, the GMM mode, or some other frame-level prediction mode. If the mode is the GMM mode, then the model parameters and reference frame are then coded. The frame header may further indicate whether the frame is coded using direct frame prediction without residual coding, using direct frame prediction with residual coding, or using a traditional block-based coding. That a frame is coded using one of these modes can mean that at least most of the blocks of the frame (e.g., the residue-less blocks) are coded using that mode.

[0108] As such, in an example, the frame-level prediction mode can be the global-motion prediction mode (i.e., the GMM mode). The encoder can encode, and the decoder can decode, the parameters of the global-motion model from the frame header. In an example, the framelevel prediction mode can be the TIP mode. That is, the frame-level prediction mode can be based on temporal interpolated picture.

[0109] At 904, first blocks (i.e., residue-less blocks) of the current frame are coded using the frame-level prediction mode. That is, no residual blocks are associated with (i.e., coded for) any of the first blocks. As no residuals are associated with the first blocks, then no transform-related operation (transforming into the frequency domain at the encoder, and inverse-transforming from the frequency domain back into the pixel domain at the decoder) are performed therewith. Additionally, loop filtering, such as described with respect to loop filtering stage 416 of FIG. 4 and the loop filtering stage 512 may not be applied to, or performed with respect to, the first blocks. As already described above, no coding mode information are encoded in, or decoded from, respective headers of the first blocks.

[0110] At 906, one or more second blocks (i.e., residue-based blocks) of the current frame are coded. Coding each of the one or more second blocks includes, at 906 2, coding a respective residual block for the block. As described above, when the technique 900 is performed at the encoder, coding a residual block includes encoding transform coefficients into the compressed bitstream; and when the technique 900 is performed at the decoder, coding a residual block includes decoding transform coefficients from the compressed bitstream.

[OHl] In an example, the one or more second blocks may be coded based on the same frame-level prediction mode. That is, respective prediction blocks for the one or more second blocks are obtained using the frame-level prediction mode. In an example, at least one of the one or more second blocks may be coded using a prediction mode that is different from the frame-level prediction mode. As such, the prediction mode of the at least one of the one or more second blocks may be coded in respective headers of the residue-based blocks.

[0112] As described above, the technique 900 may identify respective locations of the one or more second blocks (i.e., the residue-based blocks). That is, the technique 900 identifies which of the blocks of the current frame are residue-based blocks. In an example, the frame header of the current frame may include information indicating which of the blocks of the current frame are the residue-based blocks. Many different ways may be available for identifying the residue-based blocks. To illustrate, and without limitations, a bit may be associated with the blocks of the current frame. As such, the encoder may encode a bitstring indicating which of the blocks, in a raster scan order, are residue-based blocks. To illustrate, the bitstring 01001 .. ., may indicate that the first, third, and fourth blocks are residue-less blocks and that the second and fifth are residue-based blocks. The bitstring may be coded using run-length encoding. However, other techniques for coding the bitstring are possible. [0113] As such, identifying the respective locations of the one or more second blocks (i.e., the residue-based blocks) can include decoding, from the compressed bitstream, the respective locations of the one or more second blocks. Identifying the respective locations of the one or more second blocks (i.e., the residue-based blocks) equivalently means identifying which of the blocks of the current frame are residue-based blocks.

[0114] In an example, the locations of residue-based blocks may be inferred. The locations of residue-based blocks may be inferred based on the frame-level prediction mode. The locations of residue-based blocks may be inferred as described with respect to FIGS. 7 and 8A-8D. In an example, the technique 900 may infer locations of some of the residuebased blocks and may code the locations of other residue-based blocks in the frame header. [0115] A codec may include a set of available transform types. To illustrate, and without limitations, the set may include 16 different transform types, as further described below. Typically, a decoder may decode a transform type (or more accurately, an inverse transform type) from a block header. In an example, direct frame prediction with residual coding may use a pre-determined (e.g., pre-configured or fixed) transform type for residue-based blocks. To illustrate, and without limitations, the pre-determined transform type may be the DCT DCT transform type. In an example, the pre-determined transform type may be based on (e.g., depends on) the frame-level prediction mode. In another example, only a subset of the available transform types may be used for residue-based blocks. In an example, the subset of the available transform types may depend on the frame-level prediction type. To illustrate, a first subset may be used with the TIP mode; and a second, different, subset may be used with the GMM mode. As such, coding a residue-based block can include coding a transform type, such as in the header of the residue-based block. The encoder may encode the transform type in the block header and the decoder uses the transform type to obtain (such as by inverse transforming the transform coefficients) the residual block of the residue-based block. If a residue-based block is further partitioned into subblocks, then respective transform types may be coded for (e.g., with respect to) at least some of the subblocks.

[0116] As mentioned above, a residue-based block may be partitioned into one or more transform blocks. That a residue-based block is partitioned into one transform block means that the transform block is co-extensive with the residue-based block. A transform block can have associated therewith a transform type.

[0117] The transform type can include a horizontal transform type (e.g., a kernel) to be applied to the rows of the transform block and a vertical transform type (e.g., a kernel) to be applied to the columns of the transform block, independently. A separable two-dimensional (2D) transform process can be applied to prediction residuals. For the forward transform (e.g., at an encoder), a one-directional (ID) vertical transform is first performed on each column of the input residual block, then a horizontal transform is performed on each row of the vertical transform output. For the backward transform (e.g., at a decoder), a ID horizontal transform is first performed on each row of the input dequantized coefficient block, then a vertical transform is performed on each column of the horizontal transform output.

[0118] In an example, the transform kernels available in a codec, such as the AVI codec, may include four different types of transforms: a discrete cosine transforms (DCT), an asymmetric discrete cosine transforms (ADST), a flipped version of the ADST (FLIPADST), and an identity transform (IDT). Each of these transforms (i.e., kernels) may be available at different points. For example, 4-, 8-, 16-, 32-, and 64-point DCT kernels may be available; 4- , 8-, and 16-point ADST and FLIPADST kernels may be available; and 4-, 8-, 16-, and 32- point identity transforms (IDTs) may be available. Again, more, fewer, or other kernels are possible.

[0119] The DCT kernel is widely used in signal compression and is known to approximate the optimal linear transform, the Karhunen-Loeve transform (KLT), for consistently correlated data. The ADST, on the other hand, approximates the KLT where one-sided smoothness is assumed and can be naturally suitable for coding, inter alia, some intra-prediction residuals. Similarly, the FLIPADST can capture one-sided smoothness from the opposite end. The IDT can be used to accommodate situations where sharp transitions are contained in the block and where neither DCT nor ADST is effective. Also, the IDT, combined with other 1-D transforms, provides the 1-D transforms themselves, therefore allowing for better compression of horizontal and vertical patterns in the residual.

[0120] Accordingly, the available transform types include sixteen 2D transforms comprising combinations of four ID transforms as follows: DCT DCT (transform rows with DCT and columns with DCT), ADST DCT (transform rows with ADST and columns with DCT), DCT ADST (transform rows with DCT and columns with ADST), ADST ADST (transform rows with ADST and columns with ADST), FLIPADST DCT (transform rows with FLIP ADST and columns with DCT), DCT FLIPADST (transform rows with DCT and columns with FLIP ADST), FLIPAD ST FLIP AD ST (transform rows with FLIP ADST and columns with FLIP ADST), ADST FLIPADST (transform rows with ADST and columns with FLIP ADST), FLIPADST ADST (transform rows with FLIP ADST and columns with ADST), IDT (transform rows with identity and columns with identity), V DCT (transform rows with identity and columns with DCT), H DCT (transform rows with DCT and columns with identity), V ADST (transform rows with identity and columns with ADST), H ADST (transform rows with ADST and columns with identity), V FLIPADST (transform rows with identity and columns with FLIP ADST), and H FLIPADST (transform rows with FLIP ADST and columns with identity).

[0121] In an example, a predefined transform size may be used to code a residue-based block. As such, the residue-based block can be partitioned into transform blocks each equaling in size to predefined transform size. In an example, the predefined transform size can be the smallest transform block size supported by the codec. In an example, the predefined transform size can be the largest transform block size supported by the codec. In an example, the predefined transform size can be some other fixed transform block size. In another example, the transform block size may be signaled in the compressed bitstream. In an example, the transform block size may be signaled in the header of the residue-based block. In another example, the transform block size may be signaled in the frame header. As such, the signaled transform block size may be used for all residue-based blocks of the current frame.

[0122] A deblocking filter may be applied to at least one of the one or more reconstructed blocks. As mentioned above, a residue-based block may be further partitioned into subblocks. As such, the deblocking filter may also be applied to each of the subblocks. When the technique 900 is implemented at an encoder, deblocking may be performed as described with respect to the loop filtering stage 416 of FIG. 4; and when the technique 900 is implemented at a decoder, deblocking may be performed as described with respect to the loop filtering stage 512 of FIG. 5.

[0123] In an example, post loop filters may be applied to at least some of the residuebased blocks (or subblocks thereof) after reconstruction, such as described with respect to the post filter stage 514 of FIG. 5. A codec may include or implement several post-loop filters, which may be referred to as restoration filters. To illustrate, and without limitations, the AVI codec implements a constrained directional enhancement filter (CDEF) and a Wiener filter. [0124] Which, if any, additional restoration filter (which may be more than one post-loop filter) is to be applied to enhance the reconstruction may be indicated in the block header of a residue-based block. In another example, the post-loop filter may be pre-determined. In another example, the post-loop filter may be based on (e.g., selected based on) the framelevel prediction mode.

[0125] A restoration filter may be associated with (e.g., signaled at) a restoration unit level. A restoration unit refers to a portion of a frame with which a restoration filter is associated. In an example, the restoration unit can be 256x256 luma pixels. In another example, the restoration filter to be applied at a block level may not have associated therewith any block-level signaling. For example, a deblocking filter strength may be signaled at the frame level (e.g., in the frame header), then the deblocking filter may be applied adaptively to a residue-based block based on the prediction types of the block, whether transform coding is skipped or not, or both.

[0126] For simplicity of explanation, the technique 900 of FIG. 9 is depicted and described as respective series of steps or operations. However, the steps or operations in accordance with this disclosure can occur in various orders and/or concurrently. Additionally, other steps or operations not presented and described herein may be used. Furthermore, not all illustrated steps or operations may be required to implement a technique in accordance with the disclosed subject matter.

[0127] The aspects of encoding and decoding described above illustrate some examples of encoding and decoding techniques. However, it is to be understood that encoding and decoding, as those terms are used in the claims, could mean compression, decompression, transformation, or any other processing or change of data.

[0128] The word “example” is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as “example” is not necessarily to be construed as being preferred or advantageous over other aspects or designs. Rather, use of the word “example” is intended to present concepts in a concrete fashion. As used in this application, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless specified otherwise or clearly indicated otherwise by the context, the statement “X includes A or B” is intended to mean any of the natural inclusive permutations thereof. That is, if X includes A; X includes B; or X includes both A and B, then “X includes A or B” is satisfied under any of the foregoing instances. In addition, the articles “a” and “an” as used in this application and the appended claims should generally be construed to mean “one or more,” unless specified otherwise or clearly indicated by the context to be directed to a singular form. Moreover, use of the term “an implementation” or the term “one implementation” throughout this disclosure is not intended to mean the same embodiment or implementation unless described as such.

[0129] Implementations of the transmitting station 102 and/or the receiving station 106 (and the algorithms, methods, instructions, etc., stored thereon and/or executed thereby, including by the encoder 400 and the decoder 500) can be realized in hardware, software, or any combination thereof. The hardware can include, for example, computers, intellectual property (IP) cores, application-specific integrated circuits (ASICs), programmable logic arrays, optical processors, programmable logic controllers, microcode, microcontrollers, servers, microprocessors, digital signal processors, or any other suitable circuit. In the claims, the term “processor” should be understood as encompassing any of the foregoing hardware, either singly or in combination. The terms “signal” and “data” are used interchangeably. Further, portions of the transmitting station 102 and the receiving station 106 do not necessarily have to be implemented in the same manner.

[0130] Further, in one aspect, for example, the transmitting station 102 or the receiving station 106 can be implemented using a general purpose computer or general purpose processor with a computer program that, when executed, carries out any of the respective methods, algorithms, and/or instructions described herein. In addition, or alternatively, for example, a special purpose computer/processor can be utilized which can contain other hardware for carrying out any of the methods, algorithms, or instructions described herein. [0131] The transmitting station 102 and the receiving station 106 can, for example, be implemented on computers in a video conferencing system. Alternatively, the transmitting station 102 can be implemented on a server, and the receiving station 106 can be implemented on a device separate from the server, such as a handheld communications device. In this instance, the transmitting station 102, using an encoder 400, can encode content into an encoded video signal and transmit the encoded video signal to the communications device. In turn, the communications device can then decode the encoded video signal using a decoder 500. Alternatively, the communications device can decode content stored locally on the communications device, for example, content that was not transmitted by the transmitting station 102. Other suitable transmitting and receiving implementation schemes are available. For example, the receiving station 106 can be a generally stationary personal computer rather than a portable communications device, and/or a device including an encoder 400 may also include a decoder 500.

[0132] Further, all or a portion of implementations of the present disclosure can take the form of a computer program product accessible from, for example, a computer-usable or computer-readable medium. A computer-usable or computer-readable medium can be any device that can, for example, tangibly contain, store, communicate, or transport the program for use by or in connection with any processor. The medium can be, for example, an electronic, magnetic, optical, electromagnetic, or semiconductor device. Other suitable mediums are also available.

[0133] The above-described embodiments, implementations, and aspects have been described in order to facilitate easy understanding of this disclosure and do not limit this disclosure. On the contrary, this disclosure is intended to cover various modifications and equivalent arrangements included within the scope of the appended claims, which scope is to be accorded the broadest interpretation as is permitted under the law so as to encompass all such modifications and equivalent arrangements.

Claims

What is claimed is:

1. A method for coding blocks of a current frame, comprising: coding a frame-level prediction mode in a header of the current frame; coding first blocks of the current frame using the frame-level prediction mode, wherein no residual blocks are associated with any of the first blocks; and coding one or more second blocks of the current frame, wherein coding the one or more second blocks comprises: coding respective residual blocks for the one or more second blocks.

2. The method of claim 1, wherein the current frame is partitioned into fixed-size blocks.

3. The method of claim 1, wherein a partitioning of at least one of the one or more second blocks is identified based on a quad-tree partitioning.

4. The method of any one of claims 1 to 3, wherein coding the one or more second blocks of the current frame comprises: coding the one or more second blocks of the current frame based on the frame-level prediction mode.

5. The method of any one of claims 1 to 3, wherein coding the one or more second blocks of the current frame comprises: coding one of the one or more second blocks based on a prediction mode coded in a header of the one of the one or more second blocks.

6. The method of any one of claims 1 to 5, further comprising: identifying respective locations of the one or more second blocks.

7. The method of claim 6, wherein identifying the respective locations of the one or more second blocks comprises: inferring the respective locations of the one or more second blocks based on the frame-level prediction mode.

8. The method of claim 6, wherein identifying the respective locations of the one or more second blocks comprises: decoding, from a compressed bitstream, the respective locations of the one or more second blocks.

9. The method of any one of claims 1 to 8, wherein the frame-level prediction mode is a global-motion prediction mode, the method further comprising: coding parameters of the global-motion prediction mode.

10. The method of any one of claims 1 to 9, further comprising: inferring locations of the one or more second blocks based on the frame-level prediction mode.

11. The method of any one of claims 1 to 8 or 10, wherein the frame-level prediction mode is based on temporal interpolated picture.

12. The method of any one of claims 1 to 11, wherein at least some of the one or more second blocks of the current frame are coded using the frame-level prediction mode.

13. The method of any one of claims 1 to 12, further comprising: applying a deblocking filter to at least one of the one or more second blocks.

14. The method of any one of claims 1 to 13, wherein coding the one or more second blocks comprises: coding the one or more second blocks based on a fixed transform type.

15. The method of any one of claims 1 to 13, wherein coding the one or more second blocks comprises: identifying a transform type, from a set of available transform types, for coding at least one of the one or more second blocks based on a fixed transform type.

16. The method of any one of claims 1 to 15, wherein coding the one or more second blocks comprises: coding the one or more second blocks based on a predefined transform size.

17. The method of claim 16, wherein the predefined transform size is one of a smallest transform block size, a largest transform block size, or a fixed transform block size.

18. The method of any one of claims 1 to 13, wherein coding the one or more second blocks comprises: identifying, for a block of the one or more second blocks, signaled transform block sizes for coding the block.

19. A device, comprising: a processor that is configured to perform the method of any of claims 1-18.

20. A device, comprising: a memory; and a processor, the processor configured to execute instructions stored in the memory to perform the method of any of claims 1-18.

21. A non-transitory computer-readable storage medium, comprising executable instructions that, when executed by a processor, facilitate performance of operations, comprising operations that perform the method of any of claims 1-18.

22. A non-transitory computer-readable storage medium having stored thereon an encoded bitstream, wherein the encoded bitstream is configured for decoding by the method of any of claims 1-18.

23. A non-transitory computer-readable storage medium having stored thereon an encoded bitstream, wherein the encoded bitstream is generated by an encoder performing the method of any of claims 1-7 and 9-17.