WO2025206716A1 - Method and apparatus for matrix-based intra prediction on basis of various reference pixels - Google Patents

Abstract

Disclosed are a method and an apparatus for matrix-based intra prediction on the basis of various reference pixels. According to the disclosed method, coding efficiency and prediction accuracy can be improved by using various reference pixels for MIP prediction.

Description

Matrix-based intra prediction method and device based on various reference pixels

The present disclosure relates to a video encoding/decoding method, device, and recording medium for storing a bitstream, and more specifically, to a matrix-based intra prediction method and device based on various reference pixels.

The content described below merely provides background information related to the present embodiment and does not constitute prior art.

Since video data has a large amount of data compared to voice data or still image data, it requires a lot of hardware resources, including memory, to store or transmit it without processing for compression.

Therefore, when storing or transmitting video data, the encoder compresses the video data and stores or transmits it, and the decoder receives the compressed video data, decompresses it, and plays it back. These video compression technologies include H.264/AVC, HEVC (High Efficiency Video Coding), and VVC (Versatile Video Coding), which improves encoding efficiency by about 30% compared to HEVC.

However, as the size, resolution, and frame rate of images are gradually increasing, and the amount of data that needs to be encoded is also increasing, a new compression technology that has better encoding efficiency and better image quality improvement than existing compression technologies is required.

In particular, the demand for high-resolution, high-quality images, such as HD (High Definition) images and UHD (Ultra High Definition) images, has been increasing in various fields. As image data becomes higher in resolution and quality, the amount of information or bits transmitted increases relatively compared to existing image data. This increase in the amount of information or bits transmitted leads to an increase in transmission and storage costs. In addition, interest in and demand for immersive media, such as VR (Virtual Reality), AR (Artificial Reality) content, and holograms, has been increasing recently, and the broadcasting of videos/images with different image characteristics from real images, such as game images, is increasing. Accordingly, a highly efficient image compression technology is required to effectively compress, transmit, store, and play information of high-resolution, high-quality images with various characteristics as described above.

The present disclosure provides a method and device for image encoding or decoding for efficiently encoding or decoding an image and improving objective and subjective image quality of a restored image, and a recording medium for storing a bitstream generated by the image encoding method/device.

One aspect of the present disclosure provides an image decoding method, comprising: determining a reference template including decoded pixels; determining a weight matrix and an offset for matrix-based intra prediction of a current block; and generating a prediction block of the current block based on the reference template of the current block, the weight matrix, and the offset.

One aspect of the present disclosure provides an image encoding method, comprising: determining a reference template including encoded pixels; determining a weight matrix and an offset for matrix-based intra prediction of a current block; and generating a prediction block of the current block based on the reference template of the current block, the weight matrix, and the offset.

One aspect of the present disclosure provides a non-transitory computer-readable recording medium storing a bitstream generated by an image encoding device. The bitstream is generated by an image encoding method. The image encoding method includes the steps of: determining a reference template including encoded pixels; determining a weight matrix and an offset for matrix-based intra prediction of a current block; and generating a prediction block of the current block based on the reference template of the current block, the weight matrix, and the offset.

One aspect of the present disclosure provides a method comprising: generating a bitstream for an image, the method comprising: performing at least one processor; and transmitting data including the bitstream. The step of generating the bitstream comprises: determining a reference template including encoded pixels; determining a weight matrix and an offset for matrix-based intra prediction of a current block; and generating a prediction block of the current block based on the reference template of the current block, the weight matrix, and the offset.

According to the present disclosure, an image encoding/decoding device can efficiently encode/decode an image and improve the objective/subjective image quality of a restored image by diversifying reference pixels used for MIP prediction.

FIG. 1 is an exemplary block diagram of an image encoding device capable of implementing the techniques of the present disclosure.

Figure 2 is a drawing for explaining a method of dividing a block using the QTBTTT (QuadTree plus BinaryTree TernaryTree) structure.

FIGS. 3A and 3B are diagrams illustrating multiple intra prediction modes, including wide-angle intra prediction modes.

Figure 4 is an example diagram of the surrounding blocks of the current block.

FIG. 5 is an exemplary block diagram of an image decoding device capable of implementing the techniques of the present disclosure.

FIG. 6 is a flowchart of a matrix-based intra prediction method according to one embodiment of the present disclosure.

FIGS. 7A, 7B, and 7C are diagrams illustrating reference templates adjacent to a current block according to one embodiment of the present disclosure.

FIG. 8 is a drawing for explaining a reference template that is not adjacent to a current block according to one embodiment of the present disclosure.

FIG. 9 is a drawing for explaining a reference template indicated by movement information of a surrounding block according to one embodiment of the present disclosure.

FIG. 10 is a flowchart of an image encoding method according to one embodiment of the present disclosure.

FIG. 11 is a flowchart of an image decoding method according to one embodiment of the present disclosure.

Hereinafter, some embodiments of the present disclosure will be described in detail with reference to exemplary drawings. When designating components in each drawing, it should be noted that, where possible, identical components are given the same reference numerals, even if they appear in different drawings. Furthermore, in describing the present embodiments, detailed descriptions of related known structures or functions will be omitted if they are deemed to obscure the gist of the present embodiments.

FIG. 1 is an exemplary block diagram of an image encoding device capable of implementing the techniques of the present disclosure. Hereinafter, the image encoding device and its subcomponents will be described with reference to the illustration in FIG. 1.

The video encoding device may be configured to include a picture segmentation unit (110), a prediction unit (120), a subtractor (130), a transformation unit (140), a quantization unit (145), a reordering unit (150), an entropy encoding unit (155), an inverse quantization unit (160), an inverse transformation unit (165), an adder (170), a loop filter unit (180), and a memory (190).

Each component of the video encoding device may be implemented in hardware, software, or a combination of hardware and software. Furthermore, the functions of each component may be implemented in software, with a microprocessor executing the software functions corresponding to each component.

A single image (video) is composed of one or more sequences containing multiple pictures. Each picture is divided into multiple regions, and encoding is performed for each region. For example, a single picture is divided into one or more tiles and/or slices. Here, one or more tiles can be defined as a tile group. Each tile or slice is divided into one or more Coding Tree Units (CTUs). Each CTU is then divided into one or more Coding Units (CUs) by a tree structure. Information applied to each CU is encoded as the syntax of the CU, and information commonly applied to CUs included in a CTU is encoded as the syntax of the CTU. In addition, information commonly applied to all blocks within a single slice is encoded as the syntax of the slice header, and information applied to all blocks constituting one or more pictures is encoded in the Picture Parameter Set (PPS) or the picture header. Furthermore, information commonly referenced by multiple pictures is encoded in a Sequence Parameter Set (SPS). And, information commonly referenced by one or more SPS is encoded in a Video Parameter Set (VPS). In addition, information commonly applied to one tile or tile group may be encoded as syntax of a tile or tile group header. Syntaxes included in an SPS, PPS, slice header, tile or tile group header may be referred to as high level syntax.

The picture segmentation unit (110) determines the size of the CTU. Information about the size of the CTU (CTU size) is encoded as the syntax of SPS or PPS and transmitted to the image decoding device.

The picture segmentation unit (110) divides each picture constituting an image into a plurality of CTUs having a predetermined size, and then recursively divides the CTUs using a tree structure. A leaf node in the tree structure becomes a CU, which is a basic unit of encoding.

The tree structure may be a QuadTree (QT) in which an upper node (or parent node) is divided into four lower nodes (or child nodes) of the same size, a BinaryTree (BT) in which an upper node is divided into two lower nodes, or a TernaryTree (TT) in which an upper node is divided into three lower nodes in a 1:2:1 ratio, or a structure that mixes two or more of the QT structures, BT structures, and TT structures. For example, a QTBT (QuadTree plus BinaryTree) structure may be used, or a QTBTTT (QuadTree plus BinaryTree TernaryTree) structure may be used. Here, BTTT may be combined and referred to as a MTT (Multiple-Type Tree).

Figure 2 is a drawing for explaining a method of dividing a block using the QTBTTT structure.

As illustrated in FIG. 2, a CTU may first be split into a QT structure. The quadtree splitting may be repeated until the size of the splitting block reaches the minimum block size (MinQTSize) of the leaf node allowed in the QT. A first flag (QT_split_flag) indicating whether each node of the QT structure is split into four nodes of the lower layer is encoded by the entropy encoding unit (155) and signaled to the image decoding device. If the leaf node of the QT is not larger than the maximum block size (MaxBTSize) of the root node allowed in the BT, it may be further split into one or more of the BT structure or the TT structure. There may be multiple splitting directions in the BT structure and/or the TT structure. For example, there may be two directions in which the block of the corresponding node is split horizontally and two directions in which the block is split vertically. As illustrated in FIG. 2, when MTT splitting begins, a second flag (mtt_split_flag) indicating whether nodes have been split, and if splitting has occurred, a flag indicating the splitting direction (vertical or horizontal) and/or a flag indicating the splitting type (Binary or Ternary) are encoded by the entropy encoding unit (155) and signaled to the image decoding device.

Alternatively, before encoding the first flag (QT_split_flag) indicating whether each node is split into four nodes of a lower layer, a CU split flag (split_cu_flag) indicating whether the node is split may be encoded. If the CU split flag (split_cu_flag) value indicates that the node is not split, the block of the corresponding node becomes a leaf node in the split tree structure and becomes a CU (coding unit), which is a basic unit of encoding. If the CU split flag (split_cu_flag) value indicates that the node is split, the video encoding device starts encoding from the first flag in the above-described manner.

As another example of a tree structure, when QTBT is used, there may be two types: a type that horizontally splits the block of the corresponding node into two blocks of the same size (i.e., symmetric horizontal splitting) and a type that vertically splits it (i.e., symmetric vertical splitting). A split flag (split_flag) indicating whether each node of the BT structure is split into blocks of a lower layer and split type information indicating the type of split are encoded by the entropy encoding unit (155) and transmitted to the image decoding device. Meanwhile, there may additionally be a type that splits the block of the corresponding node into two blocks of an asymmetrical shape. The asymmetric shape may include a shape that splits the block of the corresponding node into two rectangular blocks with a size ratio of 1:3, or a shape that splits the block of the corresponding node in a diagonal direction.

A CU can have various sizes depending on the QTBT or QTBTTT partitioning from the CTU. Hereinafter, the block corresponding to the CU to be encoded or decoded (i.e., the leaf node of the QTBTTT) is referred to as the "current block." Depending on the QTBTTT partitioning employed, the current block may be rectangular as well as square.

The prediction unit (120) predicts the current block and generates a prediction block. The prediction unit (120) includes an intra prediction unit (122) and an inter prediction unit (124).

In general, each current block within a picture can be predictively coded. Prediction of the current block can typically be performed using either intra-prediction (using data from the picture containing the current block) or inter-prediction (using data from a picture coded before the picture containing the current block). Inter-prediction encompasses both unidirectional and bidirectional prediction.

The intra prediction unit (122) predicts pixels within the current block using pixels (reference pixels) located around the current block within the current picture including the current block. There are multiple intra prediction modes depending on the prediction direction. For example, as shown in Fig. 3a, the multiple intra prediction modes may include two non-directional modes including the Planar mode and the DC mode, and 65 directional modes. The surrounding pixels to be used and the calculation formula are defined differently depending on each prediction mode.

For efficient directional prediction for a rectangular current block, directional modes (intra prediction modes 67 to 80 and -1 to -14) indicated by dotted arrows in Fig. 3b may be additionally used. These may be referred to as "wide-angle intra-prediction modes." In Fig. 3b, the arrows point to corresponding reference samples used for prediction, and do not indicate the prediction direction. The prediction direction is opposite to the direction indicated by the arrows. Wide-angle intra-prediction modes are modes that perform prediction in the opposite direction of a specific directional mode without additional bit transmission when the current block is rectangular. At this time, among the wide-angle intra-prediction modes, some wide-angle intra-prediction modes available for the current block may be determined based on the ratio of the width and height of the rectangular current block. For example, wide-angle intra prediction modes (intra prediction modes 67 to 80) having an angle less than 45 degrees are available when the current block is a rectangular shape whose height is smaller than its width, and wide-angle intra prediction modes (intra prediction modes -1 to -14) having an angle greater than -135 degrees are available when the current block is a rectangular shape whose width is larger than its height.

The intra prediction unit (122) can determine the intra prediction mode to be used to encode the current block. In some examples, the intra prediction unit (122) can encode the current block using multiple intra prediction modes and select an appropriate intra prediction mode to be used from the tested modes. For example, the intra prediction unit (122) can calculate bit-rate distortion values using rate-distortion analysis for multiple tested intra prediction modes and select the intra prediction mode with the best bit-rate distortion characteristics among the tested modes.

The intra prediction unit (122) selects one intra prediction mode from among multiple intra prediction modes and predicts the current block using surrounding pixels (reference pixels) and an operation formula determined according to the selected intra prediction mode. Information about the selected intra prediction mode is encoded by the entropy encoding unit (155) and transmitted to the image decoding device.

The inter prediction unit (124) generates a prediction block for the current block using a motion compensation process. The inter prediction unit (124) searches for a block most similar to the current block within reference pictures that were encoded and decoded before the current picture, and generates a prediction block for the current block using the searched block. Then, a motion vector (MV) corresponding to the displacement between the current block within the current picture and the prediction block within the reference picture is generated. Generally, motion estimation is performed on the luma component, and the motion vector calculated based on the luma component is used for both the luma component and the chroma component. The motion information including information on the reference picture used to predict the current block and information on the motion vector is encoded by the entropy encoding unit (155) and transmitted to the image decoding device.

The inter prediction unit (124) may perform interpolation on a reference picture or a reference block to improve prediction accuracy. That is, subsamples between two consecutive integer samples are interpolated by applying filter coefficients to a plurality of consecutive integer samples including the two integer samples. When a process of searching for a block most similar to the current block is performed on the interpolated reference picture, the motion vector can be expressed up to a precision in decimal units rather than a precision in integer sample units. The precision or resolution of the motion vector can be set differently for each target region to be encoded, such as a slice, tile, CTU, CU, etc. When such adaptive motion vector resolution (AMVR) is applied, information on the motion vector resolution to be applied to each target region must be signaled for each target region. For example, when the target region is a CU, information on the motion vector resolution applied to each CU is signaled. Information on the motion vector resolution may be information indicating the precision of a differential motion vector, which will be described later.

Meanwhile, the inter prediction unit (124) can perform inter prediction using bi-prediction. In the case of bi-prediction, two reference pictures and two motion vectors indicating the block position most similar to the current block within each reference picture are used. The inter prediction unit (124) selects a first reference picture and a second reference picture from reference picture list 0 (RefPicList0) and reference picture list 1 (RefPicList1), respectively, and searches for a block similar to the current block within each reference picture to generate a first reference block and a second reference block. Then, the first reference block and the second reference block are averaged or weighted averaged to generate a prediction block for the current block. Then, motion information including information on two reference pictures used to predict the current block and information on two motion vectors is transmitted to the entropy encoding unit (155). Here, reference picture list 0 may be composed of pictures that are before the current picture in display order among the restored pictures, and reference picture list 1 may be composed of pictures that are after the current picture in display order among the restored pictures. However, this is not necessarily limited to this, and restored pictures that are after the current picture in display order may be additionally included in reference picture list 0, and conversely, restored pictures that are before the current picture may be additionally included in reference picture list 1.

Various methods can be used to minimize the number of bits required to encode motion information.

For example, if the reference picture and motion vector of the current block are identical to those of a neighboring block, the motion information of the current block can be transmitted to the image decoding device by encoding information that can identify the neighboring block. This method is called 'merge mode'.

In merge mode, the inter prediction unit (124) selects a predetermined number of merge candidate blocks (hereinafter referred to as 'merge candidates') from the surrounding blocks of the current block.

As the surrounding blocks for deriving merge candidates, all or part of the left block (A0), the lower left block (A1), the upper block (B0), the upper right block (B1), and the upper left block (B2) adjacent to the current block within the current picture may be used, as illustrated in FIG. 4. In addition, a block located within a reference picture (which may or may not be the same as the reference picture used to predict the current block) other than the current picture in which the current block is located may be used as a merge candidate. For example, a block co-located with the current block within the reference picture or blocks adjacent to the block at the co-located block may be additionally used as a merge candidate. If the number of merge candidates selected by the method described above is less than a preset number, a 0 vector is added to the merge candidates.

The inter prediction unit (124) uses these surrounding blocks to construct a merge list containing a predetermined number of merge candidates. Among the merge candidates included in the merge list, the merge candidate to be used as motion information of the current block is selected and merge index information for identifying the selected candidate is generated. The generated merge index information is encoded by the entropy encoding unit (155) and transmitted to the video decoding device.

Merge Skip mode is a special case of merge mode. After quantization, when all transform coefficients for entropy encoding are close to zero, only neighboring block selection information is transmitted without transmitting residual signals. By utilizing merge skip mode, relatively high encoding efficiency can be achieved for low-motion images, still images, and screen content images.

Hereinafter, merge mode and merge skip mode are collectively referred to as merge/skip mode.

Another method for encoding motion information is Advanced Motion Vector Prediction (AMVP) mode.

In AMVP mode, the inter prediction unit (124) derives predicted motion vector candidates for the motion vector of the current block using neighboring blocks of the current block. As neighboring blocks used to derive predicted motion vector candidates, all or some of the left block (A0), the lower left block (A1), the upper block (B0), the upper right block (B1), and the upper left block (B2) adjacent to the current block in the current picture as shown in FIG. 4 may be used. In addition, a block located in a reference picture (which may or may not be the same as the reference picture used to predict the current block) other than the current picture in which the current block is located may be used as the neighboring block used to derive predicted motion vector candidates. For example, a block co-located with the current block in the reference picture or blocks adjacent to the block in the co-located block may be used. If the number of motion vector candidates is less than a preset number by the method described above, a 0 vector is added to the motion vector candidates.

The inter prediction unit (124) derives predicted motion vector candidates using the motion vectors of these surrounding blocks, and determines a predicted motion vector for the motion vector of the current block using the predicted motion vector candidates. Then, the predicted motion vector is subtracted from the motion vector of the current block to produce a differential motion vector.

The predicted motion vector can be obtained by applying a predefined function (e.g., median, mean, etc.) to the predicted motion vector candidates. In this case, the image decoding device also knows the predefined function. In addition, since the surrounding blocks used to derive the predicted motion vector candidates are blocks that have already been encoded and decoded, the image decoding device also already knows the motion vectors of the surrounding blocks. Therefore, the image encoding device does not need to encode information to identify the predicted motion vector candidates. Therefore, in this case, information about the differential motion vector and information about the reference picture used to predict the current block are encoded.

Alternatively, the predicted motion vector can be determined by selecting one of the predicted motion vector candidates. In this case, information for identifying the selected predicted motion vector candidate is additionally encoded, along with information about the differential motion vector and the reference picture used to predict the current block.

The subtractor (130) subtracts the prediction block generated by the intra prediction unit (122) or inter prediction unit (124) from the current block to generate a residual block.

The transformation unit (140) transforms residual signals within a residual block having pixel values in a spatial domain into transform coefficients in a frequency domain. The transformation unit (140) may transform the residual signals within the residual block using the entire size of the residual block as a transformation unit, or may divide the residual block into a plurality of sub-blocks and use the sub-blocks as transformation units to perform the transformation. Alternatively, the residual signals may be transformed using only the transformation domain sub-block as a transformation unit by dividing the sub-blocks into two sub-blocks, that is, a transformation domain and a non-transform domain. Here, the transformation domain sub-block may be one of two rectangular blocks having a size ratio of 1:1 with respect to the horizontal axis (or vertical axis). In this case, a flag (cu_sbt_flag) indicating that only a sub-block has been converted, directionality (vertical/horizontal) information (cu_sbt_horizontal_flag), and/or position information (cu_sbt_pos_flag) are encoded by the entropy encoding unit (155) and signaled to the image decoding device. In addition, the size of the conversion area sub-block may have a size ratio of 1:3 with respect to the horizontal axis (or vertical axis), and in this case, a flag (cu_sbt_quad_flag) distinguishing the corresponding division is additionally encoded by the entropy encoding unit (155) and signaled to the image decoding device.

Meanwhile, the transformation unit (140) can individually perform transformations on the residual block in the horizontal and vertical directions. For the transformation, various types of transformation functions or transformation matrices can be used. For example, a pair of transformation functions for horizontal transformation and vertical transformation can be defined as a Multiple Transform Set (MTS). The transformation unit (140) can select one transformation function pair with the best transformation efficiency among the MTS and transform the residual block in the horizontal and vertical directions, respectively. Information (mts_idx) on the transformation function pair selected among the MTS is encoded by the entropy encoding unit (155) and signaled to the image decoding device.

The quantization unit (145) quantizes the transform coefficients output from the transform unit (140) using quantization parameters and outputs the quantized transform coefficients to the entropy encoding unit (155). The quantization unit (145) may directly quantize a related residual block without transformation for a certain block or frame. The quantization unit (145) may also apply different quantization coefficients (scaling values) according to the positions of the transform coefficients within the transform block. The quantization matrix applied to the quantized transform coefficients arranged in two dimensions may be encoded and signaled to an image decoding device.

The rearrangement unit (150) can perform rearrangement of coefficient values for quantized residual values.

The reordering unit (150) can change a two-dimensional coefficient array into a one-dimensional coefficient sequence by using coefficient scanning. For example, the reordering unit (150) can output a one-dimensional coefficient sequence by scanning from the DC coefficient to the coefficients of the high-frequency region by using a zig-zag scan or a diagonal scan. Depending on the size of the transformation unit and the intra prediction mode, a vertical scan that scans the two-dimensional coefficient array in the column direction or a horizontal scan that scans the two-dimensional block-shaped coefficients in the row direction may be used instead of the zig-zag scan. That is, depending on the size of the transformation unit and the intra prediction mode, the scanning method to be used may be determined among the zig-zag scan, the diagonal scan, the vertical scan, and the horizontal scan.

The entropy encoding unit (155) generates a bitstream by encoding a sequence of one-dimensional quantized transform coefficients output from the rearrangement unit (150) using various encoding methods such as CABAC (Context-based Adaptive Binary Arithmetic Code) and Exponential Golomb.

In addition, the entropy encoding unit (155) encodes information related to block division, such as CTU size, CU division flag, QT division flag, MTT division type, and MTT division direction, so that the image decoding device can divide the block in the same manner as the image encoding device. In addition, the entropy encoding unit (155) encodes information about a prediction type indicating whether the current block is encoded by intra prediction or inter prediction, and encodes intra prediction information (i.e., information about an intra prediction mode) or inter prediction information (information about an encoding mode of motion information (merge mode or AMVP mode), a merge index in the case of a merge mode, and a reference picture index and a differential motion vector in the case of an AMVP mode) according to the prediction type. In addition, the entropy encoding unit (155) encodes information related to quantization, that is, information about a quantization parameter and information about a quantization matrix.

The inverse quantization unit (160) inversely quantizes the quantized transform coefficients output from the quantization unit (145) to generate transform coefficients. The inverse transform unit (165) transforms the transform coefficients output from the inverse quantization unit (160) from the frequency domain to the spatial domain to restore the residual block.

An adder (170) adds the restored residual block and the predicted block generated by the prediction unit (120) to restore the current block. The pixels within the restored current block are used as reference pixels when intra-predicting the next block.

The loop filter unit (180) performs filtering on restored pixels to reduce blocking artifacts, ringing artifacts, blurring artifacts, etc. that occur due to block-based prediction and transformation/quantization. The loop filter unit (180) may include all or part of a deblocking filter (182), a sample adaptive offset (SAO) filter (184), and an adaptive loop filter (ALF, 186) as an in-loop filter.

The deblocking filter (182) filters the boundaries between restored blocks to remove blocking artifacts caused by block-based encoding/decoding, and the SAO filter (184) and the ALF (186) perform additional filtering on the deblocking-filtered image. The SAO filter (184) and the ALF (186) are filters used to compensate for the differences between restored pixels and original pixels caused by lossy coding. The SAO filter (184) improves not only subjective image quality but also encoding efficiency by applying an offset in units of CTUs. In contrast, the ALF (186) performs block-based filtering, and compensates for distortion by applying different filters by distinguishing the edges and degrees of variation of the corresponding block. Information on filter coefficients to be used in the ALF can be encoded and signaled to an image decoding device.

The restored blocks filtered through the deblocking filter (182), SAO filter (184), and ALF (186) are stored in the memory (190). When all blocks within a picture are restored, the restored picture can be used as a reference picture for inter-predicting blocks within a picture to be encoded later.

The video encoding device can store the bitstream of encoded video data on a non-transitory storage medium or transmit it to the video decoding device using a communication network.

FIG. 5 is an exemplary block diagram of an image decoding device capable of implementing the techniques of the present disclosure. Hereinafter, the image decoding device and its subcomponents will be described with reference to FIG. 5.

The video decoding device may be configured to include an entropy decoding unit (510), a rearrangement unit (515), an inverse quantization unit (520), an inverse transformation unit (530), a prediction unit (540), an adder (550), a loop filter unit (560), and a memory (570).

Similar to the video encoding device of FIG. 1, each component of the video decoding device may be implemented in hardware, software, or a combination of hardware and software. Furthermore, the functions of each component may be implemented in software, with a microprocessor executing the software functions corresponding to each component.

The entropy decoding unit (510) decodes the bitstream generated by the image encoding device to extract information related to block division, thereby determining the current block to be decoded, and extracts prediction information, information on residual signals, etc. required to restore the current block.

The entropy decoding unit (510) extracts information about the CTU size from the Sequence Parameter Set (SPS) or the Picture Parameter Set (PPS), determines the size of the CTU, and divides the picture into CTUs of the determined size. Then, the CTU is determined as the top layer of the tree structure, i.e., the root node, and the CTU is divided using the tree structure by extracting division information about the CTU.

For example, when splitting a CTU using the QTBTTT structure, first, the first flag (QT_split_flag) related to the splitting of QT is extracted, and each node is split into four nodes of the lower layer. Then, for the nodes corresponding to the leaf nodes of QT, the second flag (mtt_split_flag) related to the splitting of MTT and the split direction (vertical / horizontal) and/or split type (binary / ternary) information are extracted, and the corresponding leaf nodes are split into the MTT structure. Accordingly, each node below the leaf nodes of QT are split recursively into the BT or TT structure.

As another example, when splitting a CTU using the QTBTTT structure, the CU split flag (split_cu_flag) indicating whether the CU is split is first extracted, and if the block is split, the first flag (QT_split_flag) may be extracted. During the splitting process, each node may undergo zero or more repeated QT splits followed by zero or more repeated MTT splits. For example, a CTU may undergo an MTT split right away, or conversely, may undergo only multiple QT splits.

As another example, when splitting a CTU using the QTBT structure, the first flag (QT_split_flag) related to the splitting of QT is extracted, and each node is split into four nodes of the lower layer. Furthermore, for nodes corresponding to leaf nodes of QT, a split flag (split_flag) indicating whether to further split into BTs and splitting direction information are extracted.

Meanwhile, when the entropy decoding unit (510) determines the current block to be decoded by using the division of the tree structure, it extracts information on the prediction type indicating whether the current block is intra-predicted or inter-predicted. If the prediction type information indicates intra-prediction, the entropy decoding unit (510) extracts syntax elements for intra-prediction information (intra-prediction mode) of the current block. If the prediction type information indicates inter-prediction, the entropy decoding unit (510) extracts syntax elements for inter-prediction information, i.e., information indicating a motion vector and a reference picture referenced by the motion vector.

Additionally, the entropy decoding unit (510) extracts information about the quantized transform coefficients of the current block as information related to quantization and information about residual signals.

The rearrangement unit (515) can change the sequence of one-dimensional quantized transform coefficients entropy-decoded in the entropy decoding unit (510) back into a two-dimensional coefficient array (i.e., block) in the reverse order of the coefficient scanning performed by the image encoding device.

The inverse quantization unit (520) inversely quantizes the quantized transform coefficients and inversely quantizes the quantized transform coefficients using the quantization parameters. The inverse quantization unit (520) may also apply different quantization coefficients (scaling values) to the quantized transform coefficients arranged in two dimensions. The inverse quantization unit (520) may perform inverse quantization by applying a matrix of quantized coefficients (scaling values) from an image encoding device to a two-dimensional array of quantized transform coefficients.

The inverse transform unit (530) inversely transforms the inverse quantized transform coefficients from the frequency domain to the spatial domain to restore residual signals, thereby generating a residual block for the current block.

In addition, when the inverse transform unit (530) inversely transforms only a portion of a transform block (sub-block), it extracts a flag (cu_sbt_flag) indicating that only a sub-block of the transform block has been transformed, directionality (vertical/horizontal) information (cu_sbt_horizontal_flag) of the sub-block, and/or position information (cu_sbt_pos_flag) of the sub-block, and inversely transforms the transform coefficients of the corresponding sub-block from the frequency domain to the spatial domain to restore residual signals, and fills “0” values with residual signals for areas that have not been inversely transformed, thereby generating a final residual block for the current block.

In addition, when MTS is applied, the inverse transform unit (530) determines a transform function or a transform matrix to be applied in the horizontal and vertical directions using MTS information (mts_idx) signaled from the image encoding device, and performs inverse transform on the transform coefficients within the transform block in the horizontal and vertical directions using the determined transform function.

The prediction unit (540) may include an intra prediction unit (542) and an inter prediction unit (544). The intra prediction unit (542) is activated when the prediction type of the current block is intra prediction, and the inter prediction unit (544) is activated when the prediction type of the current block is inter prediction.

The intra prediction unit (542) determines the intra prediction mode of the current block among a plurality of intra prediction modes from the syntax elements for the intra prediction mode extracted from the entropy decoding unit (510), and predicts the current block using reference pixels around the current block according to the intra prediction mode.

The inter prediction unit (544) uses the syntax elements for the inter prediction mode extracted from the entropy decoding unit (510) to determine the motion vector of the current block and the reference picture referenced by the motion vector, and predicts the current block using the motion vector and the reference picture.

An adder (550) adds the residual block output from the inverse transform unit (530) and the predicted block output from the inter prediction unit (544) or the intra prediction unit (542) to restore the current block. The pixels within the restored current block are used as reference pixels when intra-predicting a block to be decoded later.

The loop filter unit (560) may include a deblocking filter (562), an SAO filter (564), and an ALF (566) as in-loop filters. The deblocking filter (562) deblocks the boundaries between restored blocks to remove blocking artifacts caused by block-by-block decoding. The SAO filter (564) and the ALF (566) perform additional filtering on restored blocks after deblocking filtering to compensate for differences between restored pixels and original pixels caused by lossy coding. The filter coefficients of the ALF are determined using information about filter coefficients decoded from the non-stream.

The restored blocks filtered through the deblocking filter (562), SAO filter (564), and ALF (566) are stored in the memory (570). When all blocks within a picture are restored, the restored picture is used as a reference picture for inter-predicting blocks within a picture to be encoded later.

Hereinafter, an improved coding tool performed by the aforementioned video encoding device or video decoding device is disclosed. The following embodiments may be performed by the intra prediction unit (122) within the video encoding device. Additionally, the following embodiments may be performed by the intra prediction unit (542) within the video decoding device. In the present disclosure, the video processing device refers to at least one of the video encoding device or the video decoding device.

The present disclosure relates to a matrix-based intra prediction (MIP) mode. The MIP mode is a mode that derives prediction samples of a current block by multiplying reference pixels in a neighboring reference template of the current block by a weight matrix.

The process for deriving a prediction sample of the current block in MIP mode can be expressed as in mathematical expression 1.

In Equation 1, P(x, y) represents a prediction sample of the current block, x and y represent the positions of the prediction samples within the current block, r(k) represents the kth reference pixel of the current block, and F(x, y, k) represents the matrix weights corresponding to the prediction sample (x, y). k represents the index of the reference pixel within the reference template of the current block. The value of the prediction sample of the current block is determined by the multiplication between the weight matrix of the current block and the reference pixel. The size of the prediction block of the current block is the same as the size of the current block.

MIP mode can be enabled or disabled based on the size of the current block. MIP mode can be enabled for blocks whose width and height are each less than or equal to a predefined value. For example, the predefined value can be 32. MIP mode can be used for block sizes where both width and height are at most 32. The predefined value can be other values besides 32. On the other hand, MIP mode may not be enabled for blocks of certain sizes. For example, MIP mode may not be enabled for blocks whose width x height are 4x32, 32x4, 8x32, or 32x8.

The MIP mode can be activated or deactivated based on the intra prediction mode of the current block. For example, the MIP mode can be activated when the intra prediction mode number of the current block is 0, 1, or (2 + 2 × k), where k is an integer including 0. As another example, the MIP mode can be activated when the intra prediction mode number of the current block is 0, 1, or (2 + 4 × k).

The MIP mode can be activated or deactivated based on at least one of the size of the current block or the intra prediction mode. The MIP mode can be activated for blocks smaller than or equal to a preset size and for preset intra prediction modes. For example, when the width and height of the current block are 16 or less and the number of the intra prediction mode of the current block is 0, 1, or (2 + 2 × k), the MIP mode can be activated. Here, k is an integer including 0. For another example, when the width and height of the current block are 32 or less and the number of the intra prediction mode of the current block is 0, 1, or (2 + 4 × k), the MIP mode can be activated. Here, k is an integer including 0.

When the width or height of the current block is greater than 16, a 16x16 prediction block is generated using the MIP mode, and interpolation or bilinear interpolation is performed on the 16x16 prediction block to generate a prediction block of the current block.

When MIP mode is enabled, the intra prediction mode of the current block is replaced with the MIP mode. Specifically, the existing intra prediction mode or the derived intra prediction mode of the current block can be replaced with the MIP mode. Some directional intra prediction modes can be replaced with the MIP mode. Once it is determined whether the intra prediction mode and MIP mode of the current block are enabled, the current block is predicted with the MIP mode without additional signaling.

In particular, the present disclosure provides a video coding method and apparatus for performing MIP prediction using various reference pixels. According to the present disclosure, various reference pixels are used for MIP prediction of a current block without limiting the shape, location, and characteristics of the reference pixels. According to the present disclosure, the accuracy of the intra prediction mode can be improved, and encoding and decoding efficiency can be increased. Furthermore, an offset can be used for MIP prediction according to the present disclosure.

FIG. 6 is a flowchart of a matrix-based intra prediction method according to one embodiment of the present disclosure.

In step S610, the image processing device determines a reference template of the current block.

A reference template refers to a set of pixels encoded or decoded before the current block.

The reference template may include pixels adjacent to the current block. For example, the reference template may include at least one reference line adjacent to the current block.

A reference template may contain pixels that are not adjacent to the current block. For example, a reference template may contain reference blocks that are not adjacent to the current block.

A reference template may include a reference region indicated by motion information of surrounding blocks of the current block. The motion information may be referred to as a motion vector or a block vector.

A reference template can be pre-defined between a video encoding device and a video decoding device.

Alternatively, the reference template may be determined based on either the intra prediction mode of the current block or signaling information. The signaling information may include information about the shape, position, size, etc. of the reference template. For example, the signaling information may include information for indicating a reference template including pixels adjacent to the current block, a reference template including non-adjacent pixels, or a reference template indicated using motion information of a neighboring block.

The size of the reference template may be predefined. Alternatively, the size of the reference template may be determined based on at least one of the size of the current block or the size of the weight matrix. The size of the reference template may further be determined based on the intra prediction mode.

At least one of the pixels in the reference template may be a pixel generated from another pixel using at least one of padding or interpolation.

The image processing device can adjust the reference template by performing a weighted sum of pixels within the reference template in units of a predetermined shape.

Information about whether to perform padding, interpolation, adjustment, etc. on the reference template may be agreed upon or signaled between the video encoding device and the video decoding device.

In step S620, the image processing device determines a weight matrix for matrix-based intra prediction of the current block. Furthermore, the image processing device may determine an offset for matrix-based intra prediction.

The weight matrix may be derived based on at least one of the size of the current block or the intra prediction mode. For example, the weight matrix of the current block may be selected from predefined weight matrices based on at least one of the size of the current block or the intra prediction mode. As another example, each element of the weight matrix of the current block may be derived based on at least one of the size of the current block or the intra prediction mode.

Alternatively, the weight matrix may be determined based on signaling information between the image encoding device and the image decoding device. Information for indicating the weight matrix of the current block may be signaled from predefined weight matrices.

The offset is a bias used to adjust the result of matrix multiplication. The offset can be 0 or a real number. The offset can have the same value regardless of the position of the prediction samples in the current block. Alternatively, the offset can have different values for each position of the prediction samples in the current block.

The offset may be a value selected from predefined values based on signaling information between the video encoding device and the video decoding device. Alternatively, the offset may be derived based on information of the current block or neighboring blocks.

In step S630, the image processing device generates a prediction block of the current block based on the reference template and weight matrix of the current block. An offset may further be used to generate the prediction block of the current block.

The process for generating a prediction block of the current block can be expressed as in mathematical expression 2.

In Equation 2, b(k) represents an offset corresponding to the k-th reference pixel. P(x, y) represents a prediction sample of the current block, x and y represent the positions of the prediction sample within the current block, r(k) represents the k-th reference pixel of the current block, and F(x, y, k) represents the matrix weights corresponding to the prediction sample (x, y). k represents the index of the reference pixel within the reference template of the current block. The value of the prediction sample of the current block is determined by the multiplication of the weight matrix of the current block and the reference pixel. The size of the prediction block of the current block is the same as the size of the current block.

According to another embodiment of the present disclosure, a prediction block of a current block is generated through reduction, matrix multiplication, and upsampling of a reference template.

Reference pixels are derived and can be used for MIP prediction. Deriving the reference pixels may include reduction of the reference template, matrix operations, or upsampling.

First, the reference template is reduced by performing averaging or sampling on the pixels within the reference template. The reduced reference template may be in the form of a vector or matrix. Here, the number of pixels to which averaging is applied, the sampling rate, etc., may be agreed upon between the image encoding device and the image decoding device.

Thereafter, a boundary vector is derived by multiplying the reduced reference template by one of the predefined matrices. The matrix used in the operation may be selected from the predefined matrices based on at least one of the size of the current block or the intra prediction mode.

By upsampling the boundary vector, a predicted block of the current block is generated. For example, a predicted block of the current block can be derived by interpolating elements of the boundary vector.

The generation of a prediction block of the current block based on the reduction and upsampling of the reference template can be performed based on at least one of the size or the intra prediction mode of the current block being a predefined value. In other words, when at least one of the size or the intra prediction mode of the current block falls within a predetermined range, the prediction block of the current block can be generated from the reduction, matrix multiplication, and upsampling of the reference template.

Alternatively, whether to perform generation of a prediction block of the current block based on reduction and upsampling of the reference template can be determined based on signaling information.

In this way, the image processing device can improve coding efficiency and prediction accuracy by using various reference templates for MIP prediction.

FIGS. 7A, 7B, and 7C are diagrams illustrating reference templates adjacent to a current block according to one embodiment of the present disclosure.

Referring to FIGS. 7a, 7b and 7c, the reference template may be any one of an L-shaped reference template, a left reference template or an upper reference template.

In other embodiments, reference templates of various shapes may be used.

In Fig. 7a, an L-shaped reference template is illustrated.

An L-shaped reference template includes at least one reference line adjacent to the current block. Specifically, the L-shaped reference template includes at least one left reference line, at least one top reference line, and at least one top-left reference pixel.

In other words, the L-shaped reference template may include the left and top surrounding pixels of the current block. Specifically, the L-shaped reference template may include the bottom-left surrounding pixels, the left surrounding pixels, the top-left surrounding pixels, the top-right surrounding pixels, and the top-right surrounding pixels of the current block.

The size or area of an L-shaped reference template can be expressed as (C _W ×W × L1)+(C _H ×H × L2)+(L1 × L2). Here, W is the width of the current block, H is the height of the current block, and C _W and C _H are weights for the height of the width of the current block, respectively. L1 is the number of upper reference lines, and L2 is the number of left reference lines. C _W , C _H , L1 , and L2 are positive integers.

For example, a reference template may include four reconstructed samples of the left neighboring line of the current block (L2 = 4) and four reconstructed samples of the top neighboring line of the current block (L1 = 4). The width and height of the reference template may have values of 2W and 2H, respectively.

In Fig. 7b, the left reference template is shown.

The left reference template includes at least one left reference line adjacent to the current block. In other words, the left reference template includes the left surrounding pixels of the current block. Specifically, the left reference template includes the lower left surrounding pixels and the left surrounding pixels of the current block. Furthermore, the left reference template may include the upper left surrounding pixels of the current block.

The size or area of the left reference template can be expressed as (C _H × H × L3), where L3 represents the number of left reference lines.

In Fig. 7c, the upper reference template is shown.

The upper reference template includes at least one upper reference line adjacent to the current block. In other words, the upper reference template includes the upper peripheral pixels of the current block. Specifically, the upper reference template includes the upper peripheral pixels and the upper-right peripheral pixels of the current block. Furthermore, the upper reference template may include the upper-left peripheral pixels of the current block.

The size or area of the upper reference template can be expressed as (C _W ×W × L4), where L4 represents the number of upper reference lines.

In FIGS. 7a, 7b and 7c, the sizes of the reference templates, i.e., C _W , C _H , L1, L2, L3 and L4, can be determined in various ways.

C _W , C _H , L1, L2, L3 and L4 can be pre-defined between the image encoding device and the image decoding device.

As an example, C _W and C _H can be 2, and L1, L2, L3 and L4 can be 4.

As another example, C _W and C _H can be 2, and L1, L2, L3 and L4 can be 2.

As another example, C _W and C _H can be 2, and L1, L2, L3, and L4 can be 1.

As another example, C _W and C _H can be 1, and L1, L2, L3, and L4 can be 1.

Alternatively, C _W , C _H , L1, L2, L3 and L4 can be determined based on the size of the current block. L1, L2, L3 and L4 can be selected from predefined values based on the size of the current block. Here, the predefined values can be 1, 2, or 4, etc.

As an example, when the size of the current block is less than or equal to 16x16, C _W and C _H can be 2, and L1, L2, L3, and L4 can be 2.

As another example, when the width and height of the current block are greater than 16x16 and the width and height of the current block are less than or equal to 32x32, C _W and C _H can be 2, and L1, L2, L3, and L4 can be 1.

Alternatively, C _W , C _H , L1, L2, L3 and L4 may be determined based on at least one of the size of the current block or the size of the weight matrix. In the left reference template, when the size of the weight matrix is MxN, the height of the left reference template is N, and C _H may be a value obtained by dividing N by H.

As an example, when the current block size is less than or equal to 16x16, L3 can be 2.

Alternatively, C _W , C _H , L1, L2, L3 and L4 may be determined based on at least one of the size of the current block or the intra prediction mode.

As an example, when the size of the current block is less than or equal to 16x16 and the intra prediction mode of the current block is included in mode number 0, 1, or (2 + 2×k), C _W and C _H may be 2, and L1, L2, L3, and L4 may be 2. Here, k is 0 or a positive integer.

As another example, when the width and height of the current block are greater than 16x16, the width and height of the current block are less than or equal to 32x32, and the intra prediction mode of the current block is included in mode number 0, 1, or (2 + 4×k), C _W and C _H may be 2, and L1, L2, L3, and L4 may be 1, where k is 0 or a positive integer.

Alternatively, C _W and C _H can be determined based on the intra prediction mode. For example, for intra prediction modes 18 to 50, C _W and C _H can be set to 1. For the remaining intra prediction modes, C _W and C _H can be set to 2.

Meanwhile, the reference template adjacent to the current block can be determined either implicitly or explicitly.

As an implicit method, a reference template can be determined based on an intra prediction mode that is replaced by a MIP mode. Various reference templates can be predefined or mapped to intra prediction modes. If an intra prediction mode is a predefined value or falls within a predefined range, a specific reference template can be used for MIP prediction. The relationship between a reference template and an intra prediction mode can be agreed upon between a video encoding device and a video decoding device.

As an example, when the number of the intra prediction mode of the current block is 50, which indicates the intra prediction mode in the vertical direction, the upper reference template can be used for MIP prediction.

As another example, when the number of the intra prediction mode of the current block is 18, which indicates the intra prediction mode in the horizontal direction, the left reference template can be used for MIP prediction.

As another example, when the number of the intra prediction mode of the current block is 34, which indicates a diagonal intra prediction mode, the L-shaped reference template can be used for MIP prediction.

As another example, when the number of the intra prediction mode of the current block is 45, which indicates a diagonal intra prediction mode, the entire upper reference template and the upper part of the left reference template can be used.

As another implicit method, the reference template may be determined based on information other than the intra prediction mode.

As an explicit method, the reference template can be determined based on signaling information between the video encoding device and the video decoding device.

As an example, the signaling information may include an index or flag for identifying the reference template of the current block from among predefined reference templates. The reference template of the current block may be selected from among the predefined reference templates based on the signaling information.

FIG. 8 is a drawing for explaining a reference template that is not adjacent to a current block according to one embodiment of the present disclosure.

Non-adjacent surrounding blocks of the current block can be specified between the video encoding device and the video decoding device.

Figure 8 illustrates examples of non-adjacent neighboring blocks of the current block. At least one of blocks 14 to 31 may be used as a reference template for MIP prediction of the current block.

The number of each pixel represents the non-adjacent neighboring block containing the corresponding pixel. For example, the non-adjacent neighboring block containing pixel 14 can be referred to as block 14. The pixels contained in block 14 are referred to as reference pixels of the current block.

In other embodiments, various non-adjacent peripheral blocks may be utilized. Various peripheral blocks may be utilized as reference templates depending on their location, size, number, or shape.

Non-adjacent neighboring blocks for MIP prediction of the current block can be predefined or derived.

As an example, the location, size, number, or shape of non-adjacent surrounding blocks can be specified in advance.

As another example, the location, size, number, shape, etc. of non-adjacent neighboring blocks can be derived based on the size or prediction mode of the current block, etc. The image decoding device can select non-adjacent neighboring blocks for MIP prediction from predefined non-adjacent neighboring blocks based on the size or prediction mode of the current block, etc.

As another example, the location, size, number, or shape of non-adjacent peripheral blocks can be derived based on signaling information. The signaling information may include a flag or index for specifying non-adjacent peripheral blocks.

FIG. 9 is a drawing for explaining a reference template indicated by movement information of a surrounding block according to one embodiment of the present disclosure.

In Fig. 9, the neighboring block adjacent to the left of the current block is a predicted block using a reconstructed block indicated by the motion information of the neighboring block within the current picture.

A reconstructed block can be determined using template matching when predicting surrounding blocks. Specifically, template matching means finding a template in a search area that minimizes the difference from the template of a target block. The difference between the template of the target block and the template of any block in the search area is defined as a cost function. The template that minimizes the cost is searched for, and the block of the searched template is used as a prediction block of the target block. As the cost function, a sum of absolute differences (SAD), a sum of absolute transformed differences (SATD), a mean-removed sum of absolute differences (MR-SAD), a mean squared error (MSE), or a sum of squared errors (SSE) can be used. For example, in Fig. 9, the difference between the template of the surrounding block and the template of the reconstructed block is the smallest, and the predicted block of the surrounding block may be the same as the reconstructed block. The reconstructed block may be located at a distance equal to the motion information from the surrounding block and may have the same size as the surrounding block.

Motion information of surrounding blocks can indicate the relative positional relationship between the surrounding blocks of the current block and the reconstructed block. The motion information of surrounding blocks can be referred to as a block vector.

When a surrounding block has motion information, pixels at a location indicated by the motion information of the surrounding block or surrounding pixels at the indicated location can be used as a reference template of the current block.

In one example, as a reference template of the current block, a reconstructed block indicated by motion information of a surrounding block based on a surrounding block within the current picture can be used.

In another example, as a reference template of the current block, a template of a reconstructed block indicated by motion information of a surrounding block based on a surrounding block within the current picture can be used.

In another example, a block indicated by motion information of a neighboring block relative to the current block within the current picture may be used as a reference template for the current block. Here, the size and direction of the motion information of the neighboring block are maintained, and the block indicated by the motion information of the neighboring block may be the right block of the reconstructed block.

In another example, as a reference template of the current block in the current picture, a template of a block indicated by motion information of a surrounding block based on the current block may be used.

In another embodiment, the neighboring blocks of the current block can be predicted using blocks indicated by motion information of the neighboring blocks among reconstructed blocks in a reference picture different from the current picture.

Motion information of a neighboring block can indicate the positional relationship between the corresponding neighboring block in a reference picture and the reconstructed block in the reference picture. Motion information of a neighboring block can also indicate the relative positional relationship between the neighboring block in the current picture and the reconstructed block in the reference picture. Motion information of a neighboring block can be referred to as a motion vector.

The corresponding block of a neighboring block refers to a block within a reference picture that has the same relative position as the neighboring block within the current picture. In other words, the positions of the neighboring blocks within the current picture and the corresponding blocks within the reference picture are identical. Furthermore, the corresponding block of the current block refers to a block within the reference picture that has the same relative position as the current block within the current picture.

In one example, as a reference template for the current block, a block indicated by motion information of a neighboring block based on a corresponding block of a neighboring block within a reference picture may be used. The indicated block is a block located at a distance equal to the motion information from the corresponding block of the neighboring block.

In another example, as a reference template of the current block, a template of a block indicated by motion information of a surrounding block based on a corresponding block of a surrounding block in a reference picture may be used.

In another example, a block indicated by motion information of a neighboring block relative to the corresponding block of the current block within the reference picture may be used as a reference template for the current block. Here, the magnitude and direction of the motion information of the neighboring block are maintained. The block indicated by the motion information of the neighboring block may be the block to the right of the corresponding block of the reconstructed block.

In another example, as a reference template of the current block, a template of a block indicated by motion information of a surrounding block based on a corresponding block of the current block in a reference picture may be used.

According to one embodiment of the present disclosure, if the reference template of the current block is not available for MIP prediction, at least one pixel within the reference template can be generated from another pixel using at least one of padding or interpolation.

Specifically, if matrix operations are impossible due to a size mismatch between the reference template size of the current block and the size of the weight matrix, if pixels within the reference template are unavailable, or if the reference template straddles a predetermined boundary, pixels for the reference template may be generated or replaced. For example, if the height of the reference template is smaller than the width of the weight matrix, the reference template may be expanded.

To generate pixels from a reference template, padding, interpolation, vertical interpolation, horizontal interpolation, or both vertical and horizontal interpolation can be used.

For example, a reference template can be expanded by padding some pixels within the reference template. Alternatively, the reference template can be padded using a predefined value or a value based on bit depth.

As another example, a reference template can be extended by interpolating one or more pixels within the reference template. Alternatively, a reference template can be extended by interpolating surrounding pixels within the reference template. Alternatively, a reference template can be extended by performing interpolation using pixels within the reference template and surrounding pixels of the reference template.

As another example, unavailable pixels within the reference template can be replaced by pixels generated by interpolating surrounding pixels.

According to one embodiment of the present disclosure, a reference template of a current block can be adjusted by performing a weighted sum of pixels within a reference template in units of a predetermined shape. The adjusted reference template is used for MIP prediction of the current block.

The adjustment of the reference template can be expressed as in mathematical expression 3.

In Equation 3, ref _adj is an adjusted reference template, w is a weight, and ref is a weighted unit within the reference template. The shape of ref _n can be a line or a block. The sum of the weights is 1.

For example, the given unit may be a line unit. When the reference template of the current block is an upper reference template including multiple upper reference lines, the multiple upper reference lines within the reference template may be weighted vertically. Here, the number of lines in the weighted reference lines is smaller than the number of lines in the multiple upper reference lines.

As another example, the given unit may be an L-shaped unit. When the shape of the reference template of the current block is an L-shaped template, the L-shaped reference lines within the reference template may be weighted in L-shaped units. The upper reference lines and the left reference lines may be weighted, and the upper left pixels of the current block may be weighted. In Equation 3, ref ₀ may be an L-shaped reference line adjacent to the current block, and ref _n may be an L-shaped reference line farthest from the current block.

As another example, the given unit may be a block unit. When the reference template of the current block includes non-adjacent neighboring blocks of the current block, the non-adjacent neighboring blocks may be weighted. The number of weighted non-adjacent neighboring blocks is smaller than the number of non-weighted neighboring blocks.

The position, order, size, or direction of certain units for adjusting the reference template may be changed. For example, the upper reference lines may be weighted horizontally.

A reference template may be adjusted for a portion of the reference template. In other words, the adjusted reference template may be a weighted sum of a portion of lines or blocks within the original reference template.

Information for adjusting a reference template may be agreed upon or signaled between a video encoding device and a video decoding device. Information for adjusting a reference template may include whether to adjust the reference template, a unit for weighted summation, the location, size, number, or weighted sum direction of reference lines or reference blocks within the reference template.

FIG. 10 is a flowchart of an image encoding method according to one embodiment of the present disclosure.

In step S1010 of FIG. 10, the image encoding device determines an intra prediction mode of the current block.

MIP mode can be determined to be the optimal intra prediction mode.

In step S1020, the image encoding device performs intra prediction for the current block based on the intra prediction mode.

When the intra prediction mode is MIP mode, the video encoding device performs MIP prediction for the current block. MIP prediction can be performed using the methods described in FIGS. 6 to 9.

In step S1030, the image encoding device encodes information about an intra prediction mode.

Information about the intra prediction mode may include information about the use of MIP, information about the reference template, matrix, or offset used for MIP prediction, etc.

Information about the use of MIP may include a MIP constraint flag (MIP_constraint_flag) and a MIP flag. The MIP flag may be a MIP enable flag (MIP_enabled_flag) or a MIP disable flag (MIP_disabled_flag). Setting the MIP constraint flag to 0 indicates that there are no constraints on MIP prediction. The MIP enable flag or MIP disable flag may be set to 0 or 1. On the other hand, if the IP constraint flag is set to 1, the MIP enable flag is set to 0 and the MIP disable flag is set to 1.

Information about the use of a MIP can be signaled independently at higher or lower levels, or dependent on other syntactic elements.

As an example, information about the use of MIP can be signaled at a higher level, such as in a video parameter set (VPS), a sequence parameter set (SPS), a picture parameter set (PPS), a picture header (PH), or a slice header (SH).

As another example, information about the use of MIPs can be signaled at lower levels, such as coding tree units (CTUs), coding units (CUs), prediction units (PUs), or transform units (TUs). The use of MIPs is controlled on a block-by-block basis.

As another example, information regarding the use of MIP may be signaled not by an explicit flag, but may be signaled dependently based on flags of other prediction modes or the size of the current block, etc. The video decoding device may determine the MIP mode based on flags of other prediction modes or the size of the current block, etc.

FIG. 11 is a flowchart of an image decoding method according to one embodiment of the present disclosure.

In step S1110 of FIG. 11, the image decoding device decodes information about the intra prediction mode of the current block.

Information about the intra prediction mode may include information about the use of MIP, information about the reference template, matrix, or offset used for MIP prediction, etc.

In step S1120, the video decoding device determines the intra prediction mode of the current block based on information about the intra prediction mode. Here, the intra prediction mode may be a MIP mode.

In step S1130, the image decoding device performs MIP prediction for the current block. MIP prediction can be performed using the methods described in FIGS. 6 to 9.

Although the flowchart/timing diagram of this specification describes each process as being executed sequentially, this is merely an illustrative description of the technical idea of one embodiment of the present disclosure. In other words, a person of ordinary skill in the art to which one embodiment of the present disclosure belongs may modify and apply various modifications and variations by changing the order described in the flowchart/timing diagram without departing from the essential characteristics of one embodiment of the present disclosure, or by executing one or more of the processes in parallel. Therefore, the flowchart/timing diagram is not limited to a chronological order.

It should be understood that the exemplary embodiments described above can be implemented in many different ways. The functions or methods described in one or more examples can be implemented in hardware, software, firmware, or any combination thereof. It should be understood that the functional components described herein are labeled as "units" to further emphasize their implementation independence.

Meanwhile, the various functions or methods described in this embodiment may also be implemented as instructions stored on a non-transitory storage medium that can be read and executed by one or more processors. Non-transitory storage media include, for example, all types of storage devices that store data in a form readable by a computer system. For example, non-transitory storage media include storage media such as erasable programmable read-only memory (EPROM), flash drives, optical drives, magnetic hard drives, and solid-state drives (SSDs).

The above description is merely an example of the technical idea of the present embodiment, and those skilled in the art will appreciate that various modifications and variations can be made without departing from the essential characteristics of the present embodiment. Therefore, the present embodiments are not intended to limit the technical idea of the present embodiment, but rather to explain it, and the scope of the technical idea of the present embodiment is not limited by these embodiments. The scope of protection of the present embodiment should be interpreted by the claims below, and all technical ideas within a scope equivalent thereto should be interpreted as being included in the scope of rights of the present embodiment.

CROSS-REFERENCE TO RELATED APPLICATION

This patent application claims priority to Korean Patent Application No. 10-2024-0041177, filed in Korea on March 26, 2024, and Korean Patent Application No. 10-2025-0037374, filed in Korea on March 24, 2025, the entire contents of which are incorporated herein by reference.

Claims (12)

In the video decryption method, A step of determining a reference template containing decrypted pixels; A step of determining a weight matrix for matrix-based intra prediction of the current block; and A step of generating a prediction block of the current block based on the reference template of the current block and the weight matrix. A method for decrypting an image including: In the first paragraph, The above reference template is, An image decoding method including pixels adjacent to the current block among the decoded pixels. In the second paragraph, The above reference template is, An image decoding method determined based on either the intra prediction mode of the current block or signaled information. In the first paragraph, The above reference template is, An image decoding method including pixels that are not adjacent to the current block among the decoded pixels. In the first paragraph, The above reference template is, An image decoding method including a reference area indicated by motion information of surrounding blocks of the current block. In the first paragraph, At least one of the pixels within the above reference template, A method for decoding an image generated from other pixels using at least one of padding and interpolation. In the first paragraph, A step of adjusting the reference template by performing a weighted sum of pixels within the reference template in units of a predetermined shape. A method for decrypting an image further comprising: In the first paragraph, The step of generating a prediction block of the current block above is: A step of reducing the reference template based on at least one of the size or intra prediction mode of the current block being a predefined value; generating a boundary vector by multiplying one of the predefined matrices by the reduced reference template; and A step of generating a prediction block of the current block by upsampling the boundary vector. A method for decrypting an image including: In the first paragraph, The size of the above reference template is: An image decoding method determined based on at least one of the size of the current block or the size of the weight matrix. In the first paragraph, The prediction block of the current block above is, An image decoding method further generated based on an offset for matrix-based intra prediction of the current block. In the image encoding method, A step of determining a reference template containing encoded pixels; A step of determining a weight matrix for matrix-based intra prediction of the current block; and A step of generating a prediction block of the current block based on the reference template of the current block and the weight matrix. A method of encoding an image including: A method for transmitting data including a bitstream for an image, A step of generating a bitstream for the above image; and A step of transmitting data including the above bitstream Including, The step of generating the above bitstream is: A step of determining a reference template containing encoded pixels; A step of determining a weight matrix for matrix-based intra prediction of the current block; and A step of generating a prediction block of the current block based on the reference template of the current block and the weight matrix. How to include.