CN118657948A - Efficient Visual ViT Semantic Segmentation Method Based on Content Awareness and Token Sharing - Google Patents

Abstract

The present invention relates to an efficient visual ViT semantic segmentation method based on content perception and token sharing, and belongs to the field of image processing and computer vision technology. A token sharing strategy network is established, and the token sharing strategy network is trained until convergence, and an image I is input into the sharing strategy network to obtain a sharing strategy, and an image block and a token sharing strategy are input into a token sharing function to obtain a streamlined token set T' ; T' is input into a Transformer network to obtain a set of predicted token sets L containing semantic information; L and the sharing strategy are input into a token distribution function, and the token distribution function distributes and upsamples the predicted token set L according to the sharing strategy, and reorganizes it to obtain a predicted spatial feature and input it into a decoder to obtain a final semantic segmentation prediction result. The present invention can significantly improve the computational efficiency of a visual Transformer semantic segmentation network while ensuring the segmentation quality.

Description

Efficient Visual ViT Semantic Segmentation Method Based on Content Awareness and Token Sharing

Technical Field

The invention relates to an efficient visual ViT semantic segmentation method based on content perception and token sharing, and belongs to the technical field of image processing and computer vision.

Background Art

In recent years, many studies have proposed using Vision Transformer (ViT) to replace convolutional neural networks (CNNs) to solve various computer vision tasks such as image classification, object detection, and semantic segmentation. ViT processes a set of tokens generated by fixed-size image patches through a multi-layer multi-head self-attention mechanism. The existing ViT model has reached the latest research level on multiple tasks, especially the excellent pre-training performance on large-scale datasets, which significantly improves the performance of downstream tasks.

The semantic segmentation task requires classification of each pixel in the image, which places higher demands on the computational power and efficiency of the model. Traditional CNN methods gradually extract image features through layer-by-layer convolution operations, but these methods usually rely on a large receptive field to capture global information. In contrast, ViT can directly capture the global dependencies of the image using its self-attention mechanism, which theoretically helps improve the accuracy of the segmentation task.

In practical applications, ViT has been used for a variety of dense prediction tasks, including semantic segmentation tasks, and has shown excellent performance in many benchmarks. For example, the ViT model has achieved the highest performance level on standard datasets such as ADE20K and Cityscapes. The semantic segmentation architecture based on ViT divides the input image into fixed-size image patches (Patch), which are fed into the Transformer encoder after linear embedding and position encoding. The encoder uses a multi-head self-attention mechanism to process the image patch sequence to capture global context information. Subsequently, the encoder output is converted into a two-dimensional feature map by the decoder, and then restored to the original image size after upsampling. Finally, the segmentation head (usually a convolutional layer) is used to generate a category prediction for each pixel to complete the semantic segmentation task. This architecture combines the advantages of Transformer in modeling long-distance dependencies and the ability of traditional convolutional networks in processing local features, effectively improving the performance of semantic segmentation. However, despite the excellent performance of ViT in these tasks, its computational complexity and efficiency issues remain challenges that need to be addressed. To address these challenges, researchers have proposed a series of improvements, such as:

(1) Hybrid architecture: Some studies combine the advantages of CNN and ViT and propose hybrid architectures. These methods usually use convolutional layers to extract local features in the early stage and then use Transformer to capture global features in the later stage;

(2) Hierarchical Transformer: Introducing a hierarchical structure, the image is decomposed into image blocks of different scales to reduce the computational burden. For example, the Swin Transformer performs self-attention calculations at different scales through a layered window approach, thereby reducing complexity;

(3) Dynamic token cropping: Dynamically adjust the number of tokens to be processed according to the image content to reduce the waste of computing resources. For example, researchers proposed a dynamic cropping strategy that decides whether to keep each token based on its importance, thereby improving efficiency.

Although these improvements have solved the efficiency problem of ViT in semantic segmentation tasks to a certain extent, there are still some problems that need to be solved in this task scenario, such as:

(1) High computational complexity: The computational complexity of ViT is proportional to the square of the number of input tokens, which becomes particularly inefficient when processing large images. The traditional ViT architecture needs to process a large number of tokens when performing semantic segmentation tasks, which greatly increases the computational burden;

(2) Token redundancy: In semantic segmentation tasks, multiple image patches may contain the same semantic class, so in some cases it is unreasonable for these image patches to share the same token. This redundancy is not fully utilized by traditional ViT methods;

(3) Architecture complexity: To improve efficiency, some studies have proposed new ViT architectures that combine global and local attention or introduce CNN-like pyramid structures. However, these new architectures are often complex and require additional design and optimization.

Therefore, improving the efficiency of ViT-based methods in semantic segmentation task scenarios and significantly reducing computational complexity and resource consumption are issues that need to be addressed urgently.

Summary of the invention

The purpose of the present invention is to overcome the above-mentioned shortcomings and to provide an efficient visual ViT semantic segmentation method based on content perception and token sharing. By using an efficient strategy network to predict which image blocks can share tokens and reduce the number of tokens that need to be processed, the computational efficiency of the visual Transformer semantic segmentation network can be significantly improved while ensuring the segmentation quality.

The technical solution adopted by the present invention is:

An efficient visual ViT semantic segmentation method based on content perception and token sharing includes the following steps:

S1. Establish a token sharing strategy network p to predict which super-blocks in the input image contain only one semantic category, group those containing one semantic category, and then let the image blocks in each group of super-blocks share the same token;

S2. Train the token sharing strategy network p until the sharing strategy network p converges, input the image I into the sharing strategy network p , and obtain the sharing strategy P ;

S3. Divide the input image I into N image patches I _patch ;

S4. Based on the semantic segmentation architecture of ViT, the image block I _patch and the token sharing strategy P are input into the token sharing function t _s to obtain a simplified token set T ' ;

S5. Input the simplified token set T' into the Transformer network to obtain a set of predicted tokens L containing semantic information;

S6. Input the predicted token set L and the sharing strategy P into the token distribution function tu _. The token distribution function _tu distributes and upsamples the predicted token set L according to the sharing strategy P , and reorganizes the tokens to obtain the predicted spatial feature L _spat .

S7. Input the predicted spatial feature L _spat into the decoder to obtain the final semantic segmentation prediction result.

In the above method, the token sharing strategy network p described in step S1 realizes its function by training a CNN model, which predicts whether a super-block contains a single semantic category without predicting the specific category. If a super-block contains only one category, the image blocks therein can share the same token, and it only needs to implement binary classification for each super-block.

The token sharing policy network p described in step S2 is trained separately from the backbone network and is also category-independent. The labels required for training the token sharing policy network p are directly generated from the segmentation labels of the semantic segmentation dataset. For each super-block in the image, if it has only one semantic category in the segmentation label, the label of the token sharing policy network p is set to true, otherwise it is set to false. Using this binary label, the token sharing policy network p is trained to converge through standard cross entropy loss.

The token sharing function ts described in step S4 _maps the original image block I _patch and its position encoding to a new mapping set I' = {I' ₁ ,...,I' _M }, where M < N, and carries the corresponding position encoding. The mapping is performed according to the predicted sharing strategy P , which specifies which image blocks in the super-block should share tokens and which image blocks in the super- _block do not share the original tokens. After obtaining the mapping set I', the token sharing function ts projects it to the token set T'. This process can be formulated as:

,

Among them, the token sharing function ts _will downsample each super-block that can share tokens to the dimension of a single image block according to the sharing strategy P , and linearly project it into a single token, so as to achieve token sharing of each image block in the super-block.

The Transformer network described in step S5 can be any semantic segmentation network based on the Transformer architecture, preferably a segmentation network Segmenter based on ViT.

The token distribution function tu described in step S6 _first restores the token dimension through bilinear upsampling according to the sharing strategy P , that is, cancels the token sharing of each image block, and then reorganizes the token in space. The token is essentially a feature vector learned from each image block. The token is reorganized according to the position of the image block in the original image to obtain the predicted spatial feature L _spat .

The decoder described in step S7 is preferably based on CNN's UPerNet as a decoder.

An efficient visual ViT semantic segmentation system based on content perception and token sharing includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the efficient visual ViT semantic segmentation method based on content perception and token sharing as described above is implemented.

The beneficial effects of the present invention are:

In view of the high computational complexity and token redundancy of ViT, the present invention uses an efficient policy network to predict which image blocks can share tokens, thereby reducing the number of tokens that need to be processed. This can significantly improve the computational efficiency of the semantic segmentation network based on the visual Transformer while ensuring the segmentation quality. Compared with the existing token-based lightweight Transformer network technology, this method uses a more reliable binary classification network to determine whether tokens need to be shared, thereby ensuring the reliability of the lightweight process, that is, the segmentation efficiency can be significantly improved without losing the segmentation effect. The method proposed in the present invention also has a plug-and-play feature, that is, it is suitable for various ViT backbone networks and segmentation decoders, and is compatible with the latest pre-training methods. The simplicity and versatility of this method make it easy to integrate into all existing ViT-based models, providing an effective method for the study and application of efficient architectures for dense prediction tasks based on Transformer.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a network framework diagram of the method of the present invention;

FIG. 2 is a schematic diagram of image segmentation according to the present invention.

DETAILED DESCRIPTION

The present invention is further described below in conjunction with specific embodiments and drawings.

Embodiment 1 An efficient visual ViT semantic segmentation method based on content perception and token sharing includes the following steps (see FIG1 ):

S1. Establish a token sharing strategy network p to predict which super-blocks in the input image contain only one semantic category, group those containing one semantic category, and then let the image blocks in each group of super-blocks share the same token:

In the ViT-based semantic segmentation architecture, the input image I ∈ R ^3×H×W is divided into N image patches I _patch ={I ₁ ,...,I _N }, where H and W represent the height and width of image I, respectively. These image patches are converted into a set of tokens I _token ={T ₁ ,...,T _N } through linear projection, and then added with corresponding learnable position encodings, and then input into the Transformer. The total number of tokens N depends on the size of the input image and the predefined image patch size. The Transformer-based backbone network maps the input token Patch to a set of predicted tokens L ={L ₁ ,...,L _N } containing semantic information. The Transformer model can be any architecture that uses global self-attention. The next step is to input the output token L into the decoder. The decoder first rearranges the tokens into features according to their original positions. , where E is the feature dimension, H _L and W _L are the fractions of the original H and W. The feature L _spat is then input into the decoder g to obtain the segmentation prediction result. The problem to be solved by the present invention is to identify image blocks that can share tokens without reducing Under the premise of semantic segmentation quality, the size of T is reduced to reduce computational complexity and resource consumption.

The present invention is based on an obvious assumption, that is, there must be image blocks in the image that contain redundant information. This assumption is based on the phenomenon that pixels in a single object in the image are continuous. By analogy with the definition of superpixels, for example, 2×2 adjacent image blocks are defined as superblocks. The 2×2 superblocks that make up a single object only contain one semantic category. Therefore, there must be many semantically uniform areas in the image. It is assumed that efficiency can be improved without reducing segmentation quality by jointly processing these superblocks. Based on this, the present invention proposes a content-aware token sharing method, which first predicts which superblocks in the input image I contain only one semantic category, and then allows the image blocks in each group of 2×2 superblocks to share the same token.

In order to make the method applicable to any Transformer-based model, the present invention designs a token sharing function t _s , a token distribution function tu _, and a token sharing strategy model p . The token sharing function t _s , the token distribution function tu _, and the token sharing strategy model p are all implemented in a differentiable manner so as to be inserted into any current Transformer-based model for end-to-end training.

The role of the token sharing strategy network is to predict whether each super-block can share the same token without reducing the segmentation quality. The present invention achieves this function by training an efficient CNN model, which predicts whether a super-block contains a single semantic category without predicting a specific category. If a super-block contains only one category, the image blocks in it can share the same token. The model is defined as P = p ( I ), where the model network p . It can be implemented with any lightweight network, which only needs to implement binary classification for each super-block. In the present invention, Lite0, the version with the smallest number of parameters in the EfficientNet series network, is used to implement content perception and token sharing. Based on the output of network p , the S super-blocks with the highest scores are selected for token sharing, then the four image blocks in the S 2×2 super-blocks can share the same token, and finally retain M = N −3 S tokens. These shared tokens are sent to the Transformer network for learning, where N is the number of image blocks into which the image I is divided.

S2. Train the token sharing policy network p until the sharing policy network p converges, input the image I into the sharing policy network p , and obtain the sharing policy P :

The token sharing policy network is trained separately from the backbone network and is also class-independent, and the labels required to train the token sharing policy network p can be directly generated from the segmentation labels of the semantic segmentation dataset. Specifically, for each hyperblock in the image, if it has only one semantic category in the segmentation label, the label of the token sharing policy network p is set to true, otherwise it is set to false. Using this binary label, the token sharing policy network p is trained with the standard cross entropy loss. The converged policy model p is used for subsequent operations.

The token sharing strategy model p can be trained in this simple and efficient way because the semantic segmentation task has realistic and available semantic labels. Semantic segmentation annotation provides a label for each pixel, which enables the present invention to directly convert the assumption about the redundancy of semantically similar image blocks into a binary label for each super-block. In addition, the fact that the token sharing strategy model p is independent of the category is the key to the efficiency of the network. The binary classification task is much simpler than the multi-class segmentation task, so it can be performed by an efficient model, which is very critical to obtain efficiency improvements by reducing the number of tokens.

The image I is input into the trained token sharing strategy model p to obtain the sharing strategy P , which indicates whether the image block in each 2×2 super block can share the token.

S3. Divide the input image I into N image patches I _patch :

As shown in FIG2 , the Transformer-based backbone network receives an image block as input. The present invention first divides the input image into N patches and defines adjacent 2×2 image blocks as super-blocks.

S4. Based on the semantic segmentation architecture of ViT, the image patch I _patch and the token sharing strategy P are input into the token sharing function t _s to obtain a simplified token set T' :

The token sharing function ts (also known as the token sharing module, which is _the set of operations below) maps _{the original image block Ipatch} and its position encoding to a new mapping set I' = { _I'1 ,...,I _'M }, where M < N, with corresponding position encodings. This mapping is performed according to the predicted sharing strategy P , which specifies which image blocks in the super-block should share tokens and which image blocks in the super-block do not share the original tokens. After obtaining the mapping set I', the token sharing function ts projects it to the token set T' for subsequent processing _by the Transformer network. The above process can be formulated as:

,

Among them, the token sharing function ts _will downsample each super-block that can share tokens to the dimension of a single image block according to the sharing strategy P , and linearly project it into a single token, so as to achieve token sharing for each image block in the 2×2 super-block.

S5. Input the simplified token set T' into the Transformer network to obtain a set of predicted tokens L containing semantic information:

After obtaining the streamlined token set T' , it is input into the Transformer network f for feature learning to obtain a set of prediction tokens L = {L ₁ ,...,L _M } containing semantic information. The process can be formulated as:

,

The Transformer network f can be any semantic segmentation network based on the Transformer architecture. In the present invention, the classic model of the ViT-based segmentation network Segmenter is used as the network f for learning.

S6. Input the predicted token set L and the sharing strategy P into the token distribution function tu _. The token distribution function _tu distributes and upsamples the predicted token set L according to the sharing strategy P , and reorganizes the tokens to obtain the predicted spatial feature L _spat :

The predicted token set L and the sharing strategy P are input into the token distribution function tu _. The token distribution function tu ₍ i.e., the token distribution module, which is the following set of operations) first restores the token dimension by bilinear upsampling according to the sharing strategy P , that is, cancels the token sharing of each block, and then spatially reorganizes the token into the predicted spatial feature Lspat . In this process, the sharing strategy P provides the index corresponding to the image block that has been token _- shared in step S4, and the token distribution function tu _performs upsampling based on the index. The process can be formulated as:

.

S7. Input the predicted spatial feature L _spat into the decoder to obtain the final semantic segmentation prediction result:

For the predicted spatial feature L _spat , it is input into the decoder g to obtain the final semantic segmentation prediction result O . The process can be formulated as:

,

Among them, the decoder g can adopt a variety of network structures. In the present invention, for efficiency considerations, CNN-based UPerNet is used as a decoder to decode the features.

Example 2: An efficient visual ViT semantic segmentation system based on content perception and token sharing, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the efficient visual ViT semantic segmentation method based on content perception and token sharing as described in Example 1 is implemented.

The above is a further description of the present invention in combination with the embodiments, and the protection scope of the present invention is not limited thereto.

Claims (8)

1. An efficient visual ViT semantic segmentation method based on content perception and token sharing, characterized in that it includes the following steps: S1. Establish a token sharing strategy network p to predict which super-blocks in the input image contain only one semantic category, group those containing one semantic category, and then let the image blocks in each group of super-blocks share the same token; S2. Train the token sharing strategy network p until the sharing strategy network p converges, input the image I into the sharing strategy network p , and obtain the sharing strategy P ; S3. Divide the input image I into N image patches I _patch ; S4. Based on the semantic segmentation architecture of ViT, the image block I _patch and the token sharing strategy P are input into the token sharing function t _s to obtain a simplified token set T ' ; S5. Input the simplified token set T' into the Transformer network to obtain a set of predicted tokens L containing semantic information; S6. Input the predicted token set L and the sharing strategy P into the token distribution function tu _. The token distribution function _tu distributes and upsamples the predicted token set L according to the sharing strategy P , and reorganizes the tokens to obtain the predicted spatial feature L _spat . S7. Input the predicted spatial feature L _spat into the decoder to obtain the final semantic segmentation prediction result. 2. According to the efficient visual ViT semantic segmentation method based on content perception and token sharing according to claim 1, it is characterized in that the token sharing strategy network p described in step S1 realizes its function by training a CNN model, which predicts whether a super-block contains a single semantic category without predicting the specific category. If a super-block contains only one category, the image blocks therein can share the same token, and it only needs to implement binary classification for each super-block. 3. According to the content-aware and token-sharing-based efficient visual ViT semantic segmentation method of claim 1, it is characterized in that the token sharing strategy network p described in step S2 is trained separately from the backbone network and is also category-independent, and the labels required for training the token sharing strategy network p are directly generated from the segmentation labels of the semantic segmentation dataset, and for each super-block in the image, if it has only one semantic category in the segmentation label, the label of the token sharing strategy network p is set to true, otherwise it is set to false; using this binary label, the token sharing strategy network p is trained to converge through standard cross entropy loss. 4. The efficient visual ViT semantic segmentation method based on content perception and token sharing according to claim 1 is characterized in that the token sharing function t _s described in step S4 maps the original image block I _patch and its position encoding to a new mapping set I' = {I' ₁ ,...,I' _M }, where M < N, and with corresponding position encoding, and the mapping is performed according to the predicted sharing strategy P , and the sharing strategy P specifies which image blocks in the super-block should share tokens and which image blocks in the super-block do not share the original tokens. After obtaining the mapping set I', the token sharing function t _s projects it to the token set T'. This process can be formulated as: , Among them, the token sharing function ts _will downsample each super-block that can share tokens to the dimension of a single image block according to the sharing strategy P , and linearly project it into a single token, so as to achieve token sharing of each image block in the super-block. 5. According to the efficient visual ViT semantic segmentation method based on content perception and token sharing according to claim 1, it is characterized in that the Transformer network described in step S5 can be any semantic segmentation network based on the Transformer architecture. 6. The efficient visual ViT semantic segmentation method based on content perception and token sharing according to claim 1 is characterized in that the token distribution function tu described in step S6 _first restores the token dimension through bilinear upsampling according to the sharing strategy P , that is, cancels the token sharing of each image block, and then reorganizes the token according to the position of the image block in the original image to obtain the predicted spatial feature L _spat . 7. According to the efficient visual ViT semantic segmentation method based on content perception and token sharing according to claim 1, it is characterized in that the decoder described in step S7 selects CNN-based UPerNet as a decoder. 8. An efficient visual ViT semantic segmentation system based on content perception and token sharing, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein when the processor executes the program, the efficient visual ViT semantic segmentation method based on content perception and token sharing as described in any one of claims 1 to 7 is implemented.