CN111461110A - Small target detection method based on multi-scale image and weighted fusion loss - Google Patents

Abstract

The invention belongs to the field of image and video processing, and relates to a small target detection method based on multi-scale images and weighted fusion loss. Perform fusion to construct a feature pyramid; generate candidate detection frames based on the feature pyramid and filter to obtain suggested detection frames; map the suggested detection frames back to the feature map to generate them in the feature pyramid, align and intercept on the feature map; detect the aligned suggestions The box is input into the classifier layer, and the category confidence and position offset of the proposed detection box are obtained; in the testing phase, certain suggested detection boxes are screened according to the category confidence score of the proposed detection box, and non-maximum value suppression is performed; In the training stage, the loss function calculated by the feature layer for detecting small targets is weighted and fused with the loss function for detecting large and medium target layers to enhance the sensitivity of the model to small target objects.

Description

A small object detection method based on multi-scale images and weighted fusion loss

technical field

The invention belongs to the field of image and video processing, and relates to a small target detection method based on multi-scale images and weighted fusion loss.

Background technique

With the development of machine learning and deep learning, relying on the powerful learning ability of convolutional neural networks, the fields of pattern recognition and computer vision have received unprecedented attention and popularity. In the era of widespread machine automation and artificial intelligence, the role of the camera is increasingly equal to that of the human eye. The development of computer vision is particularly important and has received extensive attention from industry and academia. Among them, target detection has achieved remarkable results in the field of computer vision and is constantly improving. However, most of the target objects in pictures and videos appear in extremely tiny forms. Usually there are many objects occupying very low pixels in a frame of pictures, most of the pixels are less than 49px, the task of detecting tiny objects is arduous and very important.

The difficulty of small target detection lies in the scale. Inputting the network model with a single size to extract feature information often cannot take care of the targets of various scales. Although the existing Mask RCNN model has achieved good results in object detection, there are still problems such as single input image scale, uncertain resolution, insufficient use of context information, and insensitivity to small object detection.

SUMMARY OF THE INVENTION

Aiming at the shortcomings of the existing Mask RCNN model, the present invention provides a small target detection method based on multi-scale images and weighted fusion loss.

The present invention adopts the following technical scheme to realize:

A small object detection method based on multi-scale images and weighted fusion loss, implemented based on the improved Mask RCNN model, including:

S1. Build an improved Mask RCNN model; the improved Mask RCNN model includes: a residual backbone network, a feature pyramid network layer, a region generation network layer, a frame of interest alignment layer, a classifier layer, a loss function calculation layer and a test layer ;

S2. Constructing an image pyramid: scaling the original image, and combining the original image, the reduced-sized image, and the enlarged-sized image together to form an image pyramid;

S3. Randomly crop the images in the image pyramid;

S4. Send the randomly cropped image to the residual backbone network for convolution, batch normalization, and pooling operations, and output multiple sets of feature maps of different sizes;

S5. Fusion of multiple sets of feature maps of different scales, and further processing to obtain feature maps P2-P6;

S6. Generate unscreened candidate detection frames for the feature maps P2-P6 respectively;

S7. Generate the network layer from the input area of the feature map P2-P6, and obtain the offset and confidence of the candidate detection frame through a series of convolution operations;

S8, combine the offset of the candidate detection frame of S7 with the unscreened candidate detection frame data obtained in S6, and screen out the set amount of candidate detection frames as the detection frame of interest;

S9. Corresponding the detection frames of interest back to the feature maps P2-P6 respectively, and perform an alignment operation;

S10. Input the result of the alignment operation to the classification layer, and output the predicted interest detection frame category score, category probability, and coordinate offset;

S11. Input the predicted interest detection frame category score, category probability, and coordinate offset into the test layer, select the maximum value of the category probability in the test layer, select the predicted target category corresponding to the interest detection frame, and further pass the non- The maximum value suppression filters out redundant interest detection frames, and finally obtains the final predicted interest detection frame and the corresponding predicted target category in the test layer.

Further, the training phase also includes:

S12. Input the category score of the detection frame of interest predicted in S10 into the loss function calculation layer, and use it as the input of the cross-entropy function together with the actual category label to calculate the classification loss value, and obtain the category prediction loss of the feature maps P2-P6;

The coordinate offset of the detection frame of interest predicted in S10 and the offset of the real target frame are used as the input of the regression loss function, and the regression prediction loss of the feature maps P2-P6 is obtained;

S13, weight the category prediction losses of the feature map P2 and the feature map P3 respectively, and add them to the category prediction losses of the feature map P4, the feature map P5, and the feature map P6 to obtain the total category prediction loss;

The regression prediction loss of the feature map P2 and the feature map P3 are weighted respectively, and the total regression prediction loss is obtained by adding the regression prediction loss of the feature map P4, the feature map P5, and the feature map P6;

S14. Iteratively update the parameters and weights of the improved Mask RCNN model through backpropagation. Specifically, the total category prediction loss and the total regression prediction loss are respectively used to perform optimization iterations and change the weight value of the improved Mask RCNN model.

Further, the improvements made by the improved Mask RCNN model include:

1. The alignment of the detection frames of interest is no longer unified, but the different feature layers are aligned separately. After alignment, the incoming loss function calculation layer is not directly integrated, but is input to the classifier layer for separate classification and regression, and finally separate input The loss function calculation layer weights the loss function calculated by the feature layer for detecting small targets, and fuses it with the loss function for detecting large and medium target layers;

2. Add a layer of effective feature layer P6 to the original Mask RCNN model;

3. Remove the image segmentation module in the original Mask RCNN and cancel the mask branch.

Preferably, scaling the original image in S2 includes:

The formula for scaling an image is:

Image_New=Image*scale (1)

Among them: Image_New represents the zoomed image, Image represents the image before zooming, and scale represents the zoom scale;

The scaling scale is determined by the following factors:

After the scaling is completed, the length of the minimum side cannot be less than min_dim, min() means taking the minimum value operation, h means the height of the original image, w means the width of the original image, then when min_dim is greater than min(h, w),

scale=min_dim/min(h, w) (2)

otherwise scale=1;

Set the length of the longest side after scaling is max_dim, if the image is scaled according to formula (2), if the longest side of the scaled image has exceeded max_dim, then make:

scale=max_dim/image_max (3)

Otherwise, continue to scale by scale=min_dim/min(h, w);

The size of the final scaled picture is max_dim*max_dim. In addition, if the scale of the final scaled scale is greater than 1, that is, the original picture is enlarged, it will be enlarged by bilinear interpolation; the final scaled picture is less than max_dim*max_dim part, padded pixel values with zero values.

Preferably, the formula for randomly cropping pictures in S3 is expressed as follows:

Y ₁ =randi([0,image_size(1)-crop_size(1)]) (4)

X ₁ =randi([0,image_size(2)-crop_size(2)]) (5)

Among them: Y1 and X1 represent the ordinate and abscissa of the lower left corner of the beginning of the cropped image, respectively; randi represents a random number, and the range of the number is the range in the parentheses; image_size is the size of the image before cropping, and the first dimension stores the image width, the second dimension stores the image length; crop_size is the size of the area to be cropped, the first dimension storage area width, and the second dimension storage area length;

Y ₂ =min(image_size(1),Y1+crop_size(1)) (6)

X ₂ =min(image_size(2),X1+crop_size(2)) (7)

Among them: Y2 and X2 represent the upper right ordinate and upper right abscissa of the beginning of the cropped image, respectively; randi means random number; min() means the minimum value;

Use the two coordinates obtained from equations (4) to (7) to determine the specific position of the cropping. If the cropping area overflows the original image, pad filling will be performed to obtain the cropped image.

Preferably, the convolution of the residual backbone network includes two types of convolution modules, block1 and block2, wherein:

The convolution module block1 workflow includes:

①. For branch 1, the output and input remain the same;

②. For branch 2, use 1*1 convolution kernel, 3*3 convolution kernel, and 1*1 convolution kernel to perform convolution operation in turn, and perform mean normalization on the output feature vector after each convolution is completed. change;

The convolution module block2 workflow includes:

1. For branch 1, use a 1*1 convolution kernel to perform the convolution operation, and then perform mean normalization on the output feature vector;

②. For branch 2, use 1*1 convolution kernel, 3*3 convolution kernel, and 1*1 convolution kernel to perform convolution operation in turn, and perform mean normalization on the output feature vector after each convolution is completed. change.

Preferably, outputting multiple sets of feature maps of different sizes in S4 includes:

For the original image input, five layers of feature map output are formed: C2, C3, C4, C5, C6; for input that is doubled from the original image, five layers of feature map output are formed: C2s, C3s, C4s, C5s, C6s; for The input that is doubled relative to the original image constitutes five-layer feature map output: C2l, C3l, C4l, C5l, C6l.

Preferably, step S5 includes:

S51. Perform upsampling based on the interpolation principle on C2s-C6s, and double the C2s-C6s;

S52, perform maximum pooling on C2l-C6l, and double the size of C2l-C6l;

S53, adding C2-C6, doubled C2s-C6s and doubled C2l-C6l to obtain C2-C6 fused with image features of different scales;

S54, further processing C2-C6 fused with image features of different scales to obtain feature maps P2-P6.

Preferably, S54 includes:

Use 256 1*1 convolution kernels to convolve C6 that incorporates image features of different scales to obtain a feature map P6 with an output of 16*16*256;

Use 256 1*1 convolution kernels to convolve C5 that incorporates image features of different scales and add the output of P6 upsampling twice, and then perform 3*3 convolution to obtain an output of 32*32*256 The feature map P5 of;

Use 256 1*1 convolution kernels to convolve C4 that combines image features of different scales and add the output of P5 upsampling twice, and then perform 3*3 convolution to obtain an output of 64*64*256 The feature map P4 of;

Use 256 1*1 convolution kernels to convolve C3, which combines image features of different scales, and add the output of P4 upsampling twice, and then perform 3*3 convolution to obtain an output of 128*128*256 The feature map P3 of ;

Use 256 1*1 convolution kernels to convolve C2, which combines image features of different scales, and add the output of P3 upsampling twice, and then perform 3*3 convolution to obtain an output of 256*256*256 Feature map P2.

Preferably, the height h and width w of the candidate detection frame are:

Among them: scale_length indicates that when the candidate detection frame is a box with the same height and width, the height and width correspond to the pixel-level size of the original image.

Compared with the small target detection of the existing Mask RCNN model, the present invention has the following beneficial effects:

(1) A method of constructing an image pyramid as the input of the Mask RCNN model is proposed. An image is preprocessed into multiple scales as input instead of a single scale, which enhances the ability to extract features of small objects.

(2) Remove the image segmentation module in the original Mask RCNN model, cancel the Mask (mask) branch, reduce the network parameters, and make the training classification and regression part more efficient.

(3) The alignment of the region of interest is no longer unified, but to align different feature layers separately, and then input the classification layer for separate classification and regression, and finally separate the input loss function calculation layer to detect the small target feature layer. The loss function is weighted and fused with the loss function of detecting large and medium target layers, which enhances the influence of small target objects on the model loss function, and allows the model to better learn the characteristics of detecting small target objects.

(4) A layer of effective feature layer P6 is added to the original Mask RCNN model, which improves the detection accuracy of small target objects and ensures that the detection accuracy of large objects does not decrease, and finally achieves the improvement of the detection accuracy of small target objects in target detection. Improved performance and precision.

Description of drawings

1 is a structural diagram of an improved Mask RCNN model in an embodiment of the present invention;

2 is a schematic diagram of a convolution block in an improved Mask RCNN model in an embodiment of the present invention;

FIG. 3 is a flowchart for realizing small target detection in an embodiment of the present invention.

Detailed ways

The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

A small object detection method based on multi-scale images and weighted fusion loss, implemented based on the improved Mask RCNN model, including:

S1. Build an improved Mask RCNN model.

In a preferred embodiment, the improved Mask RCNN model includes a backbone network part, a candidate window generation part and a classification layer part, built using the keras platform, including: a residual backbone network, a feature pyramid network layer, a region proposal layer, an interesting Box alignment layer, classifier layer, loss function calculation layer, test layer. Compared with the original Mask RCNN, the improvements include:

①. The alignment of the region of interest is no longer unified, but to align different feature layers separately. After alignment, the incoming loss function is not directly integrated, but is input to the classifier layer for separate classification and regression, and finally the input loss function is calculated separately. layer, weights the loss function calculated by the feature layer for detecting small targets, and integrates it with the loss function for detecting large and medium target layers, enhancing the influence of small target objects on the loss function of the model, allowing the model to better learn about detection. Characteristics of small target objects.

2. Add a layer of effective feature layer P6 to the original Mask RCNN model to improve the detection accuracy of small target objects while ensuring that the detection accuracy of large objects does not decrease, and ultimately improve the detection accuracy of small target objects in target detection. Accuracy improvements.

3. Remove the image segmentation module in the original Mask RCNN and cancel the Mask (mask) branch, thereby reducing the network parameters and making the training classification and regression parts more efficient.

S2. Build an image pyramid.

The original image data is reduced by one time and enlarged by one time, and the original image data is retained. The original image, the reduced image, and the enlarged image together form an image pyramid.

The first step to change the image size is to zoom the image. The formula for zooming the image is expressed as:

Image_New=Image*scale (1)

Among them: Image_New represents the zoomed image, Image represents the image before zooming, and scale represents the zoom scale. The scaling scale is determined by the following factors:

After the scaling is completed, the length of the minimum side cannot be less than min_dim, min() means taking the minimum value operation, h means the height of the original image, w means the width of the original image, then when min_dim is greater than min(h, w),

scale=min_dim/min(h, w) (2)

Otherwise scale=1.

Set the length of the longest side after scaling is max_dim, if the image is scaled according to formula (2), if the longest side of the scaled image has exceeded max_dim, then make:

scale=max_dim/image_max (3)

Otherwise, continue to scale according to scale=min_dim/min(h, w).

The size of the final scaled image is max_dim*max_dim. In addition, if the scale of the final scaled scale is greater than 1, that is, the original image is enlarged, it will be enlarged by bilinear interpolation. For the part of the final scaled image that is less than max_dim*max_dim, the pixel value is filled with zero value.

S3. Perform random cropping of 512*512 on the original image, the reduced image, and the enlarged image in the image pyramid as a training input.

The formula for randomly cropping a picture is as follows:

Y ₁ =randi([0,image_size(1)-crop_size(1)]) (4)

X ₁ =randi([0,image_size(2)-crop_size(2)]) (5)

Among them: Y1 and X1 represent the ordinate and abscissa of the lower left corner of the beginning of the cropped image, respectively; randi represents a random number, and the range of the number is the range in the parentheses; image_size is the size of the image before cropping, and the first dimension stores the image Width, the second dimension stores the image length; crop_size is the size of the area to be cropped, the first dimension stores the width of the area, and the second dimension stores the length of the area.

Y ₂ =min(image_size(1),Y1+crop_size(1)) (6)

X ₂ =min(image_size(2),X1+crop_size(2)) (7)

Among them: Y2 and X2 represent the upper right ordinate and upper right abscissa of the beginning of the cropped image, respectively; randi means random number, the range of the number is the range in the parentheses, min() means the minimum value, and the parentheses are the comparison the two numbers.

Use the two coordinates obtained above to determine the specific position of the cropping. If the cropping area overflows the original image, pad filling will be performed to obtain the cropped image. Pad padding will zero-pad the overflow area with three channels of pixels, that is, assign 0 to each channel.

S4. Send the cropped image to the residual backbone network for convolution, batch normalization, and pooling operations, and output three sets of feature maps of different sizes.

In a preferred embodiment, the residual backbone network performs a total of 60 convolutions, as well as a maximum pooling at the beginning and an average pooling at the end, and a batch normalization is performed after each convolution. Using a specific convolution module, according to the size relationship of the feature map after convolution, for the original image input, five layers of feature map output are formed: C2, C3, C4, C5, C6; for the input that is doubled compared to the original image, the composition Five-layer feature map output: C2s, C3s, C4s, C5s, C6s; for the input that is doubled relative to the original image, five-layer feature map output is formed: C2l, C3l, C4l, C5l, C6l.

The convolution formula is expressed as follows:

Among them, Output is the value of each point in the feature map of the convolution output, w _{i, j} is the weight of the n*n size convolution kernel at the (i, j) position; Input _{i', j'} is the input convolution kernel The pixel value of the map at the position corresponding to the convolution kernel position (i, j).

For the maximum pooling operation, a kernel of size 3*3 is selected, and the input graph is slid with a step size of 2, and the maximum value in the sliding 3*3 local receptive field is selected as the value of the corresponding point of the output graph. The formula expresses for:

Output _max = max(Area _input ) (9)

Where: Area _input represents the value of all pixels in the local receptive field.

For the average pooling operation, the values of all pixels in the input feature map are summed and then averaged, and finally a 1*1 output is obtained without changing the number of channels. The formula is expressed as follows:

Among them: Input _{i, j} is the pixel value of the input image at the pixel point (i, j), and n*n refers to the size of the input feature map.

Batch normalization refers to transforming the value distribution of each point of the convolution graph output by each hidden layer of the deep neural network into a normal distribution value with 0 as the mean and unit variance, which can speed up the network convergence. For a batch of inputs, assuming there are n samples, the output in a hidden layer I is: {z ⁽¹⁾ ,z ⁽²⁾ ,z ⁽³⁾ ,z ⁽⁴⁾ ,...,z ⁽ⁿ⁾ } , and average the batch of outputs:

Find the variance:

The output normalization (batch normalization) is to perform the following operations on the output of each input sample:

Among them: ∈ is set to prevent invalid calculation when the variance is 0.

After batch normalizing the output of the I layer to a distribution with a mean of 0 and a variance of 1, the information expression of the original data is changed, so that the parameter information learned by the underlying network is lost, resulting in a lack of data expression ability. Based on this deficiency, the present invention introduces two learnable parameters γ and β, so as to perform the following operations on the batch normalized output obtained above:

This restores the expressiveness of the data itself.

In a preferred embodiment, the activation function of the improved Mask RCNN model uses the Sigmoid function uniformly, and its mathematical expression is:

S5, fuse the three sets of feature maps C2s-C6s, C2-C6, C2l-C6l obtained in S4 with different scales, and further obtain feature maps P2-P6. The specific operation method is as follows:

2. Upsampling C2s-C6s based on the interpolation principle to double C2s-C6s;

2. Perform maximum pooling on C2l-C6l and double the size of C2l-C6l;

The principle of maximum pooling is as follows: select a 3*3 kernel, slide in the input graph with a step size of 2, and select the maximum value in the sliding 3*3 area as the value of the corresponding point in the output graph. The formula expresses for:

Output _max = max(Area _input ) (16)

Where: Area _input represents the value of all pixels in the local receptive field.

③. Add C2-C6 to the doubled C2s-C6s and the doubled C2l-C6l to obtain the C2-C6 fused with image features of different scales.

At this time, C2-C6 is different from C2-C6 in S4. At this time, C2-C6 combines the features extracted from images of different scales.

④. Further processing the C2-C6 fused with image features of different scales to obtain feature maps P2-P6.

Specifically: 256 1*1 convolution kernels are used to convolve C6 that incorporates image features of different scales to obtain a feature map P6 with an output of 16*16*256.

Use 256 1*1 convolution kernels to convolve C5 that incorporates image features of different scales and add the output of P6 upsampling twice, and then perform 3*3 convolution to obtain an output of 32*32*256 The feature map of P5.

Use 256 1*1 convolution kernels to convolve C4 that combines image features of different scales and add the output of P5 upsampling twice, and then perform 3*3 convolution to obtain an output of 64*64*256 The feature map of P4.

Use 256 1*1 convolution kernels to convolve C3, which combines image features of different scales, and add the output of P4 upsampling twice, and then perform 3*3 convolution to obtain an output of 128*128*256 The feature map of P3.

Use 256 1*1 convolution kernels to convolve C2, which combines image features of different scales, and add the output of P3 upsampling twice, and then perform 3*3 convolution to obtain an output of 256*256*256 Feature map P2.

The P2-P6 obtained above are combined together to form a feature map matrix.

S6. Generate unscreened candidate detection frames on the feature maps P2-P6 obtained in S5.

The height h and width w of the candidate detection frame are:

Among them: scale_length indicates that when the candidate detection frame is a box with the same height and width, the height and width correspond to the pixel-level size of the original image. For P2 to P6, 32, 64, 128, 256, 512, respectively. ratios represents the three scales of each size candidate detection box: 0.5, 1, 2.

The feature maps P2-P6 are centered on the pixel points to generate candidate detection frames of different sizes and scales, and each pixel point on the feature map will generate a candidate detection frame. The coordinates of the center of the candidate detection frame are each pixel of the feature map of each layer.

S7. Input the feature maps P2-P6 obtained in S5 into RPN (Region Proposal Network, region generation network layer), and use the same convolution layer to convolve the feature maps P2-P6, but do not change the size of the feature map. Immediately after the convolution, the feature map is convolved with a 1*1 convolution kernel to obtain rpn_score, and then the softmax operation is performed, and the number of output channels is (2* the number of candidate detection frames per pixel) candidate detection The frame confidence rpn_pro, on this basis, is convolved through a 1*1 convolution kernel, and the output channel number is (4* the number of candidate detection frames that are not screened for each pixel) candidate detection frame offset information rpn_bbox.

S8. Combine the rpn_bbox of S7 with the unscreened candidate detection frame data obtained in S6 to generate the final interesting detection frame (also known as ROI frame or suggested detection frame) rpn box (complete, filtered candidate detection frame) ), and obtain the score of the detection frame of interest from the rpn_score obtained by S7, arrange the scores of the detection frame of interest from large to small, do non-maximum suppression, and finally retain the first 1500 detection boxes of interest (rpn boxes).

S9. For the 1500 detection frames of interest obtained in S8, perform an alignment operation corresponding to the feature maps from P2 to P6 respectively.

For the P5 feature layer, the average size of the detection target is 224*224. When aligning, the receptive field of the P5 layer is 32 compared to the original image, so it should be aligned to (224/32=7)*(224/32=7 ) is the size of 7*7. The receptive fields of layers P2 to P6 increase in a 2-fold relationship, and so on, P2, P3, P4, and P6 will also be aligned to a size of 7*7. After the alignment is completed, the outputs are: align_p2, align_p3, align_p4, align_p5, and align_p6.

S10. After the alignment in S9 is completed, output align_p2, align_p3, align_p4, align_p5, and align_p6 and input them to the classification layer, convert align_p2, align_p3, align_p4, align_p5, and align_p6 into vectors by convolution operation, and output the category score that predicts the detection frame of interest , class probability, coordinate offset.

Specifically, use 7*7 convolution to convert the vector (x, 256, 7, 7) into (x, 256, 1, 1) data, and then use full connection to convert the vector (x, 256, 1, 1) Converted to (x, 81), the output is the category score, of which 81 represents the 81 categories of the ROI box; the softmax operation is performed on the vector (x, 81), and the output is the category probability; then the full connection operation is used to put (x, 256, 1, 1) into (x, 81*4), where 4 represents the 4 coordinate offsets of the 81 categories of the ROI box. Note that "x" above represents the number of interesting detection frames in each layer of feature maps. In different layers, the value of "x" may be different.

S11. In the testing stage, after reaching S10, input the category score, category probability, and coordinate offset into the test layer, wherein the maximum value of the category probability is screened in the test layer, and the target category corresponding to the prediction of the interesting detection frame is selected. , and further use a threshold of 0.7 for non-maximum suppression to filter out redundant detection frames of interest. Finally, the final predicted interest detection frame and the corresponding predicted target category will be obtained in the test layer.

During the training phase, it also includes:

The loss function calculated by the feature layer for detecting small targets is weighted and fused with the loss function for detecting large and medium target layers to enhance the influence of small target objects on the loss function of the model, so that the improved Mask RCNN model can better learn about Detect features of small target objects.

Specifically, including:

S12. Input the 81 category scores output by each interest detection frame in the feature maps P2-P6 of each layer in S10 into the loss function calculation layer, and use the actual category labels as the input of the cross entropy function to calculate the classification loss value , get the class prediction loss. where the cross entropy function is expressed as:

Among them: y' _i is the ith value in the true category label, c indicates the total number of category labels, y _i is the corresponding value in the vector after the softmax normalization of the predicted category, the more accurate the classification, the closer yi _i is to 1, the loss The function value Loss _y' (y) is smaller.

The 4 coordinate offsets of 81 categories output by each interesting detection frame of each layer feature map in S10 are used as the input of the regression loss function smooth_l1 together with the real target frame offset to obtain the regression prediction loss. Its function is expressed as:

Among them: x represents the difference between the offset coordinates of the predicted detection frame of interest and the offset coordinates of the real target frame.

S13. Weight the class prediction losses Loss_class_P2 and Loss_class_P3 of the output P2 and P3 layers in S12 respectively, and add them to Loss_class_P4, Loss_class_P5, and Loss_class_P6 to obtain the total class prediction loss:

Loss_class_P2'=4*Loss_class_P2

Loss_class_P3'=2*Loss_class_P3

Loss_class=Loss_class_P2'+Loss_class_P3'+Loss_class_P4+Loss_class_P5+Loss_class_P6

(twenty one)

The regression prediction losses Loss_reg_P2 and Loss_reg_P3 of the output P2 and P3 layers in S12 are weighted respectively, and added to Loss_reg_P4, Loss_reg_P5, and Loss_reg_P6 to obtain the regression prediction loss:

Loss_reg_P2'=4*Loss_reg_P2

Loss_reg_P3'=2*Loss_reg_P3

Loss_reg=Loss_reg_P2'+Loss_reg_P3'+Loss_reg_P4+Loss_reg_P5+Loss_reg_P6

(twenty two)

In the end, Loss_class and Loss_reg are used respectively to perform optimization iteration, modify the weight value of the improved MaskRCNN model, and achieve the learning purpose of the improved MaskRCNN model.

The present invention will be described in further detail below with reference to the accompanying drawings.

See Figure 1-3, a small object detection method based on multi-scale images and weighted fusion loss, including:

(1) The residual backbone network based on the improved Mask RCNN model performs feature extraction on the original image, doubled and doubled images, and obtains three sets of features of different scales, and performs feature fusion on the three sets of features of different scales , to obtain output vector graphs C2, C3, C4, C5 and C6 that incorporate features of different scales.

In a preferred embodiment, different convolution modules are mainly used to extract features of pictures of different scales. The convolution modules in the feature extraction layer (residual backbone network) are shown in Figure 2, and there are two types, referred to as "block1" and "block2" for short.

The convolution module block1 workflow includes:

①. For branch 1, the output and input remain the same.

②. For branch 2, use 1*1 convolution kernel, 3*3 convolution kernel, and 1*1 convolution kernel to perform convolution operation in turn, and perform mean normalization on the output feature vector after each convolution is completed. change. Specifically, the number of feature vector channels after each convolution operation is completed is in a ratio of 1:1:4.

The convolution module block2 workflow includes:

①. For branch 1, use a 1*1 convolution kernel to perform the convolution operation, and then perform mean normalization on the output feature vector.

②. For branch 2, use 1*1 convolution kernel, 3*3 convolution kernel, and 1*1 convolution kernel to perform convolution operation in turn, and perform mean normalization on the output feature vector after each convolution is completed. change. Specifically, the number of feature vector channels after each convolution operation is completed is in a ratio of 1:1:4.

In a specific embodiment, assuming that the picture is input as x*x*3, it is noted that for the original picture, the size of the picture is unified to the standard size of 1024*1024, that is, x=1024, and x=512 after doubling, and zooming in x=2048 after doubling. For simplicity, the description of the mean normalization operation that must be performed after each convolution operation will be omitted below, and if the padding parameter is 0 and the step parameter is 1 in the convolution and pooling, the description will also be omitted.

的特征向量；紧接着用3*3池化核，步数为2进行最大值池化，输出

的特征向量。The image is preprocessed before the residual backbone network starts. Specifically: the input images of three different scales are convolutional with 7*7*64 convolution kernels, filled with 3, and the number of steps is 2 for convolution, and output

The feature vector of ; then use 3*3 pooling kernel, the number of steps is 2 for maximum pooling, output

eigenvectors of .

在C3层中，依次包含一个block2、一个block1、一个bock1、一个block1，输出为

在C4层中，依次包含一个block2、一个block1、一个bock1、一个block1、一个block1，输出为

在C5层中，依次包含一个block2、一个block1、一个block1，输出为

在C6层中，依次包含一个block2、一个block1、一个block1，输出为

In the C2 layer, it contains a block2, a block1, a bock1, and a block2 in turn, and the output is

In the C3 layer, it contains a block2, a block1, a bock1, and a block1 in turn, and the output is

In the C4 layer, it contains a block2, a block1, a bock1, a block1, and a block1 in turn, and the output is

In the C5 layer, it contains a block2, a block1, and a block1 in turn, and the output is

In the C6 layer, it contains a block2, a block1, and a block1 in turn, and the output is

(2) Next is the feature pyramid network layer. The present invention performs 1*1 convolution on the output of the five-layer feature vector in the previous step to change the number of channels to 256, and uses the nth layer and the (n+1)th layer to perform phase comparison. The feature vector of the fusion output is output as the Pn feature layer, and for the highest layer P6, it is directly output. In addition, more candidate detection frames can be added here, and the maximum value pooling with a step number of 2 is performed on P6 to obtain the output of the P7 feature vector. In order to save training space, this method is not adopted in this embodiment.

(3) By generating candidate detection frames on the feature maps from P2 to P6, and taking each pixel as the center for each feature map, generate candidate detection frames with three scales and three aspect ratios. According to the feature maps from P2 to P6 The proportional relationship of the size, the length, width, and center coordinates of the candidate detection frame are uniformly scaled to the range of 0 to 1.

(4) Followed by the RPN layer, through a series of convolution operations, the offset and confidence of the candidate detection frame are generated, combined with the candidate detection frame, by arranging the confidence from high to low, and non-polar The large value suppression algorithm filters out 1500 final proposed detection boxes.

(5) Next, align the 1500 recommended detection frames corresponding to the original P2, P3, P4, P5, and P6 feature maps, that is, according to the size of the recommended detection frame, retrieve the feature layer to which it belongs, and then use the corresponding On the feature map, a nonlinear interpolation algorithm is used to intercept the corresponding suggested detection frame. The aligned output of the proposed detection boxes contained in each feature map is 7*7*256.

(6) The output of the proposed detection frame alignment layer is then input into the classifier layer. The classifier layer uses the 7*7 feature vector of the 7*7 convolution input to convert the 7*7 feature vector to 1*1 feature vector but the number of channels is unchanged, and then uses the full connection respectively to convert it into the category score output of 81 classes and each class. The four coordinate position offsets of ; the softmax operation is performed on the 81 class score output vectors, and the output is the class probability.

(7) In the test phase, the category score information and position offset information obtained by the classifier layer are used to input the test layer, wherein the maximum value of the category probability is screened in the test layer, and the target category corresponding to the prediction of the interesting detection frame is selected, Further non-maximum suppression is performed with a threshold of 0.7 to filter out redundant detection frames of interest. Finally, the final predicted detection frame and the corresponding predicted target category are obtained in the test layer.

In the training phase, in the loss function calculation layer, the class score information and position offset information obtained by the classifier layer are used to calculate the loss size of the real information, and then the model parameters and weights are modified through back propagation.

For specific applications:

Step 1: Obtain a picture containing a large number of small target objects, modify it to a standard size, double upsampling and double downsampling, and form an image pyramid with the original image as the input to the improved Mask RCNN model.

Step 2: Input the image pyramid into the residual backbone network of the improved Mask RCNN model, extract features from images of three different scales in the image pyramid, and output three sets of feature vectors.

Step 3: Integrate three sets of feature vectors through the feature pyramid network layer to construct a feature pyramid.

Step 4: Generate candidate detection frames at each layer of the feature pyramid and send them to the RPN layer.

Step 5: At the RPN layer, correspondingly generate confidence and position offset information for each candidate detection frame, and screen out a certain number of effective suggested detection frames according to the confidence level and non-maximum suppression.

Step 6. For the screened suggested detection frames, corresponding to the feature maps that generated them in the feature pyramid, aligned and cut out on the feature maps, and uniformly adjusted to a fixed size, but they are still placed separately according to different feature layers.

Step 7: Finally, input the aligned suggested detection frame into the classifier layer, and obtain each category confidence and position offset information of the suggested detection frame. Determine the category of the suggested detection frame according to the maximum value of the category probability; then consider that the offset of the suggested detection frame is the offset corresponding to the maximum value of the category, and adjust the position of the suggested detection frame; remove the background of each suggested detection frame. Then, according to the confidence score of the largest category of each proposed detection frame, take a threshold of 0.7, and then filter out a certain ROI (Region of interest, region of interest), and do non-maximum suppression.

The above-mentioned embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited by the above-mentioned embodiments, and any other changes, modifications, substitutions, combinations, The simplification should be equivalent replacement manners, which are all included in the protection scope of the present invention.

Claims (10)

1. a small target detection method based on multi-scale image and weighted fusion loss, is characterized in that, is realized based on improving Mask RCNN model, comprises: S1. Build an improved Mask RCNN model; the improved Mask RCNN model includes: a residual backbone network, a feature pyramid network layer, a region generation network layer, a frame of interest alignment layer, a classifier layer, a loss function calculation layer and a test layer ; S2. Constructing an image pyramid: scaling the original image, and combining the original image, the reduced-sized image, and the enlarged-sized image together to form an image pyramid; S3. Randomly crop the images in the image pyramid; S4. Send the randomly cropped image to the residual backbone network for convolution, batch normalization, and pooling operations, and output multiple sets of feature maps of different sizes; S5. Fusion of multiple sets of feature maps of different scales, and further processing to obtain feature maps P2-P6; S6. Generate unscreened candidate detection frames for the feature maps P2-P6 respectively; S7. Generate the network layer from the input area of the feature map P2-P6, and obtain the offset and confidence of the candidate detection frame through a series of convolution operations; S8, combine the offset of the candidate detection frame of S7 with the unscreened candidate detection frame data obtained in S6, and screen out the set amount of candidate detection frames as the detection frame of interest; S9. Corresponding the detection frames of interest back to the feature maps P2-P6 respectively, and perform an alignment operation; S10. Input the result of the alignment operation to the classification layer, and output the predicted interest detection frame category score, category probability, and coordinate offset; S11. Input the predicted interest detection frame category score, category probability, and coordinate offset into the test layer, select the maximum value of the category probability in the test layer, select the predicted target category corresponding to the interest detection frame, and further pass the non- The maximum value suppression filters out redundant interest detection frames, and finally obtains the final predicted interest detection frame and the corresponding predicted target category in the test layer. 2. small target detection method according to claim 1, is characterized in that, in training stage also comprises: S12. Input the category score of the detection frame of interest predicted in S10 into the loss function calculation layer, and use it as the input of the cross-entropy function together with the actual category label to calculate the classification loss value, and obtain the category prediction loss of the feature maps P2-P6; The coordinate offset of the detection frame of interest predicted in S10 and the offset of the real target frame are used as the input of the regression loss function, and the regression prediction loss of the feature maps P2-P6 is obtained; S13, weight the category prediction losses of the feature map P2 and the feature map P3 respectively, and add them to the category prediction losses of the feature map P4, the feature map P5, and the feature map P6 to obtain the total category prediction loss; The regression prediction loss of the feature map P2 and the feature map P3 are weighted respectively, and the total regression prediction loss is obtained by adding the regression prediction loss of the feature map P4, the feature map P5, and the feature map P6; S14. Iteratively update the parameters and weights of the improved Mask RCNN model through backpropagation. Specifically, the total category prediction loss and the total regression prediction loss are respectively used to perform optimization iterations and update the weight value of the improved Mask RCNN model. 3. small target detection method according to claim 1, is characterized in that, the improvement that the Mask RCNN model of improvement makes comprises: 1. The alignment of the detection frames of interest is no longer unified, but the different feature layers are aligned separately. After alignment, the incoming loss function calculation layer is not directly integrated, but is input to the classifier layer for separate classification and regression, and finally separate input The loss function calculation layer weights the loss function calculated by the feature layer for detecting small targets, and fuses it with the loss function for detecting large and medium target layers; 2. Add a layer of effective feature layer P6 to the original Mask RCNN model; 3. Remove the image segmentation module in the original Mask RCNN and cancel the mask branch. 4. The small target detection method according to claim 1, wherein the scaling of the original image in S2 comprises: The formula for scaling an image is: Image_New=Image*scale (1) Among them: Image_New represents the zoomed image, Image represents the image before zooming, and scale represents the zoom scale; The scaling scale is determined by the following factors: After the scaling is completed, the length of the minimum side cannot be less than min_dim, min() means taking the minimum value operation, h means the height of the original image, w means the width of the original image, then when min_dim is greater than min(h, w), scale=min_dim/min(h, w) (2) otherwise scale=1; Set the length of the longest side after scaling is max_dim, if the image is scaled according to formula (2), if the longest side of the scaled image has exceeded max_dim, then make: scale=max_dim/image_max (3) Otherwise, continue to scale by scale=min_dim/min(h, w); The size of the final scaled picture is max_dim*max_dim. In addition, if the scale of the final scaled scale is greater than 1, that is, the original picture is enlarged, it will be enlarged by bilinear interpolation; the final scaled picture is less than max_dim*max_dim part, padded pixel values with zero values. 5. small target detection method according to claim 1, is characterized in that, in S3, the formula expression of randomly cropping picture is as follows: Y ₁ =randi([0,image_size(1)-crop_size(1)]) (4) X ₁ =randi([0,image_size(2)-crop_size(2)]) (5) Among them: Y1 and X1 represent the ordinate and abscissa of the lower left corner of the beginning of the cropped image, respectively; randi represents a random number, and the range of the number is the range in the parentheses; image_size is the size of the image before cropping, and the first dimension stores the image width, the second dimension stores the image length; crop_size is the size of the area to be cropped, the first dimension storage area width, and the second dimension storage area length; Y ₂ =min(image_size(1),Y1+crop_size(1)) (6) X ₂ =min(image_size(2),X1+crop_size(2)) (7) Among them: Y2 and X2 represent the upper right ordinate and upper right abscissa of the beginning of the cropped image, respectively; randi means random number; min() means the minimum value; Use the two coordinates obtained from equations (4) to (7) to determine the specific position of the cropping. If the cropping area overflows the original image, pad filling will be performed to obtain the cropped image. 6. The small target detection method according to claim 1, wherein the convolution of the residual backbone network comprises two types of convolution modules of block1 and block2, wherein: The convolution module block1 workflow includes: ①. For branch 1, the output and input remain the same; ②. For branch 2, use 1*1 convolution kernel, 3*3 convolution kernel, and 1*1 convolution kernel to perform convolution operation in turn, and perform mean normalization on the output feature vector after each convolution is completed. change; The convolution module block2 workflow includes: 1. For branch 1, use a 1*1 convolution kernel to perform the convolution operation, and then perform mean normalization on the output feature vector; ②. For branch 2, use 1*1 convolution kernel, 3*3 convolution kernel, and 1*1 convolution kernel to perform convolution operation in turn, and perform mean normalization on the output feature vector after each convolution is completed. change. 7. small target detection method according to claim 1, is characterized in that, outputting the feature maps of multiple groups of different sizes in S4 comprises: For the original image input, five layers of feature map output are formed: C2, C3, C4, C5, C6; for input that is doubled from the original image, five layers of feature map output are formed: C2s, C3s, C4s, C5s, C6s; for The input that is doubled relative to the original image constitutes five-layer feature map output: C2l, C3l, C4l, C5l, C6l. 8. The small target detection method according to claim 7, wherein step S5 comprises: S51. Perform upsampling based on the interpolation principle on C2s-C6s, and double the C2s-C6s; S52, perform maximum pooling on C2l-C6l, and double the size of C2l-C6l; S53, adding C2-C6, doubled C2s-C6s and doubled C2l-C6l to obtain C2-C6 fused with image features of different scales; S54, further processing C2-C6 fused with image features of different scales to obtain feature maps P2-P6. 9. The small target detection method according to claim 8, wherein S54 comprises: Use 256 1*1 convolution kernels to convolve C6 that incorporates image features of different scales to obtain a feature map P6 with an output of 16*16*256; Use 256 1*1 convolution kernels to convolve C5 that incorporates image features of different scales and add the output of P6 upsampling twice, and then perform 3*3 convolution to obtain an output of 32*32*256 The feature map P5 of; Use 256 1*1 convolution kernels to convolve C4 that combines image features of different scales and add the output of P5 upsampling twice, and then perform 3*3 convolution to obtain an output of 64*64*256 The feature map P4 of; Use 256 1*1 convolution kernels to convolve C3, which combines image features of different scales, and add the output of P4 upsampling twice, and then perform 3*3 convolution to obtain an output of 128*128*256 The feature map P3 of ; Use 256 1*1 convolution kernels to convolve C2, which combines image features of different scales, and add the output of P3 upsampling twice, and then perform 3*3 convolution to obtain an output of 256*256*256 Feature map P2. 10. The small target detection method according to claim 1, wherein the height h and the width w of the candidate detection frame are:

Among them: scale_length indicates that when the candidate detection frame is a box with the same height and width, the height and width correspond to the pixel-level size of the original image.

Images