CN113256623A

CN113256623A - FPC defect detection method based on improved MASK RCNN

Info

Publication number: CN113256623A
Application number: CN202110723382.0A
Authority: CN
Inventors: 邓承志; 吴朝明; 罗林杰; 汪胜前; 徐晨光; 孙小惟
Original assignee: Nanchang Institute of Technology
Current assignee: Nanchang Institute of Technology
Priority date: 2021-06-29
Filing date: 2021-06-29
Publication date: 2021-08-13
Anticipated expiration: 2041-06-29
Also published as: CN113256623B

Abstract

The invention discloses an FPC defect detection method based on an improved MASK RCNN. The steps are as follows: S1. ROI processing and image cropping, collecting target detection images, preprocessing FPC original image data, and cropping into small images suitable for network input; S2. . Perform data enhancement processing on the preprocessed images to expand the data set for model training; S3. Manual annotation; S4. Determine the number of horizontal stacking of feature map blocks N, and determine the horizontal stacking of feature map blocks according to the type and number of FPC defect detection Number of times N; S5. Model training, send the marked image data set to the improved MAKS RCNN network model for training; S6. Perform performance evaluation on the trained FPC defect detection model; S7. Parameter optimization and fine-tuning, combined with the evaluation results of S6 , to further optimize the model. The invention realizes the automatic detection of FPC defects, and the size of the defects can be obtained by dividing the image defects by the mask, which is convenient for staff to re-judgment and greatly reduces the detection cost of enterprises.

Description

FPC defect detection method based on improved MASK RCNN

Technical Field

The invention relates to the technical field of flexible circuit board defect detection and image segmentation, in particular to an FPC defect detection method based on improved MASK RCNN.

Background

The flexible Printed Circuit board is called as fpc (flexible Printed Circuit board) for short, and has the characteristics of thin thickness, light weight, and free bending and folding. Compared with the traditional circuit board, the FPC occupies less space, can greatly reduce the packaging size and weight so as to meet the requirements of high integration and mobility of electronic products, can realize the three-dimensional space wiring of the circuit, enhances the reliability of the products and reduces the assembly cost. Because the FPC material is special, has high integration level and complex process, the manufacturing process is easily affected by factors such as equipment, personnel, environment and the like to generate defects.

The MASK RCNN network is an example segmentation model based on RCNN network improvement, and can accurately segment pixels corresponding to an object while detecting the object. The feature extraction of the MASK RCNN mainly comprises a residual error network and a feature pyramid network, the extracted feature graph is sent to an RPN network to screen candidate frames, and finally target detection and segmentation are finished by a classification branch, a detection branch and a MASK branch; however, compared with the PCB, the FPC has the defect characteristics of more image types, more quantity, random positions, less characteristic information, small defects and the like; the traditional image processing method is difficult to realize accurate detection and segmentation of such FPC defect images, and the MASK RCNN network model is also difficult to accurately detect, classify and segment FPC defects, so that an improved MASK RCNN-based FPC defect detection method is urgently needed to solve the problems.

Disclosure of Invention

The invention aims to overcome the defects of the prior art, and provides an FPC defect detection method based on an improved MASK RCNN, which realizes automatic detection, classification and segmentation of main defects of an FPC, saves the labor cost of enterprises, and greatly improves the defect detection efficiency.

The technical scheme of the invention is as follows:

the method comprises the following steps:

s1, ROI processing and image cutting, wherein a field target detection image is collected, and the data of an FPC original image is preprocessed, wherein the preprocessing comprises ROI processing and image cutting;

s2, performing data enhancement processing on the preprocessed image;

s3, manual marking, namely performing manual marking on the image data after the enhancement processing, and then dividing a data set;

s4, determining the transverse stacking times N of the feature map blocks, wherein the transverse stacking times N of the feature map blocks are determined according to the types and the quantity of FPC defects to be identified;

s5, model training, namely sending the marked image data into an improved MASK RCNN network model for training to obtain an FPC defect detection model;

s6, performing performance evaluation on the trained FPC defect detection model;

and S7, optimizing and fine-tuning parameters, and further optimizing and fine-tuning the FPC defect detection model by combining the evaluation result of S6.

Further, S5 specifically includes the following steps:

s51: setting the pixel area of the RPN network anchor to 8²，16²，32²，64²，256²The aspect ratio of the anchors is set to be {3:4, 1:1, 3:1}, and the step size of the anchors is set to be 4;

s52: extracting network extraction features by using EfficientNet as a backbone feature;

s53: combining the bottom layer characteristic graph and the high layer characteristic graph semantic information to create a characteristic graph block from left to right from bottom to top;

s54: the threshold value for judging defects by the model is set to G, and if the model output score is smaller than the threshold value G, the defect is judged to be not present, and if the score is larger than the threshold value G, the defect is judged to be present.

Furthermore, the specific method of ROI processing is to cut out a portion to be detected, i.e., an ROI (region of interest) image, from the original high-resolution FPC image, and cut out the ROI image into K pieces, where the K pieces of cut-out image have a maximum resolution of 1024 × 1024 and a minimum resolution of 256 × 256, and K selects a value from the natural number set {2, 3, 4} according to the original high-resolution FPC image.

Further, the data enhancement processing method adopts image rotation, image mirroring, image contrast adjustment, image translation and random deletion to increase the training data set.

Furthermore, the manual labeling adopts a Labelme labeling tool, the real position of the defect is labeled on the data set, and the labeled data set is divided into a training set and a verification set according to the proportion of 4 to 1.

Furthermore, the number of times of transverse stacking of the feature map blocks is N times, and N selects a numerical value in the natural number set {3, 4, 5, 6} according to the FPC defect type.

Further, the specific implementation steps of the feature map block are as follows: performing 1 × 1 convolution operation on 5 feature maps C1, C2, C3, C4 and C5 generated by the trunk feature extraction network to generate C1_0, C2_0, C3_0, C4_0 and C5_0, performing upsampling on the last four feature maps C2_0, C3_0, C4_0 and C5_0 to generate C2_1, C3_1, C4_1 and C5_1, performing element-by-element addition and fusion with the previous feature map to generate C1_2, C2_2, C3_2 and C4_2, and performing element-by-element addition on the C2_2, C3_2 and C4_2 generated by the trunk feature extraction network and the C2, C3 and C4 feature maps generated by the trunk feature extraction network, wherein the steps are stacking feature maps N times; finally, performing 3-by-3 convolution on C1_2 from bottom to top to generate a valid feature map P1; after the maximum pooling operation of P1, adding C2_2 element by element, and performing 3-by-3 convolution to generate an effective characteristic map P2; performing 3 × 3 convolution and maximum pooling on the effective feature map P2, adding the effective feature map P2 to C3_2 element by element, and performing 3 × 3 convolution to generate an effective feature map P3; performing 3 × 3 convolution and maximum pooling on the effective feature map P3, adding the effective feature map P3 to C4_2 element by element, and performing 3 × 3 convolution to generate an effective feature map P4; c5_2 is convolved 3 x3 with P4 and maximally pooled, and then added pixel by pixel, and further convolved 3 x3 to generate the active signature P5.

Further, when the number of times N of transverse stacking and the number N of FPC defect types is more than or equal to 1 and less than or equal to 9, taking N as 3; when the number N of the FPC defect types is more than or equal to 10 and less than or equal to 14, N is 4; when the number N of the FPC defect types is more than or equal to 15 and less than or equal to 20, taking N as 5; when the FPC defect type N exceeds 20, taking N as 6;

further, according to the characteristics of FPC defect, the pixel area of RPN network anchor is set to be 8²，16²，32²，64²，256²The aspect ratio of the anchors is set to be {3:4, 1:1, 3:1}, and the step size of the anchors is set to be 4;

further, the performance evaluation of the trained FPC defect detection model is mainly divided into three categories, namely defect positioning, defect classification and defect segmentation; for defect positioning and defect classification, calculating the average precision mAP as an evaluation index; for defect segmentation, calculating the ratio of a defect segmentation pixel to a defect actual pixel as an evaluation index; and if the mAP of the model is more than or equal to 90% and the ratio of correctly dividing the pixels is more than or equal to 60%, determining that the model can accurately detect and divide the defects of the FPC, otherwise, optimizing and finely adjusting the model.

Further, further optimization and fine tuning of the FPC defect detection model are mainly achieved by increasing the number of training samples, adjusting the transverse stacking times N of the feature map blocks, and modifying the learning rate and the iteration times.

Compared with the prior art, the invention has the following remarkable advantages:

(1) the invention enhances the detection capability of the network model for small targets and targets with single characteristics;

(2) the invention realizes the automatic detection, classification and segmentation of FPC defects;

(3) the invention solves the problem that the traditional algorithm can only detect specific defects, and saves labor cost;

(4) the segmented defect image is covered by a mask with random colors, so that field workers can conveniently judge the defect image, and the number of the defect pixels can be obtained through the number of the mask pixels.

Drawings

FIG. 1 is a flowchart illustrating an FPC defect detection method based on modified MASK RCNN according to the present embodiment;

fig. 2 is a block network structure of a feature map of the present example.

Detailed Description

The objects, technical solutions and advantages of the present invention will be further described with reference to the accompanying drawings. It should be understood that the embodiments described herein are not intended to be construed as merely illustrative of the present invention and not limitative of the scope thereof.

As shown in fig. 1, a FPC defect detection method based on improved MASK RCNN includes the following steps:

step 1, ROI processing and image cutting, wherein a target detection image of an industrial field is acquired, and FPC original image data is preprocessed, wherein the preprocessing comprises the ROI processing and the image cutting.

And 2, performing data enhancement processing on the preprocessed image, wherein the data enhancement processing adopts a traditional image processing method: image rotation, image mirroring, image contrast adjustment, image translation and random deletion, and the data set is enlarged through data enhancement processing, so that the robustness of the model is improved.

And 3, manually labeling the FPC image data after the enhancement processing, wherein Labelme software is used for labeling the FPC image data set in the embodiment, and recording the real pixel point coordinates of the defects in the data set. Dividing a training set and a verification set into data sets according to the proportion of 4 to 1, wherein image data of the training set is used for training a model, image data of the verification set is used for verifying the detection capability of the model, and the model continuously adjusts network parameters according to the verification effect through a large amount of training.

Step 4, determining the transverse stacking times N of feature block, please refer to fig. 2, where fig. 2 is a network structure of the feature block of the present example, where a plus sign (+) in the figure indicates element-by-element addition, a dashed box indicates the feature block, and for the types and number of FPC defects to be identified, determining the transverse stacking times N of the feature block, where N is equal to or greater than 1 and equal to or less than 9, and where N is 3; when the number N of the defect types is more than or equal to 10 and less than or equal to 14, taking N as 4; when the number N of the defect types is more than or equal to 15 and less than or equal to 20, taking N as 5; when the number N of defect types exceeds 20, taking N as 6; the specific implementation steps of the feature map block are as follows: performing 1 × 1 convolution operation on 5 feature maps C1, C2, C3, C4 and C5 generated by the trunk feature extraction network to generate C1_0, C2_0, C3_0, C4_0 and C5_0, performing upsampling on the last four feature maps C2_0, C3_0, C4_0 and C5_0 to generate C2_1, C3_1, C4_1 and C5_1, performing element-by-element addition and fusion with the previous feature map to generate C1_2, C2_2, C3_2 and C4_2, and performing element-by-element addition on the C2_2, C3_2 and C4_2 generated by the trunk feature extraction network and the C2, C3 and C4 feature maps generated by the trunk feature extraction network, wherein the steps are stacking feature maps N times; finally, performing 3-by-3 convolution on C1_2 from bottom to top to generate a valid feature map P1; after the maximum pooling operation of P1, adding C2_2 element by element, and performing 3-by-3 convolution to generate an effective characteristic map P2; performing 3 × 3 convolution and maximum pooling on the effective feature map P2, adding the effective feature map P2 to C3_2 element by element, and performing 3 × 3 convolution to generate an effective feature map P3; performing 3 × 3 convolution and maximum pooling on the effective feature map P3, adding the effective feature map P3 to C4_2 element by element, and performing 3 × 3 convolution to generate an effective feature map P4; c5_2 is convolved 3 x3 with P4 and maximally pooled, and then added pixel by pixel, and further convolved 3 x3 to generate the active signature P5.

Step 5, model training, namely setting the pixel area of the RPN network anchor as 8²，16²，32²，64²，256²Setting the width-to-height ratio of the anchors as {3:4, 1:1, 3:1}, setting the step length of the anchors as 4, setting the threshold value of the defects judged by the model as G, judging whether the defects exist if the output score of the model is smaller than the threshold value G, judging whether the defects exist if the output score of the model is larger than the threshold value G, sending the marked FPC image data into an improved MASK RCNN network model for training, and training an FPC defect detection model.

Step 6, performing performance evaluation on the trained FPC defect detection model, wherein the model evaluation is mainly divided into three categories of defect positioning, defect classification and defect segmentation; for defect positioning and defect classification, calculating the average precision mAP as an evaluation index; for defect segmentation, calculating the ratio of a defect segmentation pixel to a defect actual pixel as an evaluation index; and testing the FPC image by using the trained FPC detection model, if the mAP of the model is more than or equal to 90% and the ratio of correctly dividing pixels is more than or equal to 60%, determining that the model can accurately detect and divide the defects of the FPC, and otherwise, optimizing and finely adjusting the model.

And 7, optimizing and fine-tuning parameters, further optimizing and fine-tuning the FPC defect detection model by combining the evaluation result of the step 6, and mainly realizing the optimization and fine-tuning by increasing the number of training samples, adjusting the transverse stacking times N of the feature diagram blocks, modifying the learning rate and the iteration times.

Examples

In the embodiment, the method for detecting the FPC defect based on the improved MASK RCNN comprises the steps of firstly preprocessing an acquired original FPC image, then performing data enhancement on the preprocessed image to enlarge the data volume of the FPC image, then manually marking the enhanced image data, dividing a marked data set into a training set and a verification set according to the proportion of 4: 1, then sending the divided data set into a model for training, finally evaluating the trained model, and judging whether the model meets the FPC detection requirement. The method comprises the following specific steps:

step 1: and ROI processing and image cutting, namely acquiring a target detection image of an industrial field, and preprocessing the data of the FPC original image, including ROI processing and image cutting.

Step 2: and performing data enhancement processing on the preprocessed image, wherein the data enhancement processing adopts a traditional image processing method: image rotation, image mirroring, image contrast adjustment, image translation and random deletion, and the data set is expanded to 3 times of the original data set through data enhancement processing.

And step 3: and manual labeling, namely performing manual labeling on the FPC image data after the enhancement processing, and labeling the FPC image data set by using Labelme software in the embodiment and recording the coordinates of the real pixel points of the defects in the data set. Dividing a training set and a verification set into data sets according to the proportion of 4 to 1, wherein image data of the training set is used for training a model, image data of the verification set is used for verifying the detection capability of the model, and the model continuously adjusts network parameters according to the verification effect through a large amount of training.

And 4, step 4: determining the transverse stacking times N of the feature map block, and determining the transverse stacking times N of the feature map block aiming at the types and the number of FPC (flexible printed circuit) defects to be identified, wherein the number N of the types of the FPC data defects collected in the embodiment is 3, the value of N is within the range that N is not less than 1 and not more than 9, and the transverse stacking times N are 3;

and 5: model training, the pixel area of the RPN network anchor is set as 8²，16²，32²，64²，256²The aspect ratio of the anchors is set to be {3:4, 1:1, 3:1}, and the step size of the anchors is set to be 4; and setting a threshold value of the model for judging the defects to be 0.7, judging that the defects do not exist if the output score of the model is less than the threshold value of 0.7, judging that the defects exist if the output score of the model is greater than the threshold value of 0.7, and sending the marked FPC image data into an improved MASK RCNN network model for training to obtain an FPC defect detection model.

Step 6: performing performance evaluation on the trained FPC defect detection model, wherein the model evaluation mainly comprises three types of defect positioning, defect classification and defect segmentation; for defect positioning and defect classification, calculating the average precision mAP as an evaluation index; for defect segmentation, calculating the ratio of a defect segmentation pixel to a defect actual pixel as an evaluation index; and testing the FPC image by using the trained FPC detection model, if the mAP of the model is more than or equal to 90% and the ratio of correctly dividing pixels is more than or equal to 60%, determining that the model can accurately detect and divide the defects of the FPC, and otherwise, optimizing and finely adjusting the model.

And 7: and (4) optimizing and fine-tuning parameters, further optimizing and fine-tuning the FPC defect detection model by combining the evaluation result of the step (6), and mainly realizing the optimization and fine-tuning by increasing the number of training samples, adjusting the transverse stacking times N of the feature diagram block, modifying the learning rate and the iteration times.

This example uses a pytorech deep learning framework with experimental computer hardware configured to: AMD Ryzen 93900X 12 core 24 thread processor, GeForce RTX3090 graphics card, 32G operation memory, 500G NVME solid state disk; the experimental software environment is 64-bit Windows 10 professional edition, Pycharm integrated development environment, CUDA version is 11.1, CUDNN version is 8.0.4.30, and the final result is shown in Table 1.

TABLE 1 results of the experiments

As can be seen from table 1, after training, the average accuracy maps of the defect detection of the 2 types of FPC images ((the first type of FPC image (FPC 1) and the second type of FPC image (FPC 2)) are all over 90%, the correct segmented pixel ratio of the FPC1 is 72.3%, the FPC2 data set is small, the correct segmented pixel ratio is 63.6%, and the average detection time of a single image is over 23 milliseconds.

Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that changes, modifications, substitutions and alterations can be made in these embodiments without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.

Claims

1. An FPC defect detection method based on improved MASK RCNN is characterized by comprising the following steps:

s2, performing data enhancement processing on the preprocessed image;

2. The method for FPC defect detection based on modified MASK RCNN according to claim 1, wherein: the specific method of ROI processing is that a part needing to be detected, namely an ROI image, is cut out of an original high-resolution FPC image, then the ROI image is cut out into K parts, the maximum resolution of the K parts of cut images is 1024 x 1024, the minimum resolution is 256 x 256, and K selects a numerical value in a natural number set {2, 3, 4} according to the original high-resolution FPC image.

3. The method for FPC defect detection based on modified MASK RCNN according to claim 1, wherein: the data enhancement processing method adopts image rotation, image mirroring, image contrast adjustment, image translation and random deletion to increase a training data set.

4. The method for FPC defect detection based on modified MASK RCNN according to claim 1, wherein: and the manual marking adopts a Labelme marking tool, the real position of the defect is marked on the data set, and the marked data set is divided into a training set and a verification set according to the proportion of 4 to 1.

5. The method for FPC defect detection based on modified MASK RCNN according to claim 1, wherein: the number of times of transverse stacking of the feature graphs is N times, N selects a numerical value in a natural number set {3, 4, 5, 6} according to the FPC defect types, and when the number N of the FPC defect types is more than or equal to 1 and less than or equal to 9, N is 3; when the number N of the defect types is more than or equal to 10 and less than or equal to 14, taking N as 4; when the number N of the defect types is more than or equal to 15 and less than or equal to 20, taking N as 5; when the number N of defect types exceeds 20, N is 6.

6. The method according to claim 1, wherein the step S5 specifically comprises the following steps:

7. The method according to claim 6, wherein the method comprises the following steps: the specific method of S53 is as follows: performing 1 × 1 convolution operation on 5 feature maps C1, C2, C3, C4 and C5 generated by the trunk feature extraction network to generate C1_0, C2_0, C3_0, C4_0 and C5_0, performing upsampling on the last four feature maps C2_0, C3_0, C4_0 and C5_0 to generate C2_1, C3_1, C4_1 and C5_1, performing element-by-element addition and fusion with the previous feature map to generate C1_2, C2_2, C3_2 and C4_2, and performing element-by-element addition on the C2_2, C3_2 and C4_2 generated by the trunk feature extraction network and the C2, C3 and C4 feature maps generated by the trunk feature extraction network, wherein the steps are stacking feature maps N times; finally, performing 3-by-3 convolution on C1_2 from bottom to top to generate a valid feature map P1; after the maximum pooling operation of P1, adding C2_2 element by element, and performing 3-by-3 convolution to generate an effective characteristic map P2; performing 3 × 3 convolution and maximum pooling on the effective feature map P2, adding the effective feature map P2 to C3_2 element by element, and performing 3 × 3 convolution to generate an effective feature map P3; performing 3 × 3 convolution and maximum pooling on the effective feature map P3, adding the effective feature map P3 to C4_2 element by element, and performing 3 × 3 convolution to generate an effective feature map P4; c5_2 is convolved 3 x3 with P4 and maximally pooled, and then added pixel by pixel, and further convolved 3 x3 to generate the active signature P5.

8. The method for FPC defect detection based on modified MASK RCNN according to claim 1, wherein: the performance evaluation of the trained FPC defect detection model is mainly divided into three categories of defect positioning, defect classification and defect segmentation; for defect positioning and defect classification, calculating the average precision mAP as an evaluation index; for defect segmentation, calculating the ratio of a defect segmentation pixel to a defect actual pixel as an evaluation index; and if the mAP of the model is more than or equal to 90% and the ratio of correctly dividing the pixels is more than or equal to 60%, determining that the model can accurately detect and divide the defects of the FPC, otherwise, optimizing and finely adjusting the model.

9. The method for FPC defect detection based on modified MASK RCNN according to claim 1, wherein: the further optimization and fine tuning of the FPC defect detection model are mainly realized by increasing the number of training samples, adjusting the transverse stacking times N of block of the feature diagram, and modifying the learning rate and the iteration times.