CN115147643A

CN115147643A - FPC defect classification method based on CNN and TRANSFORMER

Info

Publication number: CN115147643A
Application number: CN202210643976.5A
Authority: CN
Inventors: 邓承志; 罗林杰; 吴朝明; 汪胜前; 孙小惟; 徐晨光
Original assignee: Nanchang Institute of Technology
Current assignee: Jiangxi University Of Water Resources And Electric Power
Priority date: 2022-06-09
Filing date: 2022-06-09
Publication date: 2022-10-04
Anticipated expiration: 2042-06-09
Also published as: CN115147643B

Abstract

The invention discloses an FPC defect classification method based on CNN and TRANSFORMER. The method uses CTNet designed based on CNN and TRANSFORMER as a classification model. The steps are as follows: S1. Collect target defect images, and preprocess FPC original image data; S2. . Perform data enhancement processing on the preprocessed images to expand the image data set; S3. Manually classify the enhanced image data set; S4. Divide the manually classified image data into a training set and a validation set in proportion; S5 .Model training, send the divided image training set to the CTNet network model for training; S6. Evaluate the performance of the trained FPC classification and detection model; S7. Parameter optimization and fine-tuning, combined with the evaluation results of S6, to further model the model optimization. The invention realizes the automatic classification of FPC defects, has good generality, fast classification speed and low requirements on computer performance, and can greatly reduce the production cost of enterprises.

Description

FPC defect classification method based on CNN and TRANSFORMER

Technical Field

The invention relates to the field of defect classification of flexible circuit boards, in particular to a FPC defect classification method based on CNN and TRANSFORMER.

Background

The Flexible Printed Circuit Board is called as FPC (Flexible Printed Circuit Board) for short, and has the characteristics of thin thickness, light weight, and free bending and folding. Compared with the traditional circuit board, the FPC occupies less space, can greatly reduce the packaging size and weight so as to meet the requirements of high integration and mobility of electronic products, can realize the three-dimensional space wiring of the circuit, enhances the reliability of the products and reduces the assembly cost. Because the FPC material is special, has high integration level and complex process, the manufacturing process is easily affected by factors such as equipment, personnel, environment and the like to generate defects.

The convolutional neural network is a common deep learning network architecture, is inspired by biological visual cognition, has the characteristics of local perception and parameter sharing, has high accuracy for image recognition, and has the advantages that the TRANSFORMER structure appears in the natural language processing field at the earliest time and becomes a mainstream model in the natural language processing field in less than 4 years of birth. Because only information near a convolution kernel can be paid attention to in each convolution in the convolution neural network, information far away can not be fused, the classification accuracy of the convolution neural network is not high for the characteristics of few, large quantity and various FPC defect information, and the classification effect of a traditional image processing method on FPC defect images is poor.

Disclosure of Invention

The invention aims to overcome the defects of the prior art, and provides the FPC defect classification method based on CNN and TRANSFORMER, which realizes the automatic classification of the main defects of the FPC, saves the labor cost of enterprises, has high model classification speed and low requirement on computer performance, can greatly reduce the production cost of the enterprises, and improves the defect detection efficiency.

In order to achieve the purpose, the invention adopts the following technical scheme: a FPC defect classification method based on CNN and TRANSFORMER comprises the following steps:

s1, acquiring a defect image of a field flexible printed circuit board, and preprocessing FPC original image data;

s2, performing data enhancement processing on the preprocessed image;

s3, classifying the image data after the enhancement processing;

s4, dividing the classified image data set into a training set and a verification set according to a proportion;

s5, model training, namely sending the divided image training sets into a constructed CTNet network model for training to train an FPC defect classification model;

s6, performing performance evaluation on the trained FPC defect classification model;

and S7, optimizing and fine-tuning the parameters, and further optimizing and fine-tuning the FPC defect classification model by combining the evaluation result of the S6.

In a further aspect, the method comprises the following steps, the step S1 of preprocessing the image data specifically includes:

and (3) image cutting is carried out on the image data, the image cutting is used for cutting out parts needing to be classified, namely ROI images, from the original high-resolution FPC images, the ROI images are cut into K parts, the resolution of the K parts of cut images is 512-1024, and K is a numerical value selected from the natural number set {2,4,8, 16, 24} according to the original high-resolution FPC images.

The further scheme is that the data enhancement processing method in the step S2 adopts mirror image, brightness adjustment, translation and random deletion to increase the training data set.

Further, the step S3 of classifying the image data specifically includes:

according to the FPC defect types in actual production, images with the same type of defects are divided into the same folder, and the folder is named as the defect types.

Further, the method for dividing the data set specifically includes:

when the data set N after the data enhancement processing is more than or equal to 20000, the division ratio of the training set to the verification set is 3 to 1, when the data set 20000 more than N after the data enhancement processing is more than or equal to 10000, the ratio of the training set to the verification set is 4 to 1, and when the data set 10000 more than N after the data enhancement processing is 5 to 1.

Further, the method for constructing the CTNet network model specifically comprises the following steps:

the input image is reduced in dimension by first scaling the input image to 448 x3 dimensions, then going through 1 3 x3 convolutional layers and one max pooling layer to obtain a feature map with dimension 224 x 3.

The further scheme is that firstly, the feature graph after dimensionality reduction is subjected to 3-by-3 convolution layer once, and the channel dimensionality is improved to 128; then inputting the data into a convolution block, wherein the convolution block consists of 4 branches, the first branch is a 1 × 1 convolution layer, a 3 × 3 convolution layer and a 1 × 1 convolution layer, the second branch is 2 × 3 convolution layers, the third branch is a 1 × 1 convolution layer, a 3 × 3 convolution layer and a 1 × 1 convolution layer, and the fourth branch is a maximum pooling layer; and (3) respectively passing the feature graph with the enhanced dimensionality through the first 3 branches, performing feature fusion on the feature graphs generated by the first 3 branches, performing Concat fusion on the feature graphs generated by the fourth branch, entering the maximum pooling layer after the fusion, generating a feature graph with a dimensionality of 112 × 128, continuously passing the obtained feature graph through three CNN blocks with the same structure, and finally generating a feature graph with a dimensionality of 14 × 1024.

Further, the step of generating the feature map with the dimension of 14 × 1024 further includes:

firstly, performing a Flatten operation on a feature map generated after passing through the four CNN blocks, converting a 3-dimensional feature map into a 2-dimensional feature vector, and then sending the 2-dimensional feature vector into a TRANSFORMER block; wherein the TRANSFORMER block comprises a self-attention layer, two full-link layers and an activation function layer; the 2-dimensional feature vector block firstly passes through the self-attention layer, then passes through a full connection layer, an activation function layer and a full connection layer, the output result is subjected to feature fusion with the 2-dimensional feature vector which does not pass through the self-attention layer, and the operation of four TRANSFORMER blocks is repeated to finally generate a 2-dimensional feature vector with the dimension of 196 x 1024; repeatedly superposing the 2-dimensional feature vector with the dimension of 196 x 1024 by two TRANSFORMER blocks with the same structure for N times, wherein the superposition times N are calculated by a formula (1), N is the number of defects needing to be detected by the FPC, and the dimension of the finally generated 2-dimensional feature vector is 196 x 1536;

(1)。

the further scheme is that the step S6 of performing performance evaluation on the trained FPC defect classification model specifically includes:

using the average accuracy ACC of the verification set as an index; and if the average accuracy ACC of the model is more than or equal to 95%, determining that the model can accurately classify the defects of the FPC, otherwise, optimizing and fine-tuning the model.

The further scheme is that the step S7 of optimizing and fine-tuning the FPC defect classification model is realized by increasing training Batch, increasing the number of training samples, modifying learning rate and iteration times.

Compared with the prior art, the invention has the beneficial effects that:

(1) The method solves the problem that the classification precision of the traditional algorithm and the deep learning model for the defects of the FPC is low, and saves the labor cost;

(2) The invention realizes the automatic classification of FPC defects;

(3) The method solves the problem of low reasoning speed of the deep learning model, and can meet the requirement of online classification in industrial application.

Drawings

Fig. 1 is a flow chart of an FPC defect classification method based on CNN and transport in an embodiment of the present invention;

FIG. 2 is a diagram of a convolution block according to an embodiment of the present invention;

FIG. 3 is a diagram of a TRANSFORMER block in accordance with an embodiment of the present invention;

fig. 4 is a network structure diagram according to an embodiment of the present invention.

Detailed Description

The objects, technical solutions and advantages of the present invention will be further described with reference to the accompanying drawings. It should be understood that the detailed description and specific examples, while indicating the scope of the invention, are intended for purposes of illustration only and are not intended to limit the scope of the invention.

As shown in fig. 1, a method for classifying FPC defects based on CNN and transportormer includes the following steps:

s1, acquiring a defect image of a field flexible printed circuit board, and cutting an image of FPC (flexible printed circuit) original image data, wherein the specific method of image cutting is to cut out parts to be classified, namely ROI (region of interest) images, of the original high-resolution FPC image, then cut out the ROI images into K parts, the maximum resolution of the K parts of cut images is 1024 x 1024, the minimum resolution is 512 x, and K selects a numerical value in a natural number set {2,4,8, 16, 24} according to the original high-resolution FPC image.

S2, performing data enhancement processing on the cut image, wherein: the data enhancement processing method adopts mirror image, brightness adjustment, translation and random deletion to enlarge the data set by 5 times.

S3, carrying out manual classification processing on the image data after the enhancement processing, wherein: the classification method is to divide the images with the same type of defects into the same folder according to the types of the FPC defects in actual production, and the folder is named as the defect type.

And S4, dividing the classified image data set into a training set and a verification set according to the proportion, wherein the proportion of the training set to the verification set is 3 to 1 when the data set N after data enhancement processing is more than or equal to 20000, the proportion of the training set to the verification set is 4 to 1 when the data set 20000 after data enhancement processing is more than or equal to 10000, and the proportion of the training set to the verification set is 5 to 1 when the data set 10000 after data enhancement processing is more than N.

S5, model training, namely feeding the divided image training set into a CTNet network model for training to train an FPC defect classification model; please refer to fig. 2, fig. 3 and fig. 4 to understand the structure of the CTNet network, and the plus sign (+) in fig. 2 and fig. 3 both indicate the feature fusion by element addition. The CTNet model is specifically constructed as follows:

the input image to the CTNet model is first scaled (reshape) to 448 x3 and passed through a 3 x3 convolutional layer and a max pooling layer to generate a feature map C0, with C0 having a dimension of 224 x 3.

Before the feature map C0 enters the first CNN block, 3 × 3 convolution layers are needed to increase the channel dimension to 128; the convolution block is composed of 4 branches, the first branch is a 1 × 1 convolution layer, a 3 × 3 convolution layer and a 1 × 1 convolution layer, the second branch is two 3 × 3 convolution layers, the third branch is a 1 × 1 convolution layer, a 3 × 3 convolution layer and a 1 × 1 convolution layer, and the fourth branch is a maximum pooling layer; after the dimension of the feature graph C0 is increased, the feature graphs generated by the first 3 branches respectively pass through, feature fusion is carried out on the feature graphs generated by the first 3 branches, then Concat fusion is carried out on the feature graphs generated by the fourth branch, the feature graphs enter the maximum pooling layer after fusion to generate a feature graph C1, the dimension of the C1 is 112 x 128, the feature graph C1 generates a feature graph C2 after passing through CNN blocks with the same structure, the feature graph C2 generates a feature graph C3 after passing through CNN blocks with the same structure, the feature graph C3 generates a feature graph C4 after passing through CNN blocks with the same structure, the dimension of the C2 is 56 x 256, the dimension of the C3 is 28 x 512, and the dimension of the C4 is 14 x 1024.

Before entering the TRANSFORMER block, firstly, performing a Flatten operation on the feature map C4, converting the 3-dimensional feature map into a 2-dimensional feature vector, and then sending the 2-dimensional feature vector into the TRANSFORMER block; the TRANSFORMER block mainly comprises a self-attention layer, two full-connection layers and an activation function layer; the 2-dimensional feature vector block firstly passes through the self-attention layer, then passes through a full connection layer, an activation function layer and a full connection layer, the output result is subjected to feature fusion with the 2-dimensional feature vector which does not pass through the self-attention layer, 4 TRANSFORMER block operations are repeated, and finally a 2-dimensional feature vector C5 is generated, wherein the dimension of the C5 is 196 x 1024; the 2-dimensional feature vector C5 passes through 2 TRANSFORMER blocks with the same structure, the two TRANSFORMER blocks with the same structure can be repeatedly overlapped for N times, the number of overlapping times N is calculated by a formula (1), N is the number of defects needing to be detected by the FPC, and finally the dimension of the generated 2-dimensional feature vector C6 is 196 x 1536;

(1)

step 6, performing performance evaluation on the trained FPC defect classification model, wherein the average accuracy ACC of the verification set is mainly used as an index for performing the performance evaluation on the trained FPC defect classification model; and if the average accuracy ACC of the model is more than or equal to 95%, determining that the model can accurately classify the defects of the FPC, otherwise, optimizing and fine-tuning the model.

And 7, optimizing and fine-tuning parameters, and further optimizing and fine-tuning the FPC defect classification model by combining the evaluation result of the step 6, wherein the optimization and fine-tuning are mainly realized by increasing training Batch, increasing the number of training samples, modifying the learning rate and iterating the times.

Examples

In the FPC defect classification method based on CNN and TRANSFORMER in this embodiment, an image clipping operation is performed on an acquired original FPC image, data enhancement is performed on a preprocessed image, the data volume of the FPC image is enlarged, then, the enhanced image data is manually classified, a training set and a verification set are proportionally divided into a labeled data set, then, the divided data set is sent to model training, and finally, the trained model is evaluated to determine whether the classification precision of the model meets the requirement. The method comprises the following specific steps:

step 1: and acquiring a target defect image on site, and cutting the image of the FPC original image data.

Step 2: and performing data enhancement processing on the preprocessed image, wherein the data enhancement processing method adopts mirror image, brightness adjustment, translation and random deletion to enlarge the data set by 5 times.

And step 3: and manually classifying, namely classifying the image data after the enhancement processing, manually classifying the FPC defect types in actual production in the embodiment, and dividing the images with the same type of defects into the same folder, wherein the folder is named as the defect type.

And 4, step 4: the image data set classified manually is divided into a training set and a verification set according to a proportion, the data set dividing method is that when the data set N after data enhancement processing is larger than or equal to 20000, the dividing proportion of the training set to the verification set is 3 to 1, when the data set 20000 after data enhancement processing is larger than or equal to 10000, the proportion of the training set to the verification set is 4 to 1, and when the data set 10000 after data enhancement processing is larger than N, the proportion of the training set to the verification set is 5 to 1.

And 5: and (3) model training, namely feeding the divided image training set into the built CTNet network model for training, and training an FPC defect classification model.

And 6: performing performance evaluation on the trained FPC defect classification model, wherein the model evaluation is mainly performed on the trained FPC defect classification model by using the average accuracy ACC of a verification set as an index; and if the average accuracy ACC of the model is more than or equal to 95%, determining that the model can accurately classify the defects of the FPC, otherwise, optimizing and fine-tuning the model.

And 7: and (6) optimizing and fine-tuning parameters, further optimizing and fine-tuning the FPC defect classification model by combining the evaluation result of the step 6, and mainly realizing the optimization and fine-tuning by increasing training Batch, increasing the number of training samples, modifying learning rate and iteration times.

This example uses the deep learning framework of the pytorch version 1.9, with the experimental computer hardware configured to: AMD Ryzen9 3900X 12 core 24 thread processor, geForce RTX3090 graphics card, 32G running memory and 500G NVME solid state disk; the experimental software environment is 64-bit Windows 10 professional edition, the Pycharm integrated development environment, the CUDA version is 11.1, the CUDNN version is 8.0.4.30, the model is trained by using a GeForce RTX3090 display card under the environment, and is predicted by using a Ryzen9 3900X 12 core 24-thread processor, and the final result is shown in Table 1.

TABLE 1 FPC data information

Native resolution

Cropping resolution

Defective image

Data enhancement

Training set

Verification set

FPC1

5120*5120

1024*1024

1820 sheet of

9100 sheet

7584 sheets of paper

1516 pieces

FPC2

6576*4384

512*512

2214 sheets

11070 pieces of

8856 pieces of

2214 pieces

TABLE 2 Experimental results

	Glue overflow	Copper exposure	Foreign matter	Gold (Au)	Crush injury	Open circuit	Short circuit	Average accuracy ACC	Average detection time
										FPC1	96.7%	98.6%	99.1%	97.5%	95.6%	99%	96.3%	97.5%	21ms
FPC2	97.4%	98.1%	98.8%	97.9%	96.6%	98.7%	97.1%	97.8%	15ms

As can be seen from table 2, the average accuracy ACC of the classification method provided by the present invention for classifying the defects of the 2 types of FPC images ((the first type FPC image (FPC 1) and the second type FPC image (FPC 2)) is above 97% after training, the average detection time of the defect image of FPC1 is predicted to be 21 milliseconds by using the CPU, and the average detection time of the defect image of FPC2 is predicted to be 15 milliseconds by using the CPU.

Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that changes, modifications, substitutions and alterations can be made in these embodiments without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.

Claims

1. A FPC defect classification method based on CNN and TRANSFORMER is characterized by comprising the following steps:

s1, acquiring a defect image of a field flexible printed circuit board, and preprocessing original image data of the FPC;

s2, performing data enhancement processing on the preprocessed image;

s3, classifying the image data after the enhancement processing;

and S7, optimizing and fine-tuning parameters, and further optimizing and fine-tuning the FPC defect classification model by combining the evaluation result of the S6.

2. The method for classifying FPC defects based on CNN and TRANSFORMER according to claim 1, wherein the step S1 of preprocessing the image data specifically comprises:

and (3) image cutting is carried out on the image data, wherein the image cutting is used for cutting out parts needing to be classified, namely ROI images, from the original high-resolution FPC images, the ROI images are cut into K parts, the resolution of the K parts of cut images is 512 x 512 to 1024 x 1024, and K is a numerical value selected from natural number sets {2,4,8, 16 and 24} according to the original high-resolution FPC images.

3. The FPC defect classification method based on CNN and TRANSFORMER according to claim 1, wherein: the data enhancement processing method in the step S2 adopts mirror image, brightness adjustment, translation and random deletion to enlarge the training data set.

4. The method for classifying FPC defects based on CNN and TRANSFORMER according to claim 1, wherein the step S3 of classifying image data specifically comprises:

5. The FPC defect classification method based on CNN and TRANSFORMER according to claim 1, wherein said data set partitioning method specifically comprises:

6. The FPC defect classification method based on CNN and TRANSFORMER according to claim 1, wherein the constructing of the CTNet network model specifically comprises:

the input image is subjected to dimensionality reduction, the input image is first scaled to 448 x3 dimensions, and then passed through a 3 x3 convolutional layer and a max pooling layer to obtain a feature map with a dimension of 224 x 3.

7. The FPC defect classification method based on CNN and TRANSFORMER according to claim 6, wherein the step of performing dimension reduction processing on the input image further comprises:

firstly, performing 3 × 3 convolution on the feature map subjected to dimensionality reduction once, and promoting the channel dimensionality to 128; then inputting the data into a convolution block, wherein the convolution block consists of four branches, the first branch is a 1 × 1 convolution layer, a 3 × 3 convolution layer and a 1 × 1 convolution layer, the second branch is 2 3 × 3 convolution layers, the third branch is a 1 × 1 convolution layer, a 3 × 3 convolution layer and a 1 × 1 convolution layer, and the fourth branch is a maximum pooling layer; and the feature graph with the enhanced dimensionality passes through the first three branches respectively, feature fusion is carried out on the feature graphs generated by the first three branches, then Concat fusion is carried out on the feature graphs generated by the fourth branch, the feature graphs enter the maximum pooling layer after fusion, the feature graph with the dimensionality of 112 x 128 is generated, the obtained feature graphs continuously pass through three CNN blocks with the same structure, and finally the feature graph with the dimensionality of 14 x 1024 is generated.

8. The method of claim 7, wherein the step of generating the feature map with the dimension of 14 × 1024 is further followed by the step of:

firstly, performing a Flatten operation on a feature map generated after passing through the four CNN blocks, converting a 3-dimensional feature map into a 2-dimensional feature vector, and then sending the 2-dimensional feature vector into a TRANSFORMER block; wherein the TRANSFORMER block comprises a self-attention layer, two full-link layers and an activation function layer; the 2-dimensional feature vector block firstly passes through the self-attention layer, then passes through a full connection layer, an activation function layer and a full connection layer, the output result is subjected to feature fusion with the 2-dimensional feature vector which does not pass through the self-attention layer, and the operation of four TRANSFORMER blocks is repeated to finally generate a 2-dimensional feature vector with the dimension of 196 x 1024; the 2-dimensional feature vector with the dimension of 196 × 1024 passes through two TRANSFORMER blocks with the same structure, the two TRANSFORMER blocks with the same structure are repeatedly overlapped for N times, the number of overlapping times N is calculated by a formula (1), N is the number of defects needing to be detected by the FPC, and the dimension of the finally generated 2-dimensional feature vector is 196 × 1536;

(1)。

9. the FPC defect classification method based on CNN and TRANSFORMER according to claim 1, characterized in that: the step S6 of performing performance evaluation on the trained FPC defect classification model specifically includes:

using the average accuracy ACC of the verification set as an indicator; and if the average accuracy ACC of the model is more than or equal to 95%, determining that the model can accurately classify the defects of the FPC, otherwise, optimizing and fine-tuning the model.

10. The FPC defect classification method based on CNN and TRANSFORMER according to any one of claims 1 to 9, which is characterized in that: and the step S7 of further optimizing and fine-tuning the FPC defect classification model is realized by increasing training Batch, increasing the number of training samples, modifying the learning rate and the iteration number.