CN116824135A

CN116824135A - Atmospheric natural environment test industrial product identification and segmentation method based on machine vision

Info

Publication number: CN116824135A
Application number: CN202310585091.9A
Authority: CN
Inventors: 苏晓杰; 陈康; 吴�灿; 孙少欣; 敖文刚; 马铁东
Original assignee: Chongqing University; Chongqing Technology and Business University
Current assignee: Chongqing University; Chongqing Technology and Business University
Priority date: 2023-05-23
Filing date: 2023-05-23
Publication date: 2023-09-29

Abstract

The application provides an atmospheric natural environment test industrial product identification and segmentation method based on machine vision. The method comprises the following steps: preprocessing an initial image set to obtain a preprocessed data set, wherein the preprocessed data set comprises a training set, a verification set and a test set; clustering the annotation frames in the images of the training set to obtain candidate frames; training and testing a preset neural network model based on deep learning based on the candidate frame, the training set, the verification set and the test set to obtain a target detection model; inputting the image to be detected into a target detection model to obtain a detection result; when the detection result shows that the detection frame of the industrial product exists in the image to be detected, cutting is performed in the image to be detected based on the detection frame, and a region image is obtained; and dividing the regional image based on the division model SAM to obtain a result image corresponding to the single industrial product. Therefore, pixel-level labeling and training are not needed, and the efficiency of dividing single industrial products is improved.

Description

Atmospheric natural environment test industrial product identification and segmentation method based on machine vision

Technical Field

The application relates to the technical field of machine vision, in particular to an atmospheric natural environment test industrial product identification and segmentation method based on machine vision.

Background

The method mainly relies on a manual periodic sampling mode for data acquisition of an atmospheric natural environment industrial product test, and has the problems that the acquisition efficiency is low, the noise of acquired data is large, real-time sampling cannot be performed, manual operation cannot be performed in a severe environment and the like. Therefore, there are data collection methods that introduce robots, artificial intelligence, etc. into the test of atmospheric natural environment industrial products. In a scene of target detection by using a machine vision technology, an example product can only be circled by a horizontal rectangular frame, an example can not be separated from a background, and a great amount of noise exists in acquired image data. At present, semantic segmentation and instance segmentation algorithms can realize separation of an instance from a background, but the effect of segmentation for a single instance is poor or the efficiency is low. For example, semantic segmentation cannot distinguish between different instance products belonging to the same category, and thus cannot sample image data of different instances of the same category separately. The example segmentation can classify pixel levels and can distinguish different examples on the basis of specific categories, but the effect of the example segmentation is greatly dependent on the quality of a training set, the required labeling of the training set is pixel-level, a great deal of labeling time and labor force are required, and the training difficulty of the example segmentation is larger and the time is longer than that of the computer resource required by target detection.

Disclosure of Invention

In view of the above, an object of an embodiment of the present application is to provide a method for identifying and dividing an industrial product in an atmospheric natural environment test based on machine vision, which can solve the problems of poor effect and low efficiency in identifying and dividing an image of a single industrial product from a whole image.

In order to achieve the technical purpose, the application adopts the following technical scheme:

the embodiment of the application provides a machine vision-based atmospheric natural environment test industrial product identification and segmentation method, which comprises the following steps:

acquiring an initial image set, wherein the initial image set comprises images obtained by shooting industrial products under an atmospheric natural environment test;

preprocessing the initial image set to obtain a preprocessed data set, wherein the preprocessed data set comprises a training set, a verification set and a test set;

clustering the annotation frames in the images of the training set to obtain candidate frames, wherein the candidate frames are the annotation frames closest to each clustering center point, and the clustering center points are obtained by clustering the annotation frames;

training and testing a preset neural network model based on deep learning based on the candidate frame, the training set, the verification set and the test set to obtain a target detection model;

inputting the image to be detected into the target detection model to obtain a detection result;

when the detection result indicates that a detection frame of the industrial product exists in the image to be detected, cutting is performed in the image to be detected based on the detection frame, and a cut area image is obtained;

and dividing the regional image based on a preset division model SAM to obtain a result image corresponding to the single industrial product.

In some alternative embodiments, preprocessing the initial image set to obtain a preprocessed data set includes:

labeling each image in the initial image set through a labeling tool labelImg to obtain labeling data corresponding to each image, and forming a first data set, wherein when any one image in the initial image set has an industrial product, the labeling data corresponding to any one image comprises the position and the category of the industrial product;

cleaning and screening the first data set based on a preset cleaning strategy to obtain a second data set;

performing data enhancement operation on the second data set to obtain a third data set, wherein the data enhancement operation comprises at least one operation of randomly overturning, randomly rotating and randomly adjusting the brightness and contrast of the image in the second data set;

dividing the third data set according to a preset proportion to obtain a training set, a verification set and a test set, wherein the training set, the verification set and the test set are used as the preprocessed data set.

In some optional embodiments, performing cleaning and screening on the first data set based on a preset cleaning policy to obtain a second data set, including:

and filtering out the marked data representing the marked abnormality and the image corresponding to the marked data, and filtering out the marked data representing the image quality abnormality and the marked data corresponding to the abnormal image in the first data set to obtain the second data set.

In some optional embodiments, clustering the labeling frames in the images of the training set to obtain candidate frames includes:

extracting coordinates of a labeling frame corresponding to the industrial product in the images of the training set;

normalizing all the coordinates to obtain normalized coordinates;

clustering all the normalized coordinates by adopting a K-means algorithm to obtain K clustering center points, wherein K is an integer greater than 1;

and selecting the labeling frame closest to each clustering center point as the candidate frame aiming at the labeling frame corresponding to the industrial product in the training set.

In some optional embodiments, training and testing a preset neural network model based on the candidate frame, the training set, the verification set and the test set to obtain a target detection model includes:

training the neural network model based on the candidate frame and the training set by adopting a random gradient descent algorithm, wherein the neural network model comprises any one of a YOLO model and a Fast R-CNN model;

based on the verification set, evaluating the trained neural network model to obtain an evaluation index, wherein the evaluation index comprises at least one of precision, recall rate and F1 score;

when the evaluation index does not meet the preset condition, adjusting a super parameter or a network structure of the neural network model, and based on the adjusted neural network model, performing the step of re-executing to train the neural network model by adopting a random gradient descent algorithm based on the candidate frame and the training set, and based on the verification set, evaluating the trained neural network model until the repeated training times reach the specified times, and when the evaluation index meets the preset condition, wherein the super parameter comprises at least one of a learning rate, a batch size and iteration times;

and when the evaluation index meets the preset condition, testing the trained neural network model through a test set to obtain a trained neural network model serving as the target detection model.

In some optional embodiments, inputting the image to be detected into the target detection model to obtain a detection result, including:

inputting the image to be detected into the target detection model;

extracting image features from the image to be detected through the target detection model;

extracting feature pyramids from the image features through a feature pyramid network in the target detection model to obtain feature graphs with different sizes;

and carrying out pooling and convolution operation on the feature images with different sizes to obtain the detection result, wherein when the industrial product exists in the image to be detected, the detection result comprises the category of the industrial product and a detection frame.

In some optional embodiments, clipping is performed in the image to be detected based on the detection frame, so as to obtain a clipped region image, which includes:

amplifying the detection frame by a specified multiple by taking the central position as a fixed point to obtain a cutting frame, wherein the specified multiple is more than 1 and less than 1.5;

and cutting along the cutting frame in the image to be detected to obtain the region image of the cutting frame.

In some optional embodiments, the segmenting the area image based on a preset segmentation model SAM to obtain a result image corresponding to a single industrial product includes:

inputting the region image and the center point of the region image to the SAM for each region image to obtain a mask corresponding to the industrial product in the region image;

multiplying the mask by a four-dimensional array [255,255,255,255] through a preset broadcasting mechanism, and performing AND operation with the region image to obtain a picture with a transparent background of the industrial product as the result image.

The application adopting the technical scheme has the following advantages:

in the technical scheme provided by the application, training and testing are performed by utilizing a candidate frame, a training set, a verification set and a testing set based on a neural network model of deep learning to obtain a target detection model; then, inputting the image to be detected into the target detection model, detecting whether an industrial product exists in the image to be detected, and carrying out frame selection labeling on the industrial product in the image to be detected through a detection frame when the industrial product exists; and cutting out the region of the industrial product based on the detection frame, and finally dividing the cut region image by utilizing the SAM to obtain a result image corresponding to the single industrial product. Therefore, accurate identification and segmentation of single industrial products can be realized without pixel-level labeling and training, and the efficiency of segmenting single industrial products is improved.

Drawings

The application may be further illustrated by means of non-limiting examples given in the accompanying drawings. It is to be understood that the following drawings illustrate only certain embodiments of the application and are therefore not to be considered limiting of its scope, for the person of ordinary skill in the art may admit to other equally relevant drawings without inventive effort.

Fig. 1 is a schematic flow chart of a machine vision-based method for identifying and dividing industrial products in an atmospheric natural environment test.

Fig. 2A is a schematic diagram of an image to be measured according to an embodiment of the present application.

Fig. 2B is a schematic diagram of the region image cropped from fig. 2A.

Fig. 2C is a schematic diagram of the resulting image segmented from fig. 2B.

Detailed Description

The present application will be described in detail below with reference to the drawings and the specific embodiments, wherein like or similar parts are designated by the same reference numerals throughout the drawings or the description, and implementations not shown or described in the drawings are in a form well known to those of ordinary skill in the art. In the description of the present application, the terms "first," "second," and the like are used merely to distinguish between descriptions and are not to be construed as indicating or implying relative importance.

Referring to fig. 1, the application provides a machine vision-based method for identifying and dividing industrial products in an atmospheric natural environment test, which can be applied to electronic equipment, and the electronic equipment executes or realizes the steps of the method.

It is understood that the electronic device may include a processing module and a memory module. The storage module stores a computer program which, when executed by the processing module, enables the electronic device to perform the corresponding steps in the machine vision-based atmospheric natural environment test industrial product identification segmentation method.

The electronic device may be, but is not limited to, an intelligent robot, a personal computer, and the like. For example, if the industrial product image acquisition site is in a severe atmospheric natural environment and cannot be operated manually, at this time, the robot can be controlled to perform image acquisition on site to replace manual operation, and the robot performs identification and segmentation of the industrial product in the image.

In the method for identifying and dividing the industrial products in the atmospheric natural environment test based on machine vision, the identified and divided industrial products can be flexibly determined according to actual conditions. For example, the industrial product may be, but is not limited to, a component of a product, an entire product, etc., e.g., the industrial product may be a sheet-like structure as shown in fig. 2A.

The method for identifying and dividing the industrial products in the atmospheric natural environment test based on the machine vision comprises the following steps:

step 110, acquiring an initial image set, wherein the initial image set comprises images obtained by shooting industrial products under an atmospheric natural environment test;

step 120, preprocessing the initial image set to obtain a preprocessed data set, wherein the preprocessed data set comprises a training set, a verification set and a test set;

130, clustering the annotation frames in the images of the training set to obtain candidate frames, wherein the candidate frames are the annotation frames closest to each clustering center point, and the clustering center points are obtained by clustering the annotation frames;

step 140, training and testing a preset neural network model based on deep learning based on the candidate frame, the training set, the verification set and the test set to obtain a target detection model;

step 150, inputting the image to be detected into the target detection model to obtain a detection result;

step 160, when the detection result indicates that a detection frame of the industrial product exists in the image to be detected, cutting is performed in the image to be detected based on the detection frame, and a cut area image is obtained;

and step 170, dividing the regional image based on a preset SAM (Segment Anything Model) to obtain a result image corresponding to the single industrial product.

The following will explain in detail the steps of the machine vision-based atmospheric natural environment test industrial product identification and segmentation method, as follows:

in step 110, the initial image set is a set of original images that the user has prepared in advance for model training, testing. The initial image set comprises a large number of original images obtained by shooting industrial products under an atmospheric natural environment test. The acquired images may be used as test data in preparation for industrial product identification and segmentation.

The electronic device may obtain the initial image set from a local or other device, and the manner of obtaining the initial image set is not particularly limited herein. In addition, in the initial image set, the number of images may be flexibly determined according to actual conditions, and is not particularly limited herein.

In step 120, the initial image set is preprocessed in order to allow the preprocessed data set to be directly used for model training and testing. The preprocessing can enrich the image data volume, reduce interference data and improve the effectiveness of the data set for model training test. The preprocessing can comprise the operations of image labeling, data cleaning, data enhancement and the like.

In this embodiment, step 120 performs preprocessing on the initial image set to obtain a preprocessed data set, which may include:

In this embodiment, when labeling an image with a labelImg tool, a user may box each industrial product in the image and label the category of the industrial product. For example, the category of the industrial product may be labeled according to the actual situation, and the labeling manner and labeling content are not particularly limited.

The method for cleaning and screening the first data set based on a preset cleaning strategy comprises the following steps of:

In this embodiment, the labeling anomaly and the image quality anomaly may both be identified and selected manually. The user can delete the abnormal labeling data and the abnormal quality image by checking the abnormal labeling data and the abnormal quality image.

The labeling data of an anomaly may refer to: the type of annotation is not consistent with the actual type of the corresponding object in the image.

An image of abnormal quality may refer to: blurred or distorted images, and images that contain too strong shadows or highlights.

Understandably, by checking whether the labels in the first dataset are correct, wrong labels and images corresponding to the wrong labels are filtered; then, looking at whether the image in the first dataset is clearly visible, removing blurred or distorted images, removing images that contain too strong shadows or highlights, and thus obtaining the second dataset.

In this embodiment, the second data set is subjected to a data enhancement operation, which includes random flipping, random rotation, and random adjustment of brightness and contrast of the image, to obtain a third data set. Therefore, the data volume for model training can be enriched, and the training effect of the model can be improved.

In this embodiment, in the third data set, the preset proportions of the training set, the verification set and the test set may be flexibly set according to practical situations, for example, the proportion of the training set, the verification set and the test set may be 3:1:1, where the preset proportions are not specifically limited.

In this embodiment, step 130 clusters the labeling frames in the images of the training set to obtain candidate frames, which may include:

extracting coordinates of a labeling frame corresponding to the industrial product in the images of the training set, wherein the labeling frame can be a horizontal rectangular frame;

normalizing all the coordinates to obtain normalized coordinates;

As an example, the implementation of step 130 may be as follows:

firstly, extracting coordinate information of a labeling frame of all positive samples (such as pictures containing industrial products) in a training set to form an array;

normalizing the extracted labeling frame coordinate array, and scaling the values of the labeling frame coordinates to be within the range of [0,1 ];

selecting the number K of candidate frames to be generated as the clustering number of a K-means algorithm, wherein K can be flexibly selected according to actual conditions;

clustering the normalized coordinate array of the marking frame by adopting a K-means algorithm to obtain K clustering center points;

for each cluster center, selecting the labeling frame in the training set closest to the single cluster center as the representative frame of the cluster, namely the candidate frame.

In this embodiment, step 140 trains and tests a preset neural network model based on deep learning based on the candidate frame, the training set, the verification set and the test set to obtain a target detection model, which includes:

training the neural network model based on the candidate frame and the training set by adopting a random gradient descent (Stochastic Gradient Descent, SGD) algorithm, wherein the neural network model comprises any one of a YOLO model and a Fast R-CNN model;

In this embodiment, taking the neural network model as a YOLO model as an example, the implementation procedure of step 140 may be as follows:

modifying a loss function of the YOLO model according to actual requirements, wherein the loss function comprises positioning errors, classification errors, confidence errors and the like;

carrying out YOLO model training by using data such as images and labels in a training set and adopting a random gradient descent algorithm;

setting super parameters in the training process, wherein the super parameters can comprise learning rate, batch size, iteration times and the like, and can be flexibly set according to actual conditions;

evaluating the trained YOLO model through the verification set to obtain evaluation indexes such as accuracy, recall rate and F1 score, evaluating the YOLO model according to the obtained indexes, and if the YOLO model does not reach the preset condition, adjusting super parameters or network structure of the model, and retraining; wherein, meeting the preset condition may refer to: the obtained evaluation indexes such as the accuracy rate, the recall rate, the F1 score and the like are in the corresponding threshold range, and the threshold range can be flexibly set according to actual conditions;

the model is tested using the test set, and the generalization ability of the YOLO model on new data (which may be referred to as data in the test set) is verified according to the evaluation index.

In this embodiment, step 150 inputs the image to be detected into the target detection model to obtain a detection result, including:

inputting the image to be detected into the target detection model;

As an example, the implementation of step 150 may be as follows:

convoluting the picture of the industrial product (namely the picture to be detected) through a target detection model, carrying out nonlinear transformation and carrying out batch normalization to obtain the image characteristics of the industrial product;

extracting feature pyramids from image features through a feature pyramid network, adjusting the size and dimension of feature graphs, and sampling to the same size for fusion, so that feature information in the image is more comprehensively captured, and the accuracy of target detection is improved;

the class and the detection frame of the product in the example image are obtained by pooling and convolution of feature images with different sizes, and the prediction result is further optimized through an NMS (Non-Maximum Suppression ) algorithm, so that repeated prediction of the same target is avoided.

In this embodiment, step 160 includes clipping the image to be detected based on the detection frame to obtain a clipped region image, including:

As an example, the implementation of step 160 may be as follows:

acquiring the central position of an instance of the industrial product according to the coordinate information of the detection frame, and storing the central position in a corresponding txt file;

amplifying the corresponding detection frame by 1.15 times by taking the central position as a fixed point to form a cutting frame;

the industrial products appearing in the image are individually cropped along the crop box using OpenCV (Open Source Computer Vision, a cross-platform computer vision library), and the cropped images are saved one by one.

In this embodiment, step 170 of segmenting the region image based on a preset segmentation model SAM to obtain a result image corresponding to a single industrial product, includes:

As an example, the implementation of step 170 may be as follows:

initializing a SAM model by using a default weight file;

inputting a central point corresponding to the region image and the industrial product to prompt the SAM model to obtain a mask corresponding to the industrial product, wherein the mask is a binary array with the length and the width of the region image pixels;

multiplying the mask by a four-dimensional array [255,255,255,255] through a broadcasting mechanism, and performing AND operation with the region image to obtain a transparent background picture corresponding to the industrial product.

As an example, an image to be measured may refer to fig. 2A, and in general, a single image to be measured may include a plurality of industrial products. As shown in FIG. 2A, the use of a trained object detection model (e.g., a YOLO model) allows accurate predictions of the category of industrial products and the specific location of each product to be framed. The detection box in FIG. 2A provides a reference for the crop box and is also an important hint for subsequent segmentation using the SAM model.

After the region of one detection frame in fig. 2A is cut, as shown in fig. 2B, according to the industrial product image cut by the cutting frame, the example information contained after cutting is complete, and a foundation is laid for the subsequent complete division example. The size of the cut image is reduced, and the computer memory requirement during segmentation is reduced.

After the region image shown in fig. 2B is divided by the SAM model, a resultant image as shown in fig. 2C can be obtained. Understandably, the cut image is segmented based on the SAM model point of use prompt to obtain a corresponding mask, and then the cut image is processed based on the mask to obtain a transparent background image of the corresponding example, that is, the industrial product image shown in fig. 2C, where the background of the industrial product image is white or transparent. After the background of the industrial product image is removed, the result image shown in fig. 2C is obtained, so that subsequent quality detection is facilitated for a single image, and interference of the image background on subsequent quality detection is reduced.

Based on the design, the method fuses the target detection model (such as a YOLO model) and the SAM, can realize the identification and segmentation of industrial products or other targets (such as vehicles and pedestrians) in the atmospheric environment, is beneficial to reducing the labor cost and time of labeling, and improves the efficiency of identifying and segmenting single industrial products from the whole graph. In addition, after the target detection model and the SAM are fused, the automatic identification and segmentation of the industrial products can be realized, the identification speed and the segmentation effect are improved, the portability is high, and a foundation is laid for the experimental detection of the industrial products in the intelligent atmospheric environment.

In this embodiment, the processing module may be an integrated circuit chip with signal processing capability. The processing module may be a general purpose processor. For example, the processor may be a central processing unit (Central Processing Unit, CPU), digital signal processor (Digital Signal Processing, DSP), application specific integrated circuit (Application Specific Integrated Circuit, ASIC), field programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or may implement or perform the methods, steps, and logic blocks disclosed in embodiments of the application.

The memory module may be, but is not limited to, random access memory, read only memory, programmable read only memory, erasable programmable read only memory, electrically erasable programmable read only memory, and the like. In this embodiment, the storage module may be configured to store a target detection model, a segmentation model SAM, an image to be measured, a result image, and the like. Of course, the storage module may also be used to store a program, and the processing module executes the program after receiving the execution instruction.

From the foregoing description of the embodiments, it will be apparent to those skilled in the art that the present application may be implemented in hardware, or by means of software plus a necessary general hardware platform, and based on this understanding, the technical solution of the present application may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (may be a CD-ROM, a U-disc, a mobile hard disk, etc.), and includes several instructions for causing a computer device (may be a personal computer, an electronic device, or a network device, etc.) to execute the method described in the respective implementation scenario of the present application.

In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other manners as well. The above-described apparatus and method embodiments are merely illustrative, for example, flow diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions. In addition, functional modules in the embodiments of the present application may be integrated together to form a single part, or each module may exist alone, or two or more modules may be integrated to form a single part.

The above description is only an example of the present application and is not intended to limit the scope of the present application, and various modifications and variations will be apparent to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. An atmospheric natural environment test industrial product identification and segmentation method based on machine vision is characterized by comprising the following steps of:

2. The method of claim 1, wherein preprocessing the initial image set to obtain a preprocessed data set comprises:

3. The method of claim 2, wherein the performing a cleaning filter on the first data set based on a preset cleaning policy to obtain a second data set comprises:

4. The method of claim 1, wherein clustering the annotation boxes in the images of the training set to obtain candidate boxes comprises:

normalizing all the coordinates to obtain normalized coordinates;

5. The method of claim 1, wherein training and testing a predetermined deep learning-based neural network model based on the candidate box, the training set, the validation set, and the test set to obtain a target detection model comprises:

6. The method according to claim 1, wherein inputting the image to be measured into the target detection model to obtain a detection result includes:

inputting the image to be detected into the target detection model;

7. The method according to claim 1, wherein cropping the image to be detected based on the detection frame to obtain a cropped area image, comprising:

8. The method according to claim 1, wherein segmenting the region image based on a preset segmentation model SAM, results in a result image corresponding to a single industrial product, comprises: