CN118566235A - A method and system for visually detecting surface defects of fasteners - Google Patents

Abstract

The invention discloses a visual detection method and a visual detection system for surface defects of a fastener, and belongs to the technical field of intelligent manufacturing. The method comprises the following steps: collecting an image of a fastener; processing the image; respectively deploying defect detection models at a cloud end and a terminal; the terminal detects the defects of the images of each fastener through the defect detection model, and alarms after the defect types are identified; in addition, the terminal extracts an image of one fastener every N fasteners and sends the image to the cloud, the cloud detects defects through the defect detection model, and after the defect types are identified, the terminal is informed to alarm; and when the terminal gives an alarm, judging that the corresponding fastener has surface defects. According to the invention, the product quality is automatically detected in the production process of the product in a visual identification mode, so that the quality inspection work of the product is finished on one hand, and the timely early warning effect is realized on the other hand, so that the production rate of waste parts can be greatly reduced, and raw materials are saved.

Description

A method and system for visually detecting surface defects of fasteners

Technical Field

The present invention relates to the field of intelligent manufacturing technology, and in particular to a method and system for visually detecting surface defects of fasteners.

Background Art

With the rapid development of computer technology and artificial intelligence technology, the digital upgrade of the manufacturing industry has a technical foundation, and intelligent manufacturing has also become an important development direction of the manufacturing industry. During the cold pressing production process of fasteners, defects such as missing holes, scratches, burrs, and cracks will occur on the metal surface. These are surface defects of fasteners. The fastener industry belongs to the traditional manufacturing industry, and the intelligence and digitization of this industry have always been relatively low. The quality inspection of the fastener products produced has long been carried out by manual inspection. Only simple sensors are used in the production equipment to realize the counting function, and no automated detection methods are used to detect the quality of the products. This leads to a large workload for manual quality inspection on the one hand, and on the other hand, manual quality inspection is prone to missed inspections and false inspections.

At present, the mainstream methods of industrial product surface defect detection based on vision are mainly divided into the following two categories:

1. Defect detection method based on traditional methods

Traditional visual defect detection methods usually learn a model of a normal image, and implement defect detection in the detection stage based on the degree of match between the image to be detected and the model. Traditional visual defect detection methods mainly include defect detection methods based on template matching, statistical models, image decomposition, frequency domain analysis, sparse coding reconstruction, and classification surface construction. Traditional methods are suitable for recognition situations with fixed scenes and simple backgrounds. They are easily disturbed by external factors such as lighting changes and noise. They have weak robustness and generalization capabilities, poor versatility, and cannot adapt to scene changes. There are many kinds of products that need to be produced in the fastener industry. In this application, traditional methods are difficult to expand and have poor generalization.

2. Defect detection method based on deep learning

Compared with traditional methods, deep learning methods can automatically learn features and have strong algorithm generalization capabilities. They have been widely used in the field of visual defect detection. According to different data labels, they can be generally divided into fully supervised learning models, unsupervised learning models, and semi-supervised/weakly supervised learning models. However, when deep learning methods are applied to specific industrial scenarios, they are constrained by the on-site environment and resources and still need to be optimized according to actual requirements to meet the requirements of accuracy and real-time performance. In addition, the collection of training data, the accuracy of the model, and the reasoning speed of the model are also key issues that deep learning models need to address.

In short, for fastener surface defect detection, due to the limitations of the on-site environment, resources and timeliness, the deep learning model needs to be miniaturized and lightweight to meet the actual needs of the industry. To realize fastener visual inspection at the production site, the on-site environment is relatively complex, and it is necessary to consider miniaturization and integration of products, and integrate image acquisition, processing, detection and recognition, alarm and other functions into one, so as to facilitate on-site deployment. However, there is no relatively complete integrated solution in the existing technology.

Summary of the invention

In order to improve the efficiency of quality inspection and improve the inspection quality, the present invention proposes a fastener surface defect visual inspection method and system in combination with the characteristics of fastener products. The present invention automatically detects product quality during the production process by visual recognition, which can complete the product quality inspection work on the one hand, and realize timely early warning on the other hand, which can greatly reduce the generation rate of scrap parts and save raw materials.

In order to achieve the above object, the technical solution adopted by the present invention is:

A method for visually detecting surface defects of fasteners comprises the following steps:

Step 1: three cameras are set above and on both sides of the conveyor belt, and a through-beam sensor is set on both sides of the conveyor belt. When the through-beam sensor determines that a fastener passes by, the three cameras take multi-angle photos of the front, left and right sides of the fastener;

Step 2, processing the captured image;

Step 3: Deploy defect detection models on the cloud and the terminal respectively. The cloud and the terminal use the same defect detection model, but use different network parameter scales. The terminal uses a model based on YOLO-nano to achieve faster detection, while the cloud uses a version based on YOLO-large to achieve more accurate detection.

In step 4, the terminal performs defect detection on the image of each fastener through the defect detection model, and issues an alarm after identifying the defect type. In addition, the terminal extracts an image of one fastener for every N fasteners and sends it to the cloud. The cloud performs defect detection through the defect detection model, and after identifying the defect type, notifies the terminal to issue an alarm. When the terminal issues an alarm, it is determined that the corresponding fastener has a surface defect.

Furthermore, the processing described in step 2 includes:

(1) Noise reduction: Bilateral filtering is used to achieve denoising while maintaining edge information;

(2) Enhancement: Use sharpening filtering to enhance the edges and details in the image;

(3) Cutout: Cut out the fastener part in the image to reduce the impact of the background on defect identification. The cutout model is implemented based on YOLO-nano.

Furthermore, the defect detection model in step 3 simultaneously completes posture recognition and defect location, and its training method is:

(1) Sample construction

Collect real image data of fasteners at the production site. For each fastener, perform data annotation in different fastener postures according to the defect type. The defect types include: missing holes, incorrect angles, burrs on holes, R angle deviation, burrs, scratches, and multiple pieces. The postures include front, left, and right sides. The annotation uses the Label Studio tool to mark the specific defect locations by marking boxes. The samples are divided into training sets, validation sets, and test sets.

(2) Model training

The loss functions of the model include BCE loss, DFL loss and CIOU loss, where BCE loss is used as classification loss, DFL loss and CIOU loss are used as bounding box regression losses. The three losses are weighted and summed to form a total loss function, which is used to guide network training.

During the model iteration process, the mean average precision (mAP) is set as the evaluation indicator. When the indicator on the validation set becomes stable, the training ends and the model parameters with the best evaluation indicator are selected as the final model.

A fastener surface defect visual inspection system, comprising a conveyor belt, a fill light system, a shooting system, a terminal defect recognition system and a cloud defect recognition system;

The fill light system is arranged beside the conveyor belt and is used to fill light for the fasteners on the conveyor belt;

The shooting system includes three cameras arranged above and on the left and right sides of the conveyor belt, and a shooting sensor arranged on both sides of the conveyor belt; when the shooting sensor determines that a fastener passes by, the three cameras are triggered to shoot the fastener from multiple angles, such as the front, left side, and right side;

The terminal defect recognition system and the cloud defect recognition system communicate through a network, and the terminal defect recognition system and the cloud defect recognition system are respectively deployed with defect detection models; the terminal defect recognition system and the cloud defect recognition system use the same defect detection model, but use different network parameter scales, wherein the terminal uses a model based on YOLO-nano to achieve faster detection, and the cloud uses a version based on YOLO-large to achieve more accurate detection;

The terminal defect recognition system performs defect detection on the image of each fastener based on the images taken by the shooting system through the defect detection model, and issues an alarm after identifying the defect type. In addition, the terminal extracts an image of one fastener for every N fasteners and sends it to the cloud. The cloud performs defect detection through the defect detection model, and after identifying the defect type, it notifies the terminal to issue an alarm. When the terminal issues an alarm, it is determined that the corresponding fastener has surface defects.

Furthermore, after the shooting system shoots the image, it also processes the image, including:

(1) Noise reduction: Bilateral filtering is used to achieve denoising while maintaining edge information;

(2) Enhancement: Use sharpening filtering to enhance the edges and details in the image;

(3) Cutout: Cut out the fastener part in the image to reduce the impact of the background on defect identification. The cutout model is implemented based on YOLO-nano.

Furthermore, the fill light system, the shooting system, and the terminal defect recognition system are integrated into an integrated device.

The beneficial effects of the present invention are:

1. The present invention realizes online detection of various types of defects on the production line by integrating data collection and processing in the terminal and realizing defect detection of different degrees of difficulty in the terminal and the cloud.

2. The present invention integrates image acquisition, processing, detection and recognition, alarm and other functions into one in the fastener production line environment, which is easy to deploy.

3. The device of the present invention adopts a surround-type fill-light system to solve the problem of insufficient illumination, and adopts multi-angle acquisition to identify and locate defects in different postures.

4. The present invention proposes a posture-based multi-task deep defect detection model, which implements the two tasks of fastener posture recognition and defect detection in one model, that is, distinguishing the posture of the fastener while identifying defects.

5. The present invention proposes a strategy for fastener defect classification, which classifies defects according to the degree of impact of defects on the normal use of fasteners to meet the requirements of fastener defect detection in actual scenarios.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram showing the principle of a method for visually detecting surface defects of fasteners according to an embodiment of the present invention.

FIG. 2 is a schematic diagram of the structure of a fastener surface defect visual inspection system according to an embodiment of the present invention.

FIG3 is a schematic diagram of a posture-based multi-task depth defect detection model in an embodiment of the present invention.

DETAILED DESCRIPTION

The present invention is further described in detail below in conjunction with the accompanying drawings.

A method for visually detecting surface defects of fasteners comprises the following steps:

Step 1: three cameras are set above and on both sides of the conveyor belt, and a through-beam sensor is set on both sides of the conveyor belt. When the through-beam sensor determines that a fastener passes by, the three cameras take multi-angle photos of the front, left and right sides of the fastener;

Step 2, processing the captured image;

Step 3: Deploy defect detection models on the cloud and the terminal respectively. The cloud and the terminal use the same defect detection model, but use different network parameter scales. The terminal uses a model based on YOLO-nano to achieve faster detection, while the cloud uses a version based on YOLO-large to achieve more accurate detection.

In step 4, the terminal performs defect detection on the image of each fastener through the defect detection model, and issues an alarm after identifying the defect type. In addition, the terminal extracts an image of one fastener for every N fasteners and sends it to the cloud. The cloud performs defect detection through the defect detection model, and after identifying the defect type, notifies the terminal to issue an alarm. When the terminal issues an alarm, it is determined that the corresponding fastener has a surface defect.

Among them, N can be determined according to the actual processing speed and network speed to ensure that when the cloud detects the next fastener, the previous fastener has been detected.

Furthermore, the processing described in step 2 includes:

(1) Noise reduction: Bilateral filtering is used to achieve denoising while maintaining edge information;

(2) Enhancement: Use sharpening filtering to enhance the edges and details in the image;

(3) Cutout: Cut out the fastener part in the image to reduce the impact of the background on defect identification. The cutout model is implemented based on YOLO-nano.

The defect detection model in step 3 simultaneously completes posture recognition and defect location, and its training method is:

(1) Sample construction

Collect real image data of fasteners at the production site. For each fastener, perform data annotation in different fastener postures according to the defect type. The defect types include: missing holes, incorrect angles, burrs on holes, R angle deviation, burrs, scratches, and multiple pieces. The postures include front, left, and right sides. The annotation uses the Label Studio tool to mark the specific defect locations by marking boxes. The samples are divided into training sets, validation sets, and test sets.

(2) Model training

The loss functions of the model include BCE loss, DFL loss and CIOU loss, where BCE loss is used as classification loss, DFL loss and CIOU loss are used as bounding box regression losses. The three losses are weighted and summed to form a total loss function, which is used to guide network training.

During the model iteration process, the mean average precision (mAP) is set as the evaluation indicator. When the indicator on the validation set becomes stable, the training ends and the model parameters with the best evaluation indicator are selected as the final model.

A fastener surface defect visual inspection system, comprising a conveyor belt, a fill light system, a shooting system, a terminal defect recognition system and a cloud defect recognition system;

The fill light system is arranged beside the conveyor belt and is used to fill light for the fasteners on the conveyor belt;

The shooting system includes three cameras arranged above and on the left and right sides of the conveyor belt, and a shooting sensor arranged on both sides of the conveyor belt; when the shooting sensor determines that a fastener passes by, the three cameras are triggered to shoot the fastener from multiple angles, such as the front, left side, and right side;

The terminal defect recognition system and the cloud defect recognition system communicate through a network, and the terminal defect recognition system and the cloud defect recognition system are respectively deployed with defect detection models; the terminal defect recognition system and the cloud defect recognition system use the same defect detection model, but use different network parameter scales, wherein the terminal uses a model based on YOLO-nano to achieve faster detection, and the cloud uses a version based on YOLO-large to achieve more accurate detection;

The terminal defect recognition system performs defect detection on the image of each fastener based on the images taken by the shooting system through the defect detection model, and issues an alarm after identifying the defect type. In addition, the terminal extracts an image of one fastener for every N fasteners and sends it to the cloud. The cloud performs defect detection through the defect detection model, and after identifying the defect type, it notifies the terminal to issue an alarm. When the terminal issues an alarm, it is determined that the corresponding fastener has surface defects.

Furthermore, the fill light system, shooting system, and terminal defect recognition system are integrated into an integrated device, which integrates the required software and hardware to facilitate on-site deployment. The data acquisition and preprocessing, detection and recognition, and post-processing frameworks used can integrate a variety of different defect detection models and achieve the ability to be compatible with different advanced models. This device mainly includes the following three modules:

1. Image acquisition and processing: In order to capture reliable fastener product images directly on the production line, the acquisition system is designed with a surround fill light system to compensate for the lack of light at the production site. In order to capture complete product images, a multi-angle acquisition method is adopted. In addition, the captured images are pre-processed by noise reduction, enhancement, etc. to improve the image quality and prepare for subsequent recognition.

2. Defect detection: Defect detection adopts a cloud-end combination approach to provide services in both cloud and terminal forms; defect detection adopts a defect detection model based on deep learning, and adopts a recognition method that integrates posture recognition and defect positioning according to the actual situation of the images captured on the production line.

3. Post-detection processing: After detection and identification, defect alarms, automatic sorting and other actions should be performed to improve the intelligence level of the production line.

As shown in Figure 1, each module contains several data processing links. The detailed description of the data processing flow in different links is as follows:

101 Fill Light System

This device is directly used in fastener production lines. The lighting conditions on the production line are relatively poor, and there is a lot of oil on the conveyor belt. Since the oily background and the workpiece color are not easy to distinguish, a surround fill light system is designed to reduce the impact of insufficient on-site lighting on the quality of captured images.

102 Multi-angle acquisition

Since the workpieces sent from the production line have different postures after arriving at the conveyor belt, the defects of the workpieces observed from different angles are different. In order to fully observe the different defects on the front, back, and sides of the workpiece, it is necessary to capture the workpiece images from different angles at the same time, so as to simultaneously identify and locate defects in different postures during the subsequent defect identification process. For example, a workpiece with a 90-degree corner has one hole and two holes on the two faces of the corner, which results in different images seen from different perspectives, and therefore different defect types.

103 Image Preprocessing

The production site conditions are complex. Even if fill light processing is done, the captured image will be affected by the environment and there will be some problems. Before sending it to the recognition model, the image needs to be preprocessed to minimize the interference of noise on subsequent recognition. The preprocessing that needs to be done includes but is not limited to the following: image cropping, scaling, filtering and noise reduction, image enhancement, sample enhancement, cutout and other preprocessing.

201 Cloud and terminal integrated defect detection

There are many types of fastener products, and the corresponding defect types are also varied, such as the following defects: missing holes, incorrect angles, burrs on holes, deviated R angles, burrs on edges, and scratches.

It is difficult to identify various types of defects. It is difficult to identify all defects using a single model. Therefore, a cloud-end combined defect detection framework is adopted, as shown in Figure 2. For defect types that appear frequently, a lightweight model is deployed in the terminal all-in-one for real-time processing. According to the fastener production line environment, this device integrates image acquisition, processing, detection and recognition, alarm and other functions into one, and designs a miniaturized integrated terminal device to meet the needs of real-time detection in industrial sites.

For defects that are difficult to identify in real time at the terminal, the image is transmitted to the cloud through the network, and a model with stronger detection function is used for recognition. For example, a three-dimensional stereoscopic vision detection model is used to detect defects with incorrect angles, and the detection results are sent back to the terminal in real time for subsequent processing. The cloud and the terminal are independent of each other and complement each other for verification. On the one hand, the defect detection results of the cloud can make up for the insufficient defect detection caused by the insufficient processing power of the terminal; on the other hand, the cloud results can further confirm the defect detection results of the terminal and improve the detection accuracy.

202 Defect Vision Detection Model

As shown in Figure 3, various defect detection models can be compatible in the defect detection link. Here, a posture-based multi-task deep defect detection model is proposed according to the product deployment scenarios and the characteristics of fasteners. As shown in Figure 3, since the defects of the workpiece surface images taken with different postures are different, it is necessary to locate the defects with a clear workpiece posture.

In order to deploy the model to the terminal to complete the defect detection task in real time, the model uses the single-stage defect detection model Yolo, which completes two tasks at the same time, namely posture recognition and defect location, in order to improve the model recognition performance and complete the defect location task of distinguishing postures. Therefore, the goal of the Yolo model is a combination of the two goals, and the two goals constrain each other.

301 Defect Classification and Warning

The purpose of this device is to promptly detect defective workpieces during the fastener production process, so as to find out whether the mold is damaged and other problems. Therefore, for the identified defects, an alarm is used to promptly notify the staff to conduct an investigation. There are differences in the severity of fastener defects, and different defects need to be graded. For example, a serious defect such as a missing hole causes the workpiece to be unusable, which is a serious defect; scratches and burrs do not affect the use of the workpiece, but affect the appearance of the product, so the defect level is relatively low. Therefore, this device grades the defect types of fasteners according to their severity, and combines the defect identification results to make differentiated alarms according to different defect levels when issuing defect alarms.

Furthermore, the device can also realize automatic sorting of defective workpieces, which can not only reduce the defective rate but also save manpower. Defect sorting can be realized on the conveyor belt. Based on the results of defect identification, instructions are sent to the sorting device, and defective workpieces are sorted by automatic mechanical devices.

Claims (6)

1. A method for visually detecting surface defects of fasteners, comprising the following steps: Step 1: three cameras are set above and on both sides of the conveyor belt, and a through-beam sensor is set on both sides of the conveyor belt. When the through-beam sensor determines that a fastener passes by, the three cameras take multi-angle photos of the front, left and right sides of the fastener; Step 2, processing the captured image; Step 3: Deploy defect detection models on the cloud and the terminal respectively. The cloud and the terminal use the same defect detection model, but use different network parameter scales. The terminal uses a model based on YOLO-nano to achieve faster detection, while the cloud uses a version based on YOLO-large to achieve more accurate detection. In step 4, the terminal performs defect detection on the image of each fastener through the defect detection model, and issues an alarm after identifying the defect type. In addition, the terminal extracts an image of one fastener for every N fasteners and sends it to the cloud. The cloud performs defect detection through the defect detection model, and after identifying the defect type, notifies the terminal to issue an alarm. When the terminal issues an alarm, it is determined that the corresponding fastener has a surface defect. 2. A method for visually inspecting surface defects of fasteners according to claim 1, characterized in that the processing in step 2 comprises: (1) Noise reduction: Bilateral filtering is used to achieve denoising while maintaining edge information; (2) Enhancement: Use sharpening filtering to enhance the edges and details in the image; (3) Cutout: Cut out the fastener part in the image to reduce the impact of the background on defect identification. The cutout model is implemented based on YOLO-nano. 3. A method for visually inspecting surface defects of fasteners according to claim 1, characterized in that the defect detection model in step 3 simultaneously completes posture recognition and defect location, and its training method is: (1) Sample construction Collect real image data of fasteners at the production site. For each fastener, perform data annotation in different fastener postures according to the defect type. The defect types include: missing holes, incorrect angles, burrs on holes, R angle deviation, burrs, scratches, and multiple pieces. The postures include front, left, and right sides. The LabelStudio tool is used for annotation to mark the specific defect locations by marking boxes. The samples are divided into training, validation, and test sets. (2) Model training The loss functions of the model include BCEloss, DFLloss and CIOUloss, where BCE loss is used as classification loss, DFLloss and CIOUloss are used as bounding box regression losses. The three losses are weighted and summed to form a total loss function, which is used to guide network training. During the model iteration process, the mean average precision (mAP) is set as the evaluation indicator. When the indicator on the validation set becomes stable, the training ends and the model parameters with the best evaluation indicator are selected as the final model. 4. A fastener surface defect visual inspection system, characterized by comprising a conveyor belt, a fill light system, a shooting system, a terminal defect recognition system and a cloud defect recognition system; The fill light system is arranged beside the conveyor belt and is used to fill light for the fasteners on the conveyor belt; The shooting system includes three cameras arranged above and on the left and right sides of the conveyor belt, and a shooting sensor arranged on both sides of the conveyor belt; when the shooting sensor determines that a fastener passes by, the three cameras are triggered to shoot the fastener from multiple angles, such as the front, left side, and right side; The terminal defect recognition system and the cloud defect recognition system communicate through a network, and the terminal defect recognition system and the cloud defect recognition system are respectively deployed with defect detection models; the terminal defect recognition system and the cloud defect recognition system use the same defect detection model, but use different network parameter scales, wherein the terminal uses a model based on YOLO-nano to achieve faster detection, and the cloud uses a version based on YOLO-large to achieve more accurate detection; The terminal defect recognition system performs defect detection on the image of each fastener based on the images taken by the shooting system through the defect detection model, and issues an alarm after identifying the defect type. In addition, the terminal extracts an image of one fastener for every N fasteners and sends it to the cloud. The cloud performs defect detection through the defect detection model, and after identifying the defect type, it notifies the terminal to issue an alarm. When the terminal issues an alarm, it is determined that the corresponding fastener has surface defects. 5. A fastener surface defect visual inspection system according to claim 4, characterized in that after the shooting system shoots the image, it also processes the image, including: (1) Noise reduction: Bilateral filtering is used to achieve denoising while maintaining edge information; (2) Enhancement: Use sharpening filtering to enhance the edges and details in the image; (3) Cutout: Cut out the fastener part in the image to reduce the impact of the background on defect identification. The cutout model is implemented based on YOLO-nano. 6. A fastener surface defect visual inspection system according to claim 4, characterized in that the fill light system, the shooting system, and the terminal defect recognition system are integrated into an integrated device.