CN118967672B - Industrial defect detection method, system, device and storage medium - Google Patents

Industrial defect detection method, system, device and storage medium Download PDF

Info

Publication number
CN118967672B
CN118967672B CN202411435220.7A CN202411435220A CN118967672B CN 118967672 B CN118967672 B CN 118967672B CN 202411435220 A CN202411435220 A CN 202411435220A CN 118967672 B CN118967672 B CN 118967672B
Authority
CN
China
Prior art keywords
image
images
feature map
defect detection
defect
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202411435220.7A
Other languages
Chinese (zh)
Other versions
CN118967672A (en
Inventor
孙家栋
雷超
苏秀娥
王泉
阚希
翟世臣
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Wuxi University
Original Assignee
Wuxi University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Wuxi University filed Critical Wuxi University
Priority to CN202411435220.7A priority Critical patent/CN118967672B/en
Publication of CN118967672A publication Critical patent/CN118967672A/en
Application granted granted Critical
Publication of CN118967672B publication Critical patent/CN118967672B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0004—Industrial image inspection
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/84—Systems specially adapted for particular applications
    • G01N21/88—Investigating the presence of flaws or contamination
    • G01N21/8851—Scan or image signal processing specially adapted therefor, e.g. for scan signal adjustment, for detecting different kinds of defects, for compensating for structures, markings, edges
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N25/00—Investigating or analyzing materials by the use of thermal means
    • G01N25/72—Investigating presence of flaws
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N29/00—Investigating or analysing materials by the use of ultrasonic, sonic or infrasonic waves; Visualisation of the interior of objects by transmitting ultrasonic or sonic waves through the object
    • G01N29/44—Processing the detected response signal, e.g. electronic circuits specially adapted therefor
    • G01N29/4481—Neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/60—Image enhancement or restoration using machine learning, e.g. neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/70—Denoising; Smoothing
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/90—Dynamic range modification of images or parts thereof
    • G06T5/94—Dynamic range modification of images or parts thereof based on local image properties, e.g. for local contrast enhancement
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/84—Systems specially adapted for particular applications
    • G01N21/88—Investigating the presence of flaws or contamination
    • G01N21/8851—Scan or image signal processing specially adapted therefor, e.g. for scan signal adjustment, for detecting different kinds of defects, for compensating for structures, markings, edges
    • G01N2021/8887—Scan or image signal processing specially adapted therefor, e.g. for scan signal adjustment, for detecting different kinds of defects, for compensating for structures, markings, edges based on image processing techniques
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30108—Industrial image inspection
    • G06T2207/30164—Workpiece; Machine component

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Analytical Chemistry (AREA)
  • Immunology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Chemical & Material Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Biochemistry (AREA)
  • General Health & Medical Sciences (AREA)
  • Pathology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Signal Processing (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Quality & Reliability (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses an industrial defect detection method, an industrial defect detection system, an industrial defect detection device and a storage medium, which relate to the technical field of automation and comprise the steps of collecting multi-mode data of a workpiece through a high-resolution visible light camera, an infrared camera and an ultrasonic sensor, preprocessing, carrying out feature extraction and fusion on the preprocessed data by using a multi-mode data fusion algorithm to generate a comprehensive feature map, analyzing illumination conditions of the comprehensive feature map through a computer vision technology, dynamically adjusting the illumination conditions to generate a corrected comprehensive feature map, constructing a defect detection model, carrying out example segmentation and target detection on the corrected comprehensive feature map to generate a preselected frame and marking a possible defect area, introducing an online learning mechanism, updating and optimizing the defect detection model in real time according to the newly collected data, and accurately identifying and positioning defects through the multi-mode data fusion, dynamic illumination condition correction and the online learning mechanism, so that the accuracy of industrial defect detection is remarkably improved.

Description

Industrial defect detection method, system, device and storage medium
Technical Field
The invention relates to the technical field of automation, in particular to an industrial defect detection method, an industrial defect detection system, an industrial defect detection device and a storage medium.
Background
In recent years, with rapid development of manufacturing industry and continuous progress of automation technology, industrial defect detection has become a key link for improving product quality and production efficiency. The traditional defect detection method mainly relies on manual visual inspection, and the method is time-consuming and labor-consuming, is easily affected by human factors, and has low inconsistency and reliability of detection results. With the rapid development of computer vision and machine learning technologies, image-based automatic defect detection methods are becoming a research hotspot.
However, the existing defect detection method still has some defects, especially in aspects of multi-mode data fusion, illumination condition correction, model real-time update and the like. Firstly, although the multi-mode data fusion technology can improve the detection accuracy, the existing fusion method often depends on a fixed feature extraction mode, and the dynamic adaptability to different mode data is lacking, so that the detection effect under the complex working condition is poor. Secondly, the influence of the change of the illumination condition on the image quality is remarkable, and the existing illumination correction method mostly adopts a static correction strategy, cannot adapt to the environmental change in real time, and causes unstable detection results. In addition, existing defect detection models often require retraining in the face of new defect types and operating condition changes, lacking the ability to update and optimize in real time.
Disclosure of Invention
The present invention has been made in view of the above-described problems occurring in the prior art.
Therefore, the invention provides an industrial defect detection method, an industrial defect detection system, an industrial defect detection device and a storage medium, which solve the problems of insufficient multi-mode data fusion, illumination condition influence and poor model adaptability in the existing industrial defect detection method.
In order to solve the technical problems, the invention provides the following technical scheme:
In a first aspect, embodiments of the present invention provide an industrial defect detection method, including,
Collecting multi-mode data of a visible light image, an infrared image and an ultrasonic image of a workpiece to be detected, and preprocessing the multi-mode data;
Using a multi-mode data fusion algorithm to fuse the multi-mode data of the preprocessed visible light image, the preprocessed infrared image and the preprocessed ultrasonic image to generate a comprehensive feature map;
Analyzing the illumination condition of the image by using a computer vision technology, and dynamically adjusting the illumination condition to obtain a corrected comprehensive feature map;
constructing a defect detection model, performing example segmentation and target detection on the corrected comprehensive feature map, generating a preselected frame and marking a possible defect area;
outputting defect information of the workpiece according to the detection result, and performing visualization to generate a labeling image;
An online learning mechanism is introduced, and a defect detection model is updated and optimized in real time according to the newly acquired multi-mode data of the visible light image, the infrared image and the ultrasonic image.
As a preferable scheme of the industrial defect detection method, the method comprises the steps of collecting multi-mode data of a visible light image, an infrared image and an ultrasonic image of a workpiece to be detected, preprocessing the multi-mode data,
Shooting a visible light image of the workpiece by using a high-resolution visible light camera;
shooting an infrared image of the workpiece by using an infrared camera, and capturing temperature distribution information;
scanning the workpiece by using an ultrasonic sensor to acquire an ultrasonic image of internal structure information;
graying, denoising and contrast enhancement are carried out on a visible light image, temperature calibration and heat map generation are carried out on an infrared image, signal processing is carried out on an ultrasonic image, and a two-dimensional image is generated;
All the preprocessed images are unified to the same size and format.
As a preferable scheme of the industrial defect detection method, the invention adopts a multi-mode data fusion algorithm to fuse the preprocessed multi-mode data of the visible light image, the infrared image and the ultrasonic image, and the generation of the comprehensive feature map comprises the following steps,
Converting the visible light image, the infrared image and the ultrasonic image into tensor forms;
extracting a characteristic diagram of each mode by using a convolutional neural network;
The linear projection layer is used for adjusting the channel number of each feature map to be uniform dimension, and a new feature map is generated;
splicing each new mapped characteristic diagram to form a multi-mode characteristic diagram;
and carrying out feature fusion on the multi-mode feature map by using a multi-head self-attention mechanism to generate a comprehensive feature map.
As a preferable scheme of the industrial defect detection method, the invention uses computer vision technology to analyze the illumination condition of the image, dynamically adjusts the illumination condition, obtains a corrected comprehensive characteristic diagram, comprises the following steps,
Quantizing global characteristics of the image by calculating average brightness and average contrast of the image to obtain global illumination information of the image;
Capturing relative brightness change between pixels and adjacent pixels in the global illumination information image by using a local binary pattern algorithm, and identifying the positions of a highlight region and a dark region to generate a binary image;
determining the current illumination intensity and illumination direction according to the average brightness, the average contrast and the identified positions of the highlight and dark areas;
processing the image by adopting an illumination correction technology to generate a corrected image;
And carrying out histogram equalization on the corrected image to generate a corrected comprehensive feature map.
As a preferred embodiment of the industrial defect detection method of the present invention, wherein constructing a defect detection model, performing instance segmentation and object detection on the corrected integrated feature map, generating a pre-selected frame and marking a possible defect area comprises the steps of,
Inputting the corrected comprehensive feature map into a backbone network;
combining the high-level low-resolution feature map with the low-level high-resolution feature map by using the top-down path and the transverse connection of the feature pyramid to generate a multi-scale feature map;
sliding a window on each feature map to generate a set of anchor frames with a fixed number;
classifying each anchor frame and carrying out bounding box regression, judging whether the anchor frame contains a target or not, and adjusting the position and the size of the anchor frame;
selecting an anchor frame with higher score as a candidate region;
Using the region of interest pooling layer to adjust the candidate regions of different sizes to the same size;
Classifying each candidate region through the full connection layer, and determining which category belongs to;
the positions of the candidate areas are further adjusted through bounding box regression, and a final pre-selected box is generated;
generating a segmentation mask for each candidate region by segmentation branches;
generating a pre-selection frame for each candidate region classified as foreground;
reserving a preselection frame with higher confidence coefficient, and removing a background and a preselection frame with low confidence coefficient;
For each reserved pre-selected frame, acquiring the corresponding confidence score from the classification branch, and sorting the pre-selected frames according to the sequence from high to low to obtain a sorted pre-selected frame list;
initializing an empty list and setting a confidence score threshold;
When the confidence score is greater than the threshold, adding the pre-selected box to the empty list;
calculating the cross ratio of the pre-selected frame and other unprocessed pre-selected frames, wherein the expression is as follows:
;
Wherein, For the i-th pre-selected box,For the j-th pre-selected box,For the intersection ratio of the i-th pre-selected frame and the j-th pre-selected frame,For two pre-selected framesAndIs defined by the intersection area of the two,For two pre-selected framesAndIs a union area of (2);
removing all preselection frames with the cross ratio with the preselection frames being larger than a set threshold value;
Returning to the final reserved list of the pre-selected frames when all the pre-selected frames are processed;
The pre-selected boxes remaining in the list of pre-selected boxes are marked as possible defect areas.
As a preferable scheme of the industrial defect detection method, the method comprises outputting defect information of a workpiece according to a detection result and visualizing, generating a labeling image comprises the following steps,
Reading an original image, and acquiring a pre-selected frame, a classification label and a confidence score of each defect area from the processed pre-selected frame list;
drawing a boundary box and a classification label of each defect area on an original image, and storing the drawn image as a labeling image;
And calculating the severity of the defect according to the confidence score, wherein the expression is as follows:
;
Wherein, As a result of the severity of the defect,As the weight coefficient of the confidence score,For the confidence score of the ith defect,For the area of the i-th pre-selected box,As the original image is to be taken,For the area of the original image,A weight coefficient for the area occupation ratio of the defect area;
Classifying the workpieces into pass and fail according to the severity of the defects;
When the workpieces are classified as qualified, the workpieces are sent into a qualified material box;
When the workpieces are classified as unqualified, conveying the workpieces into an unqualified material box;
And saving the information of the position, the type, the confidence score and the severity of the defect into a database, and saving the marked image into a specified path.
As a preferable scheme of the industrial defect detection method, the method comprises the following steps of introducing an online learning mechanism, updating and optimizing a defect detection model in real time according to the multi-mode data of a newly acquired visible light image, infrared image and ultrasonic image,
Continuously acquiring new multi-mode data of a visible light image, an infrared image and an ultrasonic image, gradually updating the weight of the model by using an online random gradient descent algorithm, and periodically evaluating the performance of the model;
according to the evaluation result, the learning rate and the batch size are adjusted;
And using a transfer learning technology to transfer the existing large-scale labeling data to a new detection task to adapt to new defect types and changes.
In a second aspect, the present invention provides an industrial defect detection system comprising,
The data acquisition and preprocessing module acquires multi-mode data of a visible light image, an infrared image and an ultrasonic image of a workpiece to be detected and performs preprocessing;
the multi-mode data fusion module is used for fusing the multi-mode data of the preprocessed visible light image, the preprocessed infrared image and the preprocessed ultrasonic image by using a multi-mode data fusion algorithm to generate a comprehensive feature map;
The illumination condition analysis and correction module is used for analyzing the illumination condition of the image by using a computer vision technology, and dynamically adjusting the illumination condition to obtain a corrected comprehensive feature map;
The defect detection model construction module is used for constructing a defect detection model, carrying out example segmentation and target detection on the corrected comprehensive feature map, generating a preselected frame and marking a possible defect area;
The detection result visualization module outputs defect information of the workpiece according to the detection result and visualizes the defect information to generate a labeling image;
And the online learning mechanism module is used for introducing an online learning mechanism and updating and optimizing the defect detection model in real time according to the newly acquired multi-mode data of the visible light image, the infrared image and the ultrasonic image.
In a third aspect, an embodiment of the present invention provides a computer device, comprising a memory and a processor, the memory storing a computer program, wherein the computer program when executed by the processor implements any of the steps of the industrial defect detection method according to the first aspect of the present invention.
In a fourth aspect, an embodiment of the present invention provides a computer readable storage medium having a computer program stored thereon, wherein the computer program when executed by a processor implements any of the steps of the industrial defect detection method according to the first aspect of the present invention.
The method has the advantages that multimode data of a workpiece are collected through a high-resolution visible light camera, an infrared camera and an ultrasonic sensor and preprocessed, the preprocessed data are subjected to feature extraction and fusion to generate a comprehensive feature map through a multimode data fusion algorithm, illumination conditions of the comprehensive feature map are analyzed through a computer vision technology, the illumination conditions are dynamically adjusted to generate a corrected comprehensive feature map, a defect detection model is built, example segmentation and target detection are conducted on the corrected comprehensive feature map, a preselected frame is generated, possible defect areas are marked, an online learning mechanism is introduced, the defect detection model is updated and optimized in real time according to the newly collected data, and defects can be accurately identified and positioned through the multimode data fusion, dynamic illumination condition correction and the online learning mechanism, so that the accuracy of industrial defect detection is remarkably improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of the method for detecting industrial defects in example 1.
FIG. 2 is a flow chart of the industrial defect detection system in embodiment 1.
Detailed Description
In order that the above-recited objects, features and advantages of the present invention will become more readily apparent, a more particular description of the invention will be rendered by reference to specific embodiments thereof which are illustrated in the appended drawings.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention, but the present invention may be practiced in other ways other than those described herein, and persons skilled in the art will readily appreciate that the present invention is not limited to the specific embodiments disclosed below.
Further, reference herein to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic can be included in at least one implementation of the invention. The appearances of the phrase "in one embodiment" in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments.
Embodiment 1, referring to fig. 1 and 2, is a first embodiment of the present invention, and this embodiment provides an industrial defect detection method, which includes the following steps:
S1, collecting multi-mode data of a visible light image, an infrared image and an ultrasonic image of a workpiece to be detected, preprocessing the multi-mode data,
The method comprises the steps of shooting a visible light image of a workpiece by using a high-resolution visible light camera, shooting an infrared image of the workpiece by using an infrared camera, capturing temperature distribution information, scanning the workpiece by using an ultrasonic sensor to obtain an ultrasonic image of internal structure information, carrying out graying, denoising and contrast enhancement on the visible light image, carrying out temperature calibration and heat map generation on the infrared image, carrying out signal processing on the ultrasonic image to generate a two-dimensional image, and unifying all preprocessed images to the same size and format.
Still further, graying refers to converting an RGB image into a gray image expressed as:
;
Wherein R, G, B are the pixel values of the red, green and blue channels respectively, Pixel values for a gray scale image;
denoising refers to removing noise in an image by using a median filter, selecting a window size of 3x3, sequencing the neighborhood of each pixel point, and taking the median as a new value of the pixel point;
Contrast enhancement refers to enhancing the contrast of an image using a histogram equalization technique, expressed as:
;
Wherein, In order to enhance the pixel values of the image,Is the horizontal coordinate of the image and,Is the vertical coordinate of the image and,In order to accumulate the histogram,For the total number of pixels of the image,Number of gray levels (typically 256);
The temperature calibration refers to temperature calibration of an infrared image by using a radiation response curve, the heat map generation refers to mapping of calibrated temperature data into a color space by using a pseudo-color mapping technology to generate an intuitive heat map, the signal processing refers to decomposing signals of ultrasonic signals into a plurality of scales by using wavelet transformation, removing high-frequency noise, converting the denoised signals into two-dimensional images by using Fourier transformation, and the unified size and format refers to adjusting all images to the same size by using a bilinear interpolation method and converting all images into a unified format.
S2, fusing the preprocessed visible light image, infrared image and ultrasonic image multi-mode data by using a multi-mode data fusion algorithm to generate a comprehensive feature map,
The method comprises the steps of converting a visible light image, an infrared image and an ultrasonic image into tensor forms, extracting characteristic images of each mode by using a convolutional neural network, adjusting the channel number of each characteristic image into uniform dimensions by using a linear projection layer to generate new characteristic images, splicing each new characteristic image after mapping to form a multi-mode characteristic image, and carrying out characteristic fusion on the multi-mode characteristic images by using a multi-head self-attention mechanism to generate a comprehensive characteristic image.
Furthermore, by converting data of different modalities into tensor form, namely clipping or filling all images to uniform resolution (such as 256x 256), converting gray images into three-channel RGB images, converting the converted images into tensor form, usually a four-dimensional array with the shape of [ B, C, H, W ], wherein B represents batch size, C represents channel number, H and W represent height and width respectively, introducing a multi-head self-attention mechanism to enable the model to focus on important association between different modalities and capture dependency relationship between different areas, specifically, a multi-modal feature map can be mapped to three vector spaces of Query (Query), key (Key) and Value (Value) through linear transformation, then attention weights are calculated, and finally new feature representation is obtained through weighted summation. This process may be repeated multiple times, forming multiple "heads," each focusing on different information.
S3, analyzing the illumination condition of the image by using a computer vision technology, dynamically adjusting the illumination condition to obtain a corrected comprehensive characteristic diagram,
The method comprises the steps of calculating average brightness and average contrast of an image, quantifying global characteristics of the image to obtain global illumination information of the image, capturing relative brightness change between pixels in the global illumination information image and adjacent pixels of the image by using a local binary pattern algorithm, identifying positions of a highlight region and a dark region to generate a binary image, determining current illumination intensity and illumination direction according to the average brightness, the average contrast and the identified positions of the highlight region and the dark region, processing the image by adopting an illumination correction technology to generate a corrected image, and carrying out histogram equalization on the corrected image to generate a corrected comprehensive feature map.
Further, by quantizing global illumination information of an image, it is meant that the image is first converted from RGB color space to gray space, because gray images are more suitable for calculation of brightness and contrast, then the average value of all pixel values of the image is calculated as average brightness, for calculation of average contrast, the brightness difference between each pixel and its surrounding pixels can be calculated first, then the average value of these differences is taken, so that the overall brightness level and the degree of brightness variation of the image can be quantized, and applying a local binary algorithm is meant that each pixel point is compared with its surrounding pixel points, and if the brightness of the center pixel is greater than or equal to the neighborhood pixel, it is marked as 1, otherwise, it is marked as 0. This would generate an 8-bit binary number (for a 3x3 neighborhood). The binary number is converted into a decimal number as a new value for the pixel, forming a binary image. By the method, the highlight area and the dark area in the image can be effectively distinguished, texture information in the image, particularly details such as edges and corner points, can be highlighted, and the method is beneficial to accurately identifying illumination distribution conditions in the image, and illumination correction technologies comprise gamma correction, linear stretching, gaussian filtering and the like. For example, gamma correction may be used to adjust the brightness distribution of an image for an excessively bright or dark image, and Gaussian filtering may be used to smooth out the illumination differences for non-uniform illumination.
S4, constructing a defect detection model, performing example segmentation and target detection on the corrected comprehensive feature map, generating a pre-selected frame and marking a possible defect area,
Inputting the corrected comprehensive feature map into a backbone network, combining the high-level low-resolution feature map with the low-level high-resolution feature map by using the top-down path and the transverse connection of the feature pyramid to generate a multi-scale feature map, sliding a window on each feature map to generate a group of anchor frames with fixed quantity, classifying and carrying out bounding box regression on each anchor frame to judge whether the anchor frame contains a target and adjust the position and the size of the anchor frame, selecting anchor frames with higher scores as candidate areas, using an interesting pooling layer to adjust candidate areas with different sizes to the same size, classifying each candidate area by using the full connection layer to determine which category belongs to, further adjusting the position of the candidate area by the bounding box regression to generate a final pre-selected frame, generating a segmentation mask of each candidate area by segmenting branches, generating a pre-selected frame for each candidate area classified as a foreground, reserving pre-selected frames with higher confidence, removing the pre-selected frames with the background and the low confidence, acquiring the corresponding confidence score from the branch classification and carrying out the pre-selected frames according to the high confidence score and the pre-selected frames with the low confidence score and the pre-selected frame with the high confidence score and the pre-selected frame with the low confidence score and the pre-selected frame being a pre-selected confidence score and the pre-selected frame with the confidence score is calculated with a high confidence score and the pre-selected score is compared with the threshold value and the pre-selected score is calculated to the threshold value:
;
Wherein, For the i-th pre-selected box,For the j-th pre-selected box,For the intersection ratio of the i-th pre-selected frame and the j-th pre-selected frame,For two pre-selected framesAndIs defined by the intersection area of the two,For two pre-selected framesAndIs a union area of (2);
Removing all the preselection frames with the cross-over ratio greater than the set threshold value, returning to the final reserved preselection frame list when all the preselection frames are processed, and marking the reserved preselection frames in the preselection frame list as possible defect areas.
S5, outputting defect information of the workpiece according to the detection result and visualizing, generating an annotation image comprises the following steps,
The method comprises the steps of reading an original image, obtaining a pre-selected frame, a classification label and a confidence coefficient score of each defect area from a processed pre-selected frame list, drawing a boundary frame and a classification label of each defect area on an original image, storing the drawn image as a labeling image, and calculating the severity of the defect according to the confidence coefficient score, wherein the expression is as follows:
;
Wherein, As a result of the severity of the defect,As the weight coefficient of the confidence score,For the confidence score of the ith defect,For the area of the i-th pre-selected box,As the original image is to be taken,For the area of the original image,A weight coefficient for the area occupation ratio of the defect area;
classifying the workpieces into pass and fail according to the severity of the defects, feeding the workpieces into a pass box when the workpieces are classified as pass, feeding the workpieces into a fail box when the workpieces are classified as fail, storing information of the position, type, confidence score and severity of the defects into a database, and storing the marked images into a specified path.
S6, introducing an online learning mechanism, updating and optimizing a defect detection model in real time according to the newly acquired multi-mode data of the visible light image, the infrared image and the ultrasonic image,
Continuously acquiring multi-mode data of a new visible light image, an infrared image and an ultrasonic image, gradually updating the weight of the model by using an online random gradient descent algorithm, periodically evaluating the performance of the model, adjusting the learning rate and the batch size according to the evaluation result, and migrating the existing large-scale labeling data to a new detection task by using a migration learning technology to adapt to new defect types and changes.
The embodiment also provides an industrial defect detection system which comprises a data acquisition and preprocessing module, a multi-mode data fusion module, an on-line learning mechanism module, a defect detection model construction module, a detection result visualization module, a defect detection mechanism module, an on-line learning mechanism module and a defect detection model real-time updating and optimizing module, wherein the data acquisition and preprocessing module acquires multi-mode data of a visible light image, an infrared image and an ultrasonic image of a workpiece to be detected, the multi-mode data of the visible light image, the infrared image and the ultrasonic image are preprocessed, the multi-mode data fusion module fuses the multi-mode data of the preprocessed visible light image, the infrared image and the multi-mode data of the ultrasonic image by using a multi-mode data fusion algorithm to generate an integrated characteristic image, the illumination condition analysis and correction module analyzes the illumination condition of the image by using a computer vision technology, the illumination condition is dynamically adjusted to obtain the corrected integrated characteristic image, the defect detection model construction module constructs a defect detection model, performs example segmentation and object detection on the corrected integrated characteristic image, generates a pre-selected frame and marks a possible defect area, and the detection result visualization module outputs defect information of the workpiece according to the detection result, and generates a labeling image.
The embodiment also provides a computer device, which is suitable for the situation of the industrial defect detection method and comprises a memory and a processor, wherein the memory is used for storing computer executable instructions, and the processor is used for executing the computer executable instructions to realize the industrial defect detection method as proposed in the embodiment.
The computer device may be a terminal comprising a processor, a memory, a communication interface, a display screen and input means connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The communication interface of the computer device is used for carrying out wired or wireless communication with an external terminal, and the wireless mode can be realized through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, can also be keys, a track ball or a touch pad arranged on the shell of the computer equipment, and can also be an external keyboard, a touch pad or a mouse and the like.
The present embodiment also provides a storage medium having stored thereon a computer program which when executed by a processor implements the industrial defect detection method as set forth in the above embodiments, the storage medium may be implemented by any type of volatile or non-volatile Memory device or combination thereof, such as a static random access Memory (Static Random Access Memory, SRAM), an electrically erasable Programmable Read-Only Memory (ELECTRICALLY ERASABLE PROGRAMMABLE READ-Only Memory, EEPROM), an erasable Programmable Read-Only Memory (Erasable Programmable Read Only Memory, EPROM), a Programmable Read-Only Memory (PROM), a Read-Only Memory (ROM), a magnetic Memory, a flash Memory, a magnetic disk, or an optical disk.
In summary, the method comprises the steps of collecting multi-mode data of a workpiece through a high-resolution visible light camera, an infrared camera and an ultrasonic sensor, preprocessing the multi-mode data, carrying out feature extraction and fusion on the preprocessed data by using a multi-mode data fusion algorithm to generate a comprehensive feature map, analyzing illumination conditions of the comprehensive feature map through a computer vision technology, dynamically adjusting the illumination conditions to generate a corrected comprehensive feature map, constructing a defect detection model, carrying out example segmentation and target detection on the corrected comprehensive feature map to generate a pre-selected frame and marking a possible defect area, introducing an online learning mechanism, updating and optimizing the defect detection model in real time according to newly collected data, and accurately identifying and positioning defects through the multi-mode data fusion, dynamic illumination condition correction and the online learning mechanism, so that the accuracy of industrial defect detection is remarkably improved.
Example 2 referring to table 1, experimental simulation data of the industrial defect detection method are given for further verification of the technical scheme of the present invention for the second example of the present invention.
A metal product manufacturing enterprise was selected as the subject of investigation. The industry mainly produces high strength metal parts for the automotive industry, the quality of which directly affects the safety and reliability of the final product. Therefore, extremely high requirements are placed on defect detection during production. In this example, 100 samples of metal parts on the production line were selected, 50 of which were standard defect-free samples, and the other 50 of which were samples known to have various defect types, including cracks, pits, pinholes, and the like. The samples were all the same size, shape and material to ensure consistency of experimental conditions.
Before the experiment starts, a detection platform integrating various sensors is firstly built, and the detection platform comprises a high-resolution visible light camera, an infrared camera and a set of ultrasonic sensors. In addition, special computer vision systems and deep learning frameworks are deployed for performing fusion of multimodal data, feature extraction, defect detection model training, and implementation of online learning mechanisms. In order to ensure the repeatability and accuracy of the experiment, all the devices are subjected to strict calibration, and the accuracy and reliability of the measured data are ensured.
In the experimental process, each sample is firstly photographed by a high-resolution visible light camera to obtain a clear surface image, then the temperature distribution information of the sample is captured by an infrared camera to assist in identifying potential internal defects, and finally an ultrasonic sensor scans the sample to obtain detailed information of the internal structure of the sample. The acquired multi-mode data is preprocessed, and the preprocessing comprises the steps of graying, denoising and contrast enhancement of visible light images, temperature calibration and heat map generation of infrared images, signal processing and two-dimensional image generation of ultrasonic data, and then all preprocessed images are unified to the same size and format so as to facilitate subsequent fusion processing.
And extracting a feature map from the preprocessed image of each mode by using a convolutional neural network, and adjusting the channel number of the feature map to a uniform dimension through a linear projection layer to generate a new feature map. These profiles are then stitched together to form a multi-modal profile. In order to further improve the richness and robustness of the feature representation, a multi-head self-attention mechanism is adopted to perform feature fusion on the multi-mode feature map, and finally a comprehensive feature map is generated.
Next, a depth learning based defect detection model is constructed which is capable of performing instance segmentation and object detection on the corrected integrated feature map, generating pre-selected boxes and marking possible defect areas. The model not only can accurately position the defect, but also can identify the type of the defect, and provides an important basis for subsequent processing.
And outputting defect information of each workpiece according to the detection result of the model, and generating a labeling image in a visual mode. The labeling images intuitively show the specific situation of the defects, provide the severity assessment of the defects, help the factories to quickly make decisions, and reject unqualified products out of the production line.
The details are shown in table 1 below:
Table 1 table of experimental data records
Through further analysis of the experimental data, it is obvious that in terms of defect detection accuracy, the samples reach 98.5%, 97.8% and 98.2% under standard, complex and variable illumination conditions, which are significantly higher than 85.0%, 80.0% and 83.0% of the prior art. This shows that the invention can more effectively identify various defects, and can maintain higher detection precision even under the condition of poor illumination condition.
Secondly, regarding the identification rate of defect types, the identification rate of the method under standard, complex and variable illumination conditions is 97.3%, 96.2% and 96.8%, respectively, while the identification rate of the method in the prior art is only 80.0%, 75.0% and 78.0%, which means that the method can not only find defects, but also accurately distinguish different defect types.
In addition, the method has shorter treatment time, the average treatment time is between 1.2 seconds and 1.4 seconds, and the prior art needs 2.0 seconds to 2.2 seconds, so that the method is more suitable for the application of a high-speed production line, and the production efficiency can be greatly improved.
In addition, the false alarm rate of the method is very low and is only 0.01 to 0.02 percent, which is far lower than 0.05 to 0.06 percent of the prior art. A low false positive rate means that fewer normal products are falsely marked as defective, thereby reducing unnecessary waste and costs.
Finally, the identification rate of the method in the aspect of serious defect identification under standard, complex and variable illumination conditions is close to 100%, while the identification rate in the prior art is not more than 88%, the method can effectively prevent serious defect products from flowing into the market, and the product quality and the user safety are ensured.
In conclusion, the method provided by the invention has obvious technical progress in the field of industrial defect detection, not only improves the accuracy and efficiency of detection, but also reduces the false alarm rate and enhances the identification capability of serious defects.
It should be noted that the above embodiments are only for illustrating the technical solution of the present invention and not for limiting the same, and although the present invention has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solution of the present invention may be modified or substituted without departing from the spirit and scope of the technical solution of the present invention, which is intended to be covered in the scope of the claims of the present invention.

Claims (7)

1.一种工业缺陷检测方法，其特征在于：包括，1. An industrial defect detection method, characterized in that: comprising: 采集待检测工件的可见光图像、红外图像和超声波图像的多模态数据，并进行预处理；Collect multimodal data of visible light images, infrared images and ultrasonic images of the workpiece to be inspected, and perform preprocessing; 使用多模态数据融合算法，将预处理后的可见光图像、红外图像和超声波图像的多模态数据进行融合，生成综合特征图；Using a multimodal data fusion algorithm, the pre-processed multimodal data of visible light images, infrared images and ultrasonic images are fused to generate a comprehensive feature map; 使用计算机视觉技术分析图像的光照条件，动态调整光照条件，得到校正后的综合特征图；Use computer vision technology to analyze the lighting conditions of the image, dynamically adjust the lighting conditions, and obtain a corrected comprehensive feature map; 构建缺陷检测模型，对校正后的综合特征图进行实例分割和目标检测，生成预选框并标记可能的缺陷区域；Build a defect detection model, perform instance segmentation and target detection on the corrected comprehensive feature map, generate pre-selected boxes and mark possible defect areas; 根据检测结果，输出工件的缺陷信息并进行可视化，生成标注图像；According to the inspection results, the defect information of the workpiece is output and visualized to generate annotated images; 引入在线学习机制，根据新采集的可见光图像、红外图像和超声波图像多模态数据实时更新和优化缺陷检测模型；Introducing an online learning mechanism to update and optimize the defect detection model in real time based on newly acquired multimodal data of visible light images, infrared images, and ultrasonic images; 所述使用计算机视觉技术分析图像的光照条件，动态调整光照条件，得到校正后的综合特征图包括以下步骤，The method of using computer vision technology to analyze the lighting conditions of an image, dynamically adjusting the lighting conditions, and obtaining a corrected comprehensive feature map includes the following steps: 通过计算图像的平均亮度和平均对比度对图像的全局特性进行量化，得到图像的全局光照信息；The global characteristics of the image are quantified by calculating the average brightness and average contrast of the image to obtain the global illumination information of the image; 使用局部二值模式算法捕捉全局光照信息图像中像素与其邻近像素之间的相对亮度变化，识别高亮区域和暗区域的位置，生成二值图；A local binary pattern algorithm is used to capture the relative brightness changes between pixels and their neighboring pixels in the global illumination information image, identify the locations of highlight areas and dark areas, and generate a binary image; 根据平均亮度、平均对比度和识别到的高亮和暗区域的位置，确定当前的光照强度和光照方向；Determine the current light intensity and direction based on the average brightness, average contrast, and the positions of the identified highlight and dark areas; 采用光照校正技术对图像进行处理，生成校正后的图像；The image is processed by using illumination correction technology to generate a corrected image; 对校正后的图像进行直方图均衡化，生成校正后的综合特征图；Perform histogram equalization on the corrected image to generate a corrected comprehensive feature map; 构建缺陷检测模型，对校正后的综合特征图进行实例分割和目标检测，生成预选框并标记可能的缺陷区域包括以下步骤，Building a defect detection model, performing instance segmentation and target detection on the corrected comprehensive feature map, generating pre-selected boxes and marking possible defect areas include the following steps: 将校正后的综合特征图输入到主干网络中；Input the corrected comprehensive feature map into the backbone network; 使用特征金字塔的自顶向下路径和横向连接，将高层次的低分辨率特征图与低层次的高分辨率特征图相结合，生成多尺度特征图；Using the top-down path and lateral connections of the feature pyramid, high-level low-resolution feature maps are combined with low-level high-resolution feature maps to generate multi-scale feature maps; 在每个特征图上滑动一个窗口，生成一组固定数量的锚框；Slide a window on each feature map to generate a fixed number of anchor boxes; 对每个锚框进行分类和边界框回归，判断其是否包含目标并调整其位置和大小；Classify and regress each anchor box to determine whether it contains the target and adjust its position and size; 选择得分较高的锚框作为候选区域；Select anchor boxes with higher scores as candidate regions; 使用感兴趣区域池化层将不同大小的候选区域调整到相同的尺寸；Use the region of interest pooling layer to resize candidate regions of different sizes to the same size; 通过全连接层对每个候选区域进行分类，确定属于哪个类别；Classify each candidate region through the fully connected layer to determine which category it belongs to; 通过边界框回归进一步调整候选区域的位置，生成最终的预选框；The position of the candidate area is further adjusted through bounding box regression to generate the final pre-selected box; 通过分割分支生成每个候选区域的分割掩码；Generate a segmentation mask for each candidate region through the segmentation branch; 对每个分类为前景的候选区域生成预选框；Generate a pre-selected box for each candidate region classified as foreground; 保留置信度较高的预选框，去除背景和低置信度的预选框；Keep the pre-selected boxes with higher confidence and remove the background and pre-selected boxes with low confidence; 对于每个保留的预选框，从分类分支获取其对应的置信度分数，并按照从高到低进行排序，得到排序后的预选框列表；For each retained pre-selected box, obtain its corresponding confidence score from the classification branch and sort them from high to low to obtain a sorted pre-selected box list; 初始化一个空列表并设置置信度分数阈值；Initialize an empty list and set the confidence score threshold; 当置信度分数大于阈值时，将预选框加入空列表；When the confidence score is greater than the threshold, the pre-selected box is added to the empty list; 计算预选框与其他未处理的预选框的交并比，表达式为：Calculate the intersection and union ratio of the pre-selected box and other unprocessed pre-selected boxes. The expression is: 其中，Bi为第i个预选框，Bj为第j个预选框，IoU(Bi,Bj)为第i个预选框和第j个预选框的交并比，Area(Bi∩Bj)为两个预选框Bi和Bj的交集面积，Area(Bi∪Bj)为两个预选框Bi和Bj的并集面积；Wherein, Bi is the i-th pre-selected box, Bj is the j-th pre-selected box, IoU( Bi , Bj ) is the intersection-over-union ratio of the i-th pre-selected box and the j-th pre-selected box, Area( Bi∩Bj ) is the intersection area of the two pre-selected boxes Bi and Bj , and Area( Bi∪Bj ) is the union area of the two pre -selected boxes Bi and Bj ; 去除所有与预选框的交并比大于设定的阈值的预选框；Remove all pre-selected boxes whose intersection-over-union ratio with the pre-selected box is greater than the set threshold; 当所有预选框都已处理完毕，返回最终保留的预选框列表；When all pre-selected boxes have been processed, the final list of retained pre-selected boxes is returned; 将预选框列表中保留的预选框标记为可能的缺陷区域；Mark the pre-selected boxes retained in the pre-selected box list as possible defect areas; 根据检测结果，输出工件的缺陷信息并进行可视化，生成标注图像包括以下步骤，According to the inspection results, the defect information of the workpiece is output and visualized. The generation of the annotated image includes the following steps: 读取原始图像，并从处理后的预选框列表中获取每个缺陷区域的预选框、分类标签和置信度分数；Read the original image and obtain the pre-selected box, classification label and confidence score of each defect area from the processed pre-selected box list; 在原图上绘制每个缺陷区域的边界框和分类标签，并将绘制过的图像保存为标注图像；Draw the bounding box and classification label of each defect area on the original image, and save the drawn image as an annotated image; 根据置信度分数计算缺陷的严重程度，表达式为：The severity of the defect is calculated based on the confidence score, and the expression is: 其中，Ci为缺陷的严重程度，α为置信度分数的权重系数，Si为第i个缺陷的置信度分数，Area(Bi)为第i个预选框的面积，Io为原始图像，Area(Io)为原始图像的面积，β为缺陷区域面积占比的权重系数；Where, Ci is the severity of the defect, α is the weight coefficient of the confidence score, Si is the confidence score of the i-th defect, Area( Bi ) is the area of the i-th pre-selected box, Io is the original image, Area( Io ) is the area of the original image, and β is the weight coefficient of the defect area ratio; 根据缺陷的严重程度将工件分类为合格和不合格；Classify workpieces into acceptable and unacceptable based on the severity of defects; 当工件分类为合格时，将工件送入合格物料盒；When the workpiece is classified as qualified, the workpiece is sent to the qualified material box; 当工件分类为不合格时，将工件送入不合格物料盒；When the workpiece is classified as unqualified, the workpiece is sent to the unqualified material box; 将缺陷的位置、类型、置信度分数和严重程度的信息保存到数据库中，并将标注图像保存到指定路径。The information of the defect location, type, confidence score and severity is saved to the database, and the annotated image is saved to the specified path. 2.如权利要求1所述的工业缺陷检测方法，其特征在于：采集待检测工件的可见光图像、红外图像和超声波图像的多模态数据，并进行预处理包括以下步骤，2. The industrial defect detection method according to claim 1, characterized in that: collecting multimodal data of visible light image, infrared image and ultrasonic image of the workpiece to be detected and preprocessing comprises the following steps: 使用高分辨率可见光相机拍摄工件的可见光图像；Using a high-resolution visible light camera to capture a visible light image of the workpiece; 使用红外相机拍摄工件的红外图像，捕捉温度分布信息；Use an infrared camera to take infrared images of the workpiece and capture temperature distribution information; 使用超声波传感器扫描工件，获取内部结构信息的超声波图像；Use ultrasonic sensors to scan workpieces and obtain ultrasonic images of internal structural information; 对可见光图像进行灰度化、去噪和对比度增强，对红外图像进行温度校准和热图生成，对超声波图像进行信号处理，生成二维图像；Grayscale, denoise and contrast enhance visible light images, perform temperature calibration and generate thermal maps for infrared images, perform signal processing on ultrasonic images and generate two-dimensional images; 将所有预处理后的图像统一到相同的尺寸和格式。Unify all preprocessed images to the same size and format. 3.如权利要求2所述的工业缺陷检测方法，其特征在于：使用多模态数据融合算法，将预处理后的可见光图像、红外图像和超声波图像的多模态数据进行融合，生成综合特征图包括以下步骤，3. The industrial defect detection method according to claim 2, characterized in that: using a multimodal data fusion algorithm to fuse the pre-processed multimodal data of the visible light image, infrared image and ultrasonic image to generate a comprehensive feature map comprises the following steps: 将可见光图像、红外图像和超声波图像转换为张量形式；Convert visible light images, infrared images, and ultrasound images into tensor form; 使用卷积神经网络提取每个模态的特征图；Use convolutional neural networks to extract feature maps for each modality; 使用线性投影层将每个特征图的通道数调整为统一的维度，生成新的特征图；Use a linear projection layer to adjust the number of channels of each feature map to a uniform dimension and generate a new feature map; 将映射后的每个新的特征图进行拼接，形成一个多模态特征图；Each new feature map after mapping is concatenated to form a multimodal feature map; 使用多头自注意力机制对多模态特征图进行特征融合，生成综合特征图。A multi-head self-attention mechanism is used to fuse multimodal feature maps to generate a comprehensive feature map. 4.如权利要求3所述的工业缺陷检测方法，其特征在于：引入在线学习机制，根据新采集的可见光图像、红外图像和超声波图像的多模态数据实时更新和优化缺陷检测模型包括以下步骤，4. The industrial defect detection method according to claim 3 is characterized in that: introducing an online learning mechanism to update and optimize the defect detection model in real time according to the newly collected multimodal data of visible light images, infrared images and ultrasonic images comprises the following steps: 持续采集新的可见光图像、红外图像和超声波图像的多模态数据，使用在线随机梯度下降算法逐步更新模型的权重并定期评估模型的性能；Continuously collect new multimodal data of visible light images, infrared images, and ultrasound images, use the online stochastic gradient descent algorithm to gradually update the model weights and regularly evaluate the model performance; 根据评估结果调整学习率和批量大小；Adjust the learning rate and batch size based on the evaluation results; 使用迁移学习技术，将已有的大规模标注数据迁移到新的检测任务中适应新的缺陷类型和变化。Use transfer learning technology to migrate existing large-scale labeled data to new detection tasks to adapt to new defect types and changes. 5.一种工业缺陷检测系统，基于权利要求1～4任一所述的工业缺陷检测方法，其特征在于：包括，5. An industrial defect detection system, based on the industrial defect detection method according to any one of claims 1 to 4, characterized in that it comprises: 数据采集与预处理模块，采集待检测工件的可见光图像、红外图像和超声波图像的多模态数据，并进行预处理；The data acquisition and preprocessing module acquires multimodal data of visible light images, infrared images and ultrasonic images of the workpiece to be inspected, and performs preprocessing; 多模态数据融合模块，使用多模态数据融合算法，将预处理后的可见光图像、红外图像和超声波图像的多模态数据进行融合，生成综合特征图；The multimodal data fusion module uses a multimodal data fusion algorithm to fuse the preprocessed multimodal data of visible light images, infrared images, and ultrasonic images to generate a comprehensive feature map; 光照条件分析与校正模块，使用计算机视觉技术分析图像的光照条件，动态调整光照条件，得到校正后的综合特征图；The lighting condition analysis and correction module uses computer vision technology to analyze the lighting conditions of the image, dynamically adjusts the lighting conditions, and obtains the corrected comprehensive feature map; 缺陷检测模型构建模块，构建缺陷检测模型，对校正后的综合特征图进行实例分割和目标检测，生成预选框并标记可能的缺陷区域；The defect detection model building module builds the defect detection model, performs instance segmentation and target detection on the corrected comprehensive feature map, generates pre-selected boxes and marks possible defect areas; 检测结果可视化模块，根据检测结果，输出工件的缺陷信息并进行可视化，生成标注图像；The inspection result visualization module outputs the defect information of the workpiece and visualizes it according to the inspection results to generate annotated images; 在线学习机制模块，引入在线学习机制，根据新采集的可见光图像、红外图像和超声波图像的多模态数据实时更新和优化缺陷检测模型。The online learning mechanism module introduces an online learning mechanism to update and optimize the defect detection model in real time based on the newly collected multimodal data of visible light images, infrared images and ultrasonic images. 6.一种计算机设备，包括存储器和处理器，所述存储器存储有计算机程序，其特征在于：所述处理器执行所述计算机程序时实现权利要求1～4任一所述的工业缺陷检测方法的步骤。6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein the processor implements the steps of the industrial defect detection method according to any one of claims 1 to 4 when executing the computer program. 7.一种计算机可读存储介质，其上存储有计算机程序，其特征在于：所述计算机程序被处理器执行时实现权利要求1～4任一所述的工业缺陷检测方法的步骤。7. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the industrial defect detection method according to any one of claims 1 to 4.
CN202411435220.7A 2024-10-15 2024-10-15 Industrial defect detection method, system, device and storage medium Active CN118967672B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202411435220.7A CN118967672B (en) 2024-10-15 2024-10-15 Industrial defect detection method, system, device and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202411435220.7A CN118967672B (en) 2024-10-15 2024-10-15 Industrial defect detection method, system, device and storage medium

Publications (2)

Publication Number Publication Date
CN118967672A CN118967672A (en) 2024-11-15
CN118967672B true CN118967672B (en) 2025-04-29

Family

ID=93391551

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202411435220.7A Active CN118967672B (en) 2024-10-15 2024-10-15 Industrial defect detection method, system, device and storage medium

Country Status (1)

Country Link
CN (1) CN118967672B (en)

Families Citing this family (19)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN119198566A (en) * 2024-12-02 2024-12-27 光测工业智能装备(南京)有限公司 Online quality monitoring method and system based on optical multi-spectrum fusion
CN119246537A (en) * 2024-12-03 2025-01-03 宝鸡九安金属材料有限公司 A method and system for predicting the microstructure and properties of titanium plates based on surface defect analysis
CN119322076B (en) * 2024-12-19 2025-03-21 东莞市兆丰精密仪器有限公司 Battery detection method, electronic device and storage medium
CN119444739A (en) * 2025-01-08 2025-02-14 深圳精智达技术股份有限公司 A method, system, electronic device and storage medium for improving Mura defect classification accuracy
CN119880912B (en) * 2025-01-17 2025-11-14 东莞市科运户外用品有限公司 A multimodal nondestructive testing system and method for arrow release devices
CN120064307B (en) * 2025-02-28 2025-12-30 南京林业大学 System and method for detecting and evaluating wood defects based on multi-mode intelligent mode
CN120163798B (en) * 2025-03-11 2026-01-02 山东伟瑞制冷科技有限公司 A Microchannel Aluminum Flat Tube Appearance Inspection Method Based on Image Processing
CN119810094A (en) * 2025-03-12 2025-04-11 格润智能装备(深圳)有限公司 High-precision AOI detection system and method for surface defects of complex industrial products
CN120451094B (en) * 2025-04-27 2025-10-28 东莞市世成通精密电子有限公司 Method, device and system for detecting adhesive quality
CN121158446A (en) * 2025-05-28 2025-12-19 西安量子智能科技有限公司 Online detection method for tearing of conveyor belt
CN120234769B (en) * 2025-06-03 2025-08-15 上海外高桥造船有限公司 Ship final assembly construction safety quality real-time monitoring method based on AI image recognition
CN120953663B (en) * 2025-07-24 2026-02-10 江西省坤宇电子有限公司 A machine learning-based method and system for quality inspection of multilayer circuit boards
CN121190433A (en) * 2025-09-22 2025-12-23 山东艾琳智能科技有限公司 A Deep Learning-Based Method for Detecting Optical Surface Defect Data
CN120890989B (en) * 2025-09-30 2026-03-24 中交华南勘察测绘科技有限公司 AI-based prefabricated member multi-mode vision quality intelligent detection system and method
CN120953262A (en) * 2025-10-11 2025-11-14 山东大学 A 3D Printing Anomaly Alarm Method and System Based on Dual-Channel Detection and Decision Fusion
CN120948613B (en) * 2025-10-17 2026-01-30 华电电力科学研究院有限公司 Journal defect detection method and device for steam turbine rotor and electronic equipment
CN121030684B (en) * 2025-10-27 2026-01-27 徐州康翔精密制造有限公司 Automobile part quality online detection method based on multi-sensor fusion
CN121527094A (en) * 2026-01-16 2026-02-13 南昌市众拓制衣有限公司 A Real-Time Defect Detection Method and System for Textile Fabrics Based on Multimodal Feature Fusion
CN121708018A (en) * 2026-02-13 2026-03-20 南京奥看信息科技有限公司 A multi-modal AI collaborative industrial visual defect detection system

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114092389A (en) * 2021-09-13 2022-02-25 浙江大学 Glass panel surface defect detection method based on small sample learning
CN117664990A (en) * 2023-12-13 2024-03-08 苏州锐璟科技有限公司 Intelligent PCBA appearance defect detection method

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP4266246A1 (en) * 2022-04-22 2023-10-25 Imec VZW Automated defect classification and detection
CN118212220A (en) * 2024-04-15 2024-06-18 东北农业大学 A computer vision inspection method for product defect detection
CN118527377B (en) * 2024-07-26 2024-11-22 南京比逊弥特智能科技有限公司 Debris detection and removal device and method
CN118711000B (en) * 2024-08-30 2024-11-15 山东科技大学 Bearing surface defect detection method and system based on improved YOLOv10

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114092389A (en) * 2021-09-13 2022-02-25 浙江大学 Glass panel surface defect detection method based on small sample learning
CN117664990A (en) * 2023-12-13 2024-03-08 苏州锐璟科技有限公司 Intelligent PCBA appearance defect detection method

Also Published As

Publication number Publication date
CN118967672A (en) 2024-11-15

Similar Documents

Publication Publication Date Title
CN118967672B (en) Industrial defect detection method, system, device and storage medium
CN111862064B (en) Silver wire surface flaw identification method based on deep learning
KR102254773B1 (en) Automatic decision and classification system for each defects of building components using image information, and method for the same
CN113592828B (en) Nondestructive testing method and system based on industrial endoscope
KR102166458B1 (en) Defect inspection method and apparatus using image segmentation based on artificial neural network
JP5546317B2 (en) Visual inspection device, visual inspection discriminator generation device, visual inspection discriminator generation method, and visual inspection discriminator generation computer program
Eshkevari et al. Automatic dimensional defect detection for glass vials based on machine vision: A heuristic segmentation method
KR20230124713A (en) Fault detection methods, devices and systems
CN111915704A (en) Apple hierarchical identification method based on deep learning
CN117132540A (en) A PCB circuit board defect post-processing method based on segmentation model
CN116245882A (en) Circuit board electronic element detection method and device and computer equipment
CN111915572A (en) A deep learning-based self-adaptive gear pitting quantitative detection system and method
CN110264444A (en) Damage detection method and device based on weak segmentation
TW202034421A (en) Color filter inspection device, inspection device, color filter inspection method, and inspection method
CN117974601A (en) Silicon wafer surface defect detection method and system based on template matching
CN118505708B (en) Product repair system based on image reconstruction technology
CN119273992A (en) Mobile phone screen glass defect detection method based on improved YOLOv8
CN115995023A (en) Defect detection method, device, electronic device, computer readable storage medium and product detection method
CN120259191A (en) Semiconductor wafer defect detection method, device, equipment and storage medium
CN119131006A (en) A ceramic defect intelligent detection method, system, device and storage medium based on deep learning
CN117495846B (en) Image detection method, device, electronic device and storage medium
CN115861220A (en) Cold-rolled strip steel surface defect detection method and system based on improved SSD algorithm
CN120102589A (en) Surface defect detection method and related equipment for touch screen
CN119130912A (en) Strip steel surface defect detection method and system based on depth map
CN116797602A (en) Surface defect identification method and device for industrial product detection

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant