CN119130912A

CN119130912A - Strip steel surface defect detection method and system based on depth map

Info

Publication number: CN119130912A
Application number: CN202411085738.2A
Authority: CN
Inventors: 张昌明; 赵强; 朱丽婷
Original assignee: Shanghai Baosight Software Co Ltd
Current assignee: Shanghai Baosight Software Co Ltd
Priority date: 2024-08-08
Filing date: 2024-08-08
Publication date: 2024-12-13

Abstract

The present invention provides a strip surface defect detection method and system based on a depth map, comprising: step S1: data collection, obtaining point cloud data of the steel plate surface and generating a point cloud data set; step S2: preprocessing the point cloud data set after obtaining it, and performing defect detection; step S3: determining the category of the defect image; step S4: displaying the steel plate defects respectively in combination with the detection results. The present invention can display the detection results to the user in an intuitive manner through the human-computer interaction module provided, including the depth, size, proportion and category conclusion of the defect, so that the user can easily understand the defect situation on the steel plate surface.

Description

Strip steel surface defect detection method and system based on depth map

Technical Field

The invention relates to the technical field of steel plate surface defect detection, in particular to a depth map-based strip steel surface defect detection method and system.

Background

The traditional defect detection based on the 2D image depends on the contrast of the detected object, the detection effect is easily interfered by background textures, and the surface defect with depth is difficult to accurately detect. In recent years, 3D machine vision technology has been rapidly developed, and in contrast to 2D technology, 3D technology can provide a depth Z value of an object in addition to a brightness value of the object.

The patent document CN116539628A discloses a steel plate surface defect detection system based on a 3D vision technology, which comprises data acquisition ends, a central server and a monitoring device, wherein the data acquisition ends are arranged on the upper side and the lower side of the surface of a steel plate to be detected and are used for acquiring texture images of the surface of the steel plate at a specific angle to generate point cloud data, the central server is in communication connection with the data acquisition ends, the monitoring device comprises a man-machine interaction unit and a control unit, the control unit is used for controlling data acquisition work of the data acquisition ends, the man-machine interaction unit is used for controlling the control unit, and quality defect detection results obtained after the central server processes the point cloud data are displayed. But the invention does not relate to a depth map-based strip steel surface defect detection algorithm.

Patent document CN115564720a discloses a strip steel defect detection method and system based on three-dimensional scanning and machine vision, which comprises the steps of respectively acquiring two-dimensional image information and three-dimensional contour information for strip steel at the same position by using a camera and three-dimensional scanning equipment, acquiring boundaries of the strip steel according to the two-dimensional image information and the three-dimensional contour information, respectively utilizing a preset machine learning model according to preset features to identify and classify defects in an image, positioning the acquired defects based on the boundaries, outputting the identified defect images, and sending an alarm. But the invention does not relate to a depth map-based strip steel surface defect detection algorithm.

Disclosure of Invention

Aiming at the defects in the prior art, the invention aims to provide a strip steel surface defect detection method and system based on a depth map.

The invention provides a strip steel surface defect detection method based on a depth map, which comprises the following steps:

step S1, data acquisition is carried out, and a point cloud data set is generated after point cloud data on the surface of a steel plate are acquired;

s2, preprocessing after acquiring a point cloud data set, and performing defect detection;

s3, judging the type of the defect image;

and S4, respectively displaying the defects of the steel plates according to the detection results.

Preferably, in the step S1:

The method comprises the steps that data acquisition ends are arranged on the upper side and the lower side of the surface of the steel plate, a 3D laser scanning camera is adjusted to face the surface of the steel plate to different angles, the camera acquires two-dimensional image information and three-dimensional point cloud information of the surface of the steel plate in real time, and the 3D laser scanning camera acquires point cloud data of the surface of the steel plate in real time and then transmits the data to the defect detection module to generate a point cloud data set.

Preferably, in said step S2:

Processing the point cloud data set after acquiring the point cloud data set comprises the following steps:

S2.1, the software reads the image output by each camera and splices the images into a complete steel plate point cloud image according to the calibration parameters;

Step S2.2, preprocessing the point cloud data set, namely performing direct-pass filtering, line-by-line outlier filtering and smooth filtering on the point cloud data, normalizing a plurality of filtering operations on a whole-graph Z-mean value, filtering noise points of the point cloud data, and restoring the point cloud data in the real form of the steel plate;

And S2.3, carrying out gradient calculation on the point cloud data in the height direction, calculating the relative depth of each pixel point relative to the peripheral area in real time, extracting a suspicious area through binarization, and marking the defect point according to a preset threshold value.

Preferably, the calculating the relative depth process comprises:

calculating the average height of the peripheral pixels by taking the current pixel as the center, and obtaining the relative depth of the pixel point relative to the peripheral region by calculating the difference value between the height of the current pixel and the average height of the peripheral pixels;

The process of extracting suspicious regions includes:

Setting a threshold value based on the gray value of the pixel, setting a pixel with a pixel value smaller than the threshold value as black 0, setting a pixel with a pixel value larger than or equal to the threshold value as white 255, and converting the image into a binary image with only black and white colors;

S2.4, performing a clustering algorithm on the defect points to form a complete defect, determining the position of the defect, calculating the position of the defect from the head of the strip steel, and obtaining the position of the defect from the head of the strip steel by knowing the positioning of the head of the strip steel in the image and the positioning of the defect in the image for positioning the longitudinal position of the defect of a next machine set;

S2.5, filtering the detected pseudo defects by using a defect filtering rule;

And S2.6, converting the point cloud data height value into an RGB color value according to the color table, obtaining a color strip steel image, and transmitting the color strip steel image to a defect classification module.

Preferably, in said step S3:

Providing a deep learning classification model, judging the category of the defect image through the deep learning classification model, training the deep learning classification model based on a neural network algorithm and a large-scale training data set, and automatically learning and extracting the characteristics in the defect image and classifying the characteristics into different categories;

the training process for deep learning defect classification using ResNet model is performed as follows:

preparing a data set containing defect type labels, wherein the data set comprises input samples and corresponding labels, adjusting images in the data set to be 100x100 in size, and performing preprocessing operation;

Model construction, namely using PyTorch deep learning framework, importing ResNet model and modifying the final full connection layer to adapt to the category number of classification tasks, wherein the structure of ResNet model comprises a plurality of convolution layers, pooling layers and full connection layers;

The loss function definition, namely measuring the difference between the model prediction result and the real label by using a cross entropy loss function;

The data loading and training, namely dividing the prepared data set into a training set and a verification set, inputting the data into a model by using a data loader, training the model by using the training set, updating parameters of the model by calculating a loss function and using an optimization algorithm, and dynamically adjusting the learning rate by using a learning rate scheduler in the training process;

Evaluating the performance of the model by using the verification set, calculating indexes such as classification accuracy and the like, and stopping training if the performance of the model meets the requirement, otherwise, continuously adjusting the super parameters of the model or increasing the training round number;

and (3) model storage and deployment, namely after training, storing the model on a disk, and using and deploying in practical application.

The invention provides a strip steel surface defect detection system based on a depth map, which comprises the following steps:

The module M1 is used for acquiring data, acquiring point cloud data on the surface of the steel plate and generating a point cloud data set;

the module M2 is used for preprocessing after acquiring the point cloud data set and carrying out defect detection;

A module M3, judging the type of the defect image;

And a module M4, respectively displaying the defects of the steel plates according to the detection results.

Preferably, in said module M1:

Preferably, in said module M2:

Software reads the image output by each camera and splices the image into a complete steel plate point cloud image according to the calibration parameters;

the module M2.2 is used for preprocessing the point cloud data set, namely performing direct-pass filtering, line-by-line outlier filtering and smooth filtering on the point cloud data, normalizing a plurality of filtering operations on the Z average value of the whole graph, filtering noise points of the point cloud data, and restoring the point cloud data in the real form of the steel plate;

And a module M2.3, carrying out gradient calculation on the point cloud data in the height direction, calculating the relative depth of each pixel point relative to the peripheral area in real time, extracting a suspicious area through binarization, and marking the defect point according to a preset threshold value.

Preferably, the calculating the relative depth process comprises:

The process of extracting suspicious regions includes:

A module M2.4, performing a clustering algorithm on the defect points to form a complete defect, determining the position of the defect, calculating the position of the defect from the head of the strip steel, and obtaining the position of the defect from the head of the strip steel by knowing the positioning of the head of the strip steel in the image and the positioning of the defect in the image for positioning the longitudinal position of the defect of a next machine set;

a module M2.5, filtering the detected pseudo defects by using a defect filtering rule;

And a module M2.6, converting the point cloud data height value into RGB color value according to the color table, obtaining a color strip steel image and transmitting the color strip steel image to a defect classification module.

Preferably, in said module M3:

Compared with the prior art, the invention has the following beneficial effects:

1. According to the data acquisition module provided by the invention, the 3D laser scanning camera is adopted for data acquisition, so that the point cloud data on the surface of the steel plate can be acquired in real time, and the defects on the surface of the steel plate can be accurately detected through processing the data set. While providing a luminance map and a height map of the defect, the user can easily distinguish the type and severity of the defect by viewing the 2D and 3D images.

2. The invention provides a set of deep learning classification model through the defect classification module, which can classify and judge the detected defects, has certain universality and flexibility, and can be trained and adjusted according to actual requirements so as to adapt to different types of defect detection tasks. Therefore, the nature and the severity of the defects are better known, a user is helped to make more accurate judgment and decision, automatic defect detection can be realized, the requirement of manual operation is reduced, and the detection efficiency and accuracy are improved. Meanwhile, the accuracy and the robustness of the model are verified and optimized, and the model can be effectively applied to the actual industrial environment.

3. According to the invention, the detection result can be displayed to the user in an intuitive manner through the man-machine interaction module, and the user can conveniently know the defect condition of the surface of the steel plate, wherein the detection result comprises the depth, the size, the duty ratio and the category conclusion of the defect.

Drawings

Other features, objects and advantages of the present invention will become more apparent upon reading of the detailed description of non-limiting embodiments, given with reference to the accompanying drawings in which:

FIG. 1 is a schematic diagram of two-dimensional image information and three-dimensional contour information of a defect provided by the invention;

FIG. 2 is a flow chart of a strip steel surface defect detection algorithm based on a depth map;

FIG. 3 is a schematic diagram of a defect detection algorithm according to the present invention;

FIG. 4 is a diagram showing an exemplary defect classification result provided by the present invention, which is respectively pits, cracks, skin lifting, and embossing from left to right;

Fig. 5 is a schematic block diagram of a strip steel surface defect detection algorithm based on a depth map.

Fig. 6 is a schematic diagram of a man-machine interaction module display interface provided by the present invention.

Detailed Description

The present invention will be described in detail with reference to specific examples. The following examples will assist those skilled in the art in further understanding the present invention, but are not intended to limit the invention in any way. It should be noted that variations and modifications could be made by those skilled in the art without departing from the inventive concept. These are all within the scope of the present invention.

Example 1:

The development and application of the invention fills the short plate of the traditional 2D imaging technology and brings unique progress in industrial object detection. Especially for the surface state of the steel plate, the complexity of the texture of the surface can be overcome, the detection of depth defects is focused to achieve quantitative measurement, and the quality and the safety of steel production are improved. The 3D vision system is used for directly measuring the height data of the object surface, so that the sensitivity to the texture of the object surface and the change of ambient light is reduced, and the stability of a detection result is improved. Meanwhile, based on the 3D characteristics of the defects, the method can remarkably improve the accuracy of defect classification. In addition, the method can provide a brightness map and a height map of the defect, so that a user can easily judge the type and the severity of the defect by observing the 2D image and the 3D image. Through the advantages, the 3D steel plate surface defect detection method can detect and evaluate the defect condition of the steel plate surface more accurately, and improves the quality inspection efficiency and accuracy.

According to the method for detecting the surface defects of the strip steel based on the depth map, which is provided by the invention, as shown in fig. 1-6, the method comprises the following steps:

Specifically, in the step S1:

Specifically, in the step S2:

Specifically, the calculation of the relative depth process includes:

The process of extracting suspicious regions includes:

S2.5, filtering the detected pseudo defects by using a defect filtering rule;

S3, judging the type of the defect image;

Specifically, in the step S3:

Example 2:

example 2 is a preferable example of example 1 to more specifically explain the present invention.

The invention also provides a strip steel surface defect detection system based on the depth map, which can be realized by executing the flow steps of the strip steel surface defect detection method based on the depth map, namely, a person skilled in the art can understand the strip steel surface defect detection method based on the depth map as a preferable implementation mode of the strip steel surface defect detection system based on the depth map.

specifically, in the module M1:

specifically, in the module M2:

Specifically, the calculation of the relative depth process includes:

The process of extracting suspicious regions includes:

A module M3, judging the type of the defect image;

Specifically, in the module M3:

Example 3:

example 3 is a preferable example of example 1 to more specifically explain the present invention.

The invention provides a strip steel surface defect detection algorithm based on a depth map, which comprises the following steps:

And S1, a data acquisition module, wherein the data acquisition ends are arranged on the upper side and the lower side of the surface of the steel plate, the 3D laser scanning camera is adjusted to be opposite to the surface of the steel plate by lenses with different angles, and the camera acquires two-dimensional image information and three-dimensional point cloud information of the surface of the steel plate in real time. The 3D laser scanning camera acquires point cloud data on the surface of the steel plate in real time and then transmits the data to the defect detection module to generate a point cloud data set;

s2, a defect detection module, wherein the data acquisition module processes the point cloud data set after acquiring the point cloud data set, and the defect detection module comprises the following steps:

and S2.1, the software reads the image output by each camera and splices the images into a complete steel plate point cloud image according to the calibration parameters.

And S2.2, preprocessing the point cloud data set, namely performing direct-pass filtering, line-by-line outlier filtering and smoothing filtering on the point cloud data, normalizing a plurality of filtering operations on the Z average value of the whole graph, filtering noise points of the point cloud data, and restoring the point cloud data in the real form of the steel plate.

And S2.3, carrying out gradient calculation on the point cloud data in the height direction, calculating the relative depth of each pixel point relative to the peripheral area in real time, and extracting the suspicious area through binarization. And marking the defect points according to a preset threshold value.

The process of calculating the relative depth includes:

And calculating the average height of the peripheral pixels by taking the current pixel as the center, and obtaining the relative depth of the pixel point relative to the peripheral region by calculating the difference value between the height of the current pixel and the average height of the peripheral pixels.

The process of extracting suspicious regions includes:

Based on the gradation value of the pixel, a threshold value is set, a pixel whose pixel value is smaller than the threshold value is set to 0 (black), and a pixel whose pixel value is equal to or larger than the threshold value is set to 255 (white), thereby converting the image into a binary image of only two colors of black and white.

The binarized formula can be expressed as:

if pixel_value<threshold:

binary_value=0

else:

binary_value=255

where pixel_value represents a pixel value in the original image, and threshold represents a set threshold.

And S2.4, performing a clustering algorithm on the defect points to form a complete defect. The defect location is determined and the defect distance head location is calculated. The location of the defect from the head is known from the location of the head in the image and the location of the defect in the image.

And S2.5, filtering the detected pseudo defects by using a defect filtering rule.

S2.6, converting the point cloud data height value into RGB color value according to the color table, obtaining a color strip steel image and transmitting the color strip steel image to a defect classification module.

And S3, displaying defect gray scale information by a 2D image, and displaying defect depth information by a 3D image. The 2D map and the 3D map synchronize pixel level display. As shown in fig. 1.

The defect type is judged by a defect classification module, and a man-machine module displays depth, size, duty ratio and type conclusion including the defect for users to use. The display interface is shown in fig. 6, new complement.

And the defect classification module is used for providing a set of deep learning classification models, and the models can judge the types of the defect images.

The deep learning model is trained based on an advanced neural network algorithm and a large-scale training data set, and can automatically learn and extract the characteristics in the defect image and divide the characteristics into different categories.

data preparation, namely preparing a data set containing defect type labels, wherein the data set comprises input samples and corresponding labels. The images in the dataset are resized to a size of 100x100 and subjected to a preprocessing operation, such as normalization.

Model construction, using PyTorch and other deep learning frameworks, importing ResNet model and modifying the final fully connected layer to adapt to the category number of classification tasks. The ResNet model structure contains multiple convolution layers, pooling layers, and full-join layers.

The loss function definition is that a cross entropy loss function is used to measure the gap between the model predictive result and the real label.

Data loading and training, namely dividing the prepared data set into a training set and a verification set, and inputting the data into a model by using a data loader. The model is then trained using the training set, and parameters of the model are updated by calculating the loss function and using an optimization algorithm. A learning rate scheduler is used in the training process to dynamically adjust the learning rate.

And (3) model evaluation, namely evaluating the performance of the model by using a verification set, and calculating indexes such as classification accuracy and the like. If the performance of the model meets the requirement, training can be stopped, otherwise, the super-parameters of the model can be continuously adjusted or the training round number can be increased.

Model preservation and deployment, namely after training is completed, the model is preserved on a disk so as to be used and deployed in practical application.

And S4, a man-machine interaction module is used for respectively displaying the depth of the steel plate defect, the length and width of the defect, the defect duty ratio and the defect category conclusion according to the detection result.

The method can be popularized and applied to 3D detection of the surface quality of the re-rolled strip steel and 3D detection of the surface quality of medium plates, continuous casting blanks and steel pipes.

Those skilled in the art will appreciate that the invention provides a system and its individual devices, modules, units, etc. that can be implemented entirely by logic programming of method steps, in addition to being implemented as pure computer readable program code, in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers, etc. Therefore, the system and the devices, modules and units thereof provided by the invention can be regarded as a hardware component, and the devices, modules and units for realizing various functions included in the system can be regarded as structures in the hardware component, and the devices, modules and units for realizing various functions can be regarded as structures in the hardware component as well as software modules for realizing the method.

The foregoing describes specific embodiments of the present application. It is to be understood that the application is not limited to the particular embodiments described above, and that various changes or modifications may be made by those skilled in the art within the scope of the appended claims without affecting the spirit of the application. The embodiments of the application and the features of the embodiments may be combined with each other arbitrarily without conflict.

Claims

1. The strip steel surface defect detection method based on the depth map is characterized by comprising the following steps of:

s3, judging the type of the defect image;

2. The depth map-based strip steel surface defect detection method according to claim 1, wherein in the step S1:

3. The depth map-based strip steel surface defect detection method according to claim 1, wherein in the step S2:

4. The depth map-based strip steel surface defect detection method according to claim 3, wherein:

the process of calculating the relative depth includes:

The process of extracting suspicious regions includes:

S2.5, filtering the detected pseudo defects by using a defect filtering rule;

5. The depth map-based strip steel surface defect detection method according to claim 1, wherein in the step S3:

6. A depth map-based strip steel surface defect detection system, comprising:

A module M3, judging the type of the defect image;

7. The depth map based strip steel surface defect detection system of claim 6, wherein in the module M1:

8. The depth map based strip steel surface defect detection system of claim 6, wherein in the module M2:

9. The depth map based strip steel surface defect detection system of claim 8, wherein:

the process of calculating the relative depth includes:

The process of extracting suspicious regions includes:

10. The depth map based strip steel surface defect detection system of claim 6, wherein in the module M3: