CN120612502B

CN120612502B - Machine vision defect real-time detection and classification method and system based on deep learning

Info

Publication number: CN120612502B
Application number: CN202511100385.3A
Authority: CN
Inventors: 郁标; 许琪
Original assignee: Nanjing Ailong Automation Equipment Co ltd
Current assignee: Nanjing Ailong Automation Equipment Co ltd
Priority date: 2025-08-07
Filing date: 2025-08-07
Publication date: 2025-12-16
Anticipated expiration: 2045-08-07
Also published as: CN120612502A

Abstract

This invention provides a deep learning-based method and system for real-time defect detection and classification in machine vision, relating to the field of machine vision detection. The method includes determining region enhancement weights by calculating local entropy and gradient direction consistency, performing adaptive enhancement by region; establishing a feature transfer sequence and progressively fusing features; generating and correcting the probability distribution map of defect regions; and constructing a dynamic decision matrix to calculate a comprehensive score for defect classification. This method can improve the accuracy of defect detection in complex backgrounds, reduce false positives and false negatives, and achieve real-time defect localization and accurate classification.

Description

Machine vision defect real-time detection and classification method and system based on deep learning

Technical Field

The invention relates to a machine vision detection technology, in particular to a machine vision defect real-time detection and classification method and system based on deep learning.

Background

The surface defect detection of industrial products is a key link of product quality control, and the traditional manual detection mode has the problems of low efficiency, poor consistency and the like. With the development of deep learning technology, an automatic detection method based on machine vision gradually becomes a research hot spot, but under the complex background and illumination conditions, the detection precision and instantaneity still face challenges.

The existing deep learning detection method mainly has the problems that image features are extracted insufficiently, so that detection accuracy is low, defective areas are positioned inaccurately, classification effects are affected, defect evaluation standards are single, quality control requirements of different products are difficult to meet, detection process calculation is complex, and real-time requirements are difficult to achieve.

There is a need for a detection method capable of adaptively enhancing image features, optimizing feature fusion paths, accurately positioning defect regions and realizing hierarchical evaluation, so as to improve detection accuracy and instantaneity and realize intelligent detection and classification of industrial product surface defects.

Disclosure of Invention

The embodiment of the invention provides a machine vision defect real-time detection and classification method and system based on deep learning, which can solve the problems in the prior art.

In a first aspect of the embodiments of the present invention, a machine vision defect real-time detection and classification method based on deep learning is provided, including:

obtaining image data of the surface of an industrial product, calculating the local entropy value and gradient direction consistency of each pixel point in the image data, determining regional enhancement weight according to the local entropy value and gradient direction consistency, and carrying out regional self-adaptive enhancement on the image data based on the regional enhancement weight to obtain enhanced image data;

extracting structural features of the enhanced image data, establishing a feature transfer sequence, calculating a correlation matrix among features according to the feature transfer sequence, establishing a feature optimization path based on the correlation matrix, and carrying out progressive fusion on the features along the feature optimization path to obtain fused feature data;

Generating a probability distribution map of the defect area according to the fusion characteristic data, correcting the probability distribution map of the defect area by combining the area enhancement weight, determining the defect area, extracting topological structure characteristics of the defect area, establishing defect characteristic description, carrying out defect classification according to the defect characteristic description, and outputting defect types and confidence scores;

And constructing a dynamic decision matrix based on the region enhancement weight and the confidence score, calculating the comprehensive score of each defect region in the dynamic decision matrix, grading the defects according to the comprehensive score, generating a defect severity quantization index, and outputting a defect detection result.

In an alternative embodiment of the present invention,

Obtaining image data of the surface of an industrial product, calculating local entropy value and gradient direction consistency of each pixel point in the image data, determining regional enhancement weight according to the local entropy value and the gradient direction consistency, carrying out regional self-adaptive enhancement on the image data based on the regional enhancement weight, and obtaining enhanced image data comprises the following steps:

Acquiring image data of the surface of an industrial product, and carrying out region segmentation on the image data along a preset direction sequence to obtain a plurality of region sub-images;

Constructing an associated intensity matrix of each pixel point and a neighborhood thereof in each regional sub-image, calculating a local entropy value of each pixel point based on the associated intensity matrix, extracting gradient vectors of each pixel point and projecting the gradient vectors in different directions, calculating gradient direction consistency of each pixel point according to projection components, and combining the local entropy value and the gradient direction consistency to generate regional characteristic data;

extracting extreme points of the consistency of the local entropy value and the gradient direction of each pixel point in the regional characteristic data, clustering the extreme points meeting the preset extreme value range to obtain regional seed points, establishing topological connection relation among the regional seed points, calculating characteristic transfer coefficients, and generating a regional growth path;

Transmitting the local entropy value and the gradient direction consistency of the region seed points outwards along the region growing path, calculating an attenuation factor according to the distance from the pixel points to the region seed points, taking the product of the local entropy value, the gradient direction consistency and the attenuation factor as region enhancement weight, and weighting the region characteristic data based on the region enhancement weight to obtain an enhancement coefficient;

and introducing characteristic transfer coefficients to adjust the enhancement intensity at the region boundary, and recombining the enhanced region sub-images to obtain enhanced image data.

In an alternative embodiment of the present invention,

Extracting structural features of the enhanced image data, and establishing a feature transfer sequence comprises:

constructing a pyramid layered structure for the enhanced image data, dividing overlapping subareas in each layered structure, extracting local gradient distribution of the overlapping subareas, and adjusting filter direction parameters based on the local gradient distribution;

Extracting structural features of the overlapped subareas by adopting an adjusted filter, establishing corresponding mapping of the structural features in the overlapped areas of adjacent layers, determining a feature fusion sequence according to the corresponding mapping, and combining the structural features of each layer according to the feature fusion sequence to obtain multi-layer structural features;

And constructing a hierarchical transfer tree based on the feature fusion sequence, and establishing a feature transfer link in the hierarchical transfer tree, wherein the feature transfer link comprises transfer nodes between adjacent features, analyzing spatial structure distribution and direction consistency between the adjacent transfer nodes, determining node transfer priority according to the spatial structure distribution and the direction consistency, and generating a feature transfer sequence based on the node transfer priority.

In an alternative embodiment of the present invention,

Calculating a relevance matrix among the features according to the feature transfer sequence, establishing a feature optimization path based on the relevance matrix, and carrying out progressive fusion on the features along the feature optimization path to obtain fused feature data, wherein the step of obtaining the fused feature data comprises the following steps:

calculating characteristic association strength between adjacent transmission nodes of a characteristic transmission sequence, and transmitting the characteristic association strength step by step in the characteristic transmission sequence to construct an association degree matrix;

Determining initial transfer nodes in the association degree matrix, expanding next transfer nodes step by step from the initial transfer nodes based on the characteristic association strength, and establishing transfer connection among the nodes, wherein the transfer connection is connected in series according to the sequence from high to low of the characteristic association strength to form a characteristic optimization path;

And starting from the initial transfer node along the feature optimization path, carrying out weighted fusion on the features of the current transfer node and the features of the next transfer node, wherein the weighted fusion weight is determined by the feature association strength of the corresponding transfer connection, taking the fused features as new current node features, continuing to fuse with the features of the next transfer node, and finishing the fusion of the features of all the transfer nodes by progressive progression to obtain final fused feature data.

In an alternative embodiment of the present invention,

Generating a probability distribution map of the defect area according to the fusion characteristic data, correcting the probability distribution map of the defect area by combining the area enhancement weight, and determining the defect area comprises the following steps:

Calculating local statistics of the fusion characteristic data, constructing a self-adaptive kernel function based on the local statistics, decomposing the self-adaptive kernel function by adopting orthogonal transformation to obtain a basic function set, and carrying out convolution calculation on the basic function set and the fusion characteristic data to obtain a characteristic response diagram;

Performing gradient diffusion on the characteristic response map to obtain a density flow field, extracting gradient tracks in the density flow field, calculating regional clustering characteristics according to convergence points and convergence directions of the gradient tracks, and generating a defect regional probability distribution map based on the regional clustering characteristics;

Performing self-adaptive weighted fusion on the region enhancement weight and the defect region probability distribution map to obtain a corrected probability map, extracting a boundary point set of the corrected probability map, calculating multi-order moment characteristics of the boundary point set, constructing a level set function on the corrected probability map, self-adaptively adjusting the evolution speed of the level set function based on the multi-order moment characteristics, and obtaining the defect region through zero horizontal plane segmentation of the level set function.

In an alternative embodiment of the present invention,

Extracting topological structure characteristics of the defect area, establishing defect characteristic description, classifying defects according to the defect characteristic description, and outputting defect types and confidence scores, wherein the steps comprise:

Extracting a closed contour of a defect area by adopting a boundary tracking algorithm with a self-adaptive threshold value, carrying out noise reduction and smoothing on the closed contour to obtain a continuous boundary, establishing a contour point sequence, constructing a distance transformation field based on the contour point sequence, extracting a central line from the distance transformation field by a gradient descent method, and constructing a central point sequence;

mapping the contour point sequence and the center point sequence to the same coordinate space to construct a topological structure of a defect area, extracting branch nodes and crossing points in the topological structure, calculating a connection relation between adjacent nodes, constructing a topological tree according to the connection relation, and extracting structural features of the topological tree as topological structure features of the defect area;

carrying out normalization processing on each feature component of the topological structure feature, calculating correlation coefficients among the feature components after normalization, constructing a feature correlation matrix, calculating distinguishing capability of each feature component based on the feature correlation matrix, determining feature weights according to the distinguishing capability, carrying out weighted combination on the feature components by adopting the feature weights, and establishing defect feature description;

and matching the defect feature description with a preset multi-type defect sample, calculating the distance and similarity between feature descriptions, constructing a feature matching matrix, analyzing the matching degree of the sample to be detected and various types of defect samples in the feature matching matrix, determining the type with the highest matching degree as the defect type, and taking the corresponding matching degree as a confidence score.

In an alternative embodiment of the present invention,

Constructing a dynamic decision matrix based on the region enhancement weight and the confidence score, calculating a comprehensive score of each defect region in the dynamic decision matrix, grading the defects according to the comprehensive scores, generating defect severity quantization indexes, and outputting defect detection results, wherein the steps comprise:

Calculating the distribution uniformity of the region enhancement weights and the fluctuation range of the confidence scores, taking the distribution uniformity and the fluctuation range as scoring weight coefficients, and carrying out weighted fusion on the region enhancement weights and the confidence scores to generate a scoring matrix;

Extracting a change rule of the region enhancement weight along with the position to obtain a region change feature, extracting stability of the confidence score to obtain a reliability feature, calculating dimension weight of a scoring matrix based on the region change feature and the reliability feature, and constructing a dynamic decision matrix according to the dimension weight;

calculating the relative importance degree among the dimensions in the dynamic decision matrix, generating an importance weight vector, and multiplying the importance weight vector by a scoring matrix to obtain the comprehensive score of each defect area;

and constructing a grading evaluation standard for the comprehensive scores, calculating severity distribution according to the grading evaluation standard, converting the severity distribution into severity quantization indexes by adopting a membership function, classifying defect levels according to the severity quantization indexes, and outputting detection results.

In a second aspect of the embodiments of the present invention, there is provided a machine vision defect real-time detection and classification system based on deep learning, including:

the first unit is used for acquiring image data of the surface of the industrial product, calculating the local entropy value and gradient direction consistency of each pixel point in the image data, determining regional enhancement weight according to the local entropy value and gradient direction consistency, and carrying out regional self-adaptive enhancement on the image data based on the regional enhancement weight to obtain enhanced image data;

The second unit is used for extracting structural features of the enhanced image data, establishing a feature transfer sequence, calculating a correlation matrix among features according to the feature transfer sequence, establishing a feature optimization path based on the correlation matrix, and carrying out progressive fusion on the features along the feature optimization path to obtain fused feature data;

The third unit is used for generating a probability distribution map of the defect area according to the fusion characteristic data, correcting the probability distribution map of the defect area by combining the area enhancement weight, determining the defect area, extracting topological structure characteristics of the defect area, establishing defect characteristic description, carrying out defect classification according to the defect characteristic description, and outputting defect types and confidence scores;

and a fourth unit, configured to construct a dynamic decision matrix based on the region enhancement weights and the confidence scores, calculate a comprehensive score for each defect region in the dynamic decision matrix, rank the defects according to the comprehensive scores, generate a defect severity quantization index, and output a defect detection result.

In a third aspect of an embodiment of the present invention, there is provided an electronic device including:

A processor;

A memory for storing processor-executable instructions;

wherein the processor is configured to invoke the instructions stored in the memory to perform the method described previously.

In a fourth aspect of embodiments of the present invention, there is provided a computer readable storage medium having stored thereon computer program instructions which, when executed by a processor, implement the method as described above.

In the embodiment, the regional enhancement weight is calculated through the consistency of the local entropy value and the gradient direction, so that the regional self-adaptive enhancement is realized, the contrast and the definition of the defect region are improved, the problem that the low contrast and the fuzzy defect are difficult to identify by the traditional method is solved, and the accuracy and the sensitivity of defect detection are remarkably improved. The feature transfer sequence and the association degree matrix are established, progressive fusion of the features is realized, multi-scale structural information of the defects is effectively reserved, accurate classification of different types of defects is realized through topological structure feature extraction and defect feature description, and robustness of the system to defect identification under a complex background is improved. The dynamic decision matrix is constructed, the comprehensive score of the defect area is calculated, the quantitative classification of the defect severity is realized, an accurate reference basis is provided for product quality control, meanwhile, the characteristic extraction and fusion processes are optimized, the calculation complexity is reduced, the real-time performance of the system is ensured, and the method is suitable for the online detection requirement of a high-speed industrial production line.

Drawings

FIG. 1 is a flow chart of a machine vision defect real-time detection and classification method based on deep learning according to an embodiment of the invention;

FIG. 2 is a schematic diagram showing the performance of a defect detection method according to an embodiment of the present invention;

FIG. 3 is a schematic diagram of an intelligent defect detection and comprehensive rating process according to the present embodiment.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

The technical scheme of the invention is described in detail below by specific examples. The following embodiments may be combined with each other, and some embodiments may not be repeated for the same or similar concepts or processes.

Fig. 1 is a flow chart of a machine vision defect real-time detection and classification method based on deep learning according to an embodiment of the invention, as shown in fig. 1, the method includes:

In an alternative embodiment, image data of a surface of an industrial product is obtained, local entropy and gradient direction consistency of each pixel point in the image data are calculated, region enhancement weights are determined according to the local entropy and gradient direction consistency, the image data are subjected to region-division self-adaptive enhancement based on the region enhancement weights, and the enhanced image data comprise:

In this embodiment, the image data of the surface of the industrial product is obtained by imaging the surface of the product with a high resolution industrial camera. During acquisition, the camera is perpendicular to the surface of the product, so that illumination uniformity is maintained, and strong light reflection and shadow interference are avoided. The image acquisition resolution is set to 2048×1536 pixels and saved as an 8-bit gray scale image format. Image preprocessing includes denoising and contrast correction, ensuring the original image quality.

When the acquired image data is subjected to region segmentation along a preset direction sequence, the preset direction sequence comprises horizontal, vertical and diagonal directions. Sliding windows with different sizes are arranged in each direction by adopting a multi-scale method, and the window sizes are respectively 16×16, 32×32 and 64×64 pixels. The step length of the sliding window is set to be 1/4 of the window size, so that the adjacent areas are ensured to be overlapped sufficiently. The entire image is scanned through a sliding window to obtain a plurality of regional sub-images.

When an association intensity matrix of the pixel points and the neighborhood thereof is constructed in each regional sub-image, the current pixel point is taken as the center, and the neighborhood range is set as a 5 multiplied by 5 pixel region. And calculating the gray scale difference between the central pixel and each pixel in the neighborhood, wherein the smaller the difference is, the larger the association strength is. The correlation strength is determined by an inverse function of the gray level difference, and the correlation strength takes a maximum value of 1 when the gray level difference is 0, and drops below 0.1 when the gray level difference exceeds the threshold value 20.

When the local entropy value of each pixel point is calculated based on the correlation intensity matrix, the correlation intensity matrix is normalized, so that the sum of all elements is 1, and probability distribution is formed. For the pixel points with coordinates (i, j) in the image, counting the distribution situation of gray values in a 5×5 neighborhood, and calculating the entropy value of the distribution as the local entropy value of the pixel points. The higher the local entropy value, the higher the texture complexity of the region, and the lower the local entropy value, the higher the region smoothness.

When the gradient vector of each pixel point is extracted, the Sobel operator is used for calculating the gradient components in the horizontal direction and the vertical direction respectively. For the pixel point with the coordinates of (i, j) in the image, the horizontal gradient is obtained by calculating the gray level difference between the pixel points of (i, j+1) and (i, j-1), and the vertical gradient is obtained by calculating the gray level difference between the pixel points of (i+1, j) and (i-1, j). Combining the horizontal and vertical gradient components, the magnitude and direction of the gradient vector is obtained.

The gradient vector was projected in 8 different directions with an angle of 45 degrees apart. For each pixel point, the distribution condition of the gradient directions of the pixel points in the 11×11 neighborhood is counted. If the gradient component ratio in one direction exceeds 60%, the region is considered to have higher directional consistency, and if the gradient components in each direction are uniformly distributed, the directional consistency is lower. The gradient direction consistency value range is [0,1], and the closer the value is to 1, the more consistent the gradient direction in the region is.

When the local entropy value and the gradient direction consistency are combined to generate the regional characteristic data, the regional characteristic data and the regional characteristic data are weighted and averaged, and the weight ratio is 6:4. For detail rich regions, the local entropy weight is increased to 0.7, and for edge and texture regions, the gradient direction consistency weight is increased to 0.6. The combined characteristic data reflects the complexity and structural features of the region. When the extreme points of the consistency of the local entropy value of each pixel point in the regional characteristic data and the gradient direction are extracted, the local entropy value threshold value is set to be 0.75, and the gradient direction consistency threshold value is set to be 0.8. Searching for local maximum or minimum in 7×7 window, and marking the point meeting the condition as candidate extremum point. The preset extremum range is the point of local entropy value between [0.7,0.9] or gradient direction consistency between [0.75,0.95 ].

When the extreme points meeting the preset extreme value range are clustered, a density clustering algorithm is adopted, and the clustering radius is set to be 15 pixels. When the euclidean distance between two extreme points is smaller than the clustering radius and the characteristic difference is smaller than 0.2, they are classified into the same class. Each cluster center point is determined as a region seed point. For a 2048×1536 image, typically 30-50 region seed points are available.

When establishing the topological connection relation between the seed points of the areas, establishing the connection between the seed points with the distance smaller than 50 pixels. The connection strength is proportional to the characteristic similarity between the seed points and inversely proportional to the distance. And calculating characteristic transfer coefficients based on the connection strength, wherein the transfer coefficient range is [0.3,0.9], and the transfer coefficient between seed points with high similarity is close to 0.9. When the region growing path is generated, the region growing path is gradually expanded from the seed point with the highest characteristic value along the direction of the maximum connection strength. In the growth process, the characteristic change rate from the current point to the next point is not more than 0.15, so that the smoothness of a growth path is ensured. The growth path is recorded as a series of coordinate point sequences for subsequent property transfer. When the consistency of the local entropy value and the gradient direction of the region seed point is transferred outwards along the region growing path, the maximum transfer radius is set to be 100 pixels. As distance increases, the transmission strength gradually decreases. The attenuation factor adopts an exponential attenuation function, and is 0.9 when the distance is 10 pixels, 0.5 when the distance is 50 pixels, and 0.2 when the distance is 100 pixels.

The regional enhancement weight is calculated by the product of the local entropy value, the gradient direction consistency and the attenuation factor. For high texture regions, the local entropy contribution weight is 0.7, and for regions with significant directionality, the gradient direction consistency contribution weight is 0.6. The enhancement weight range is [0.1,1.0], and the larger the weight is, the more remarkable the enhancement effect is. And when the nonlinear mapping function is constructed according to the enhancement coefficient to enhance the regional sub-image, adopting S-shaped curve mapping. For the area with high enhancement coefficient, the slope of the curve is larger to enhance the contrast, and for the area with low enhancement coefficient, the slope of the curve is smaller to keep the original characteristics. The parameters of the mapping function are dynamically adjusted according to the regional characteristics, so that the enhancement effect is ensured to be self-adaptive to different regions. When the characteristic transfer coefficient adjustment enhancement intensity is introduced at the region boundary, the boundary width is set to 10 pixels. The enhancement intensity of the boundary region is a weighted average of the enhancement intensities of the adjacent regions, and the weight is determined by the characteristic transfer coefficient. When the characteristic transfer coefficient is 0.7, the enhancement strength of the boundary region inherits the characteristics of the higher enhancement region at a ratio of 70%, and inherits the characteristics of the lower enhancement region at a ratio of 30%.

And when the enhanced regional sub-images are recombined, fusing the overlapped regions by adopting a weighted average method. The weight is inversely proportional to the distance from the pixel point to the center of the region, so that the natural transition of the recombined image is ensured. The resolution of the finally output enhanced image is the same as that of the input image, but the details are clearer, the contrast ratio is more reasonable, and the method is suitable for the characteristic requirements of different areas.

According to the technical scheme, the regional seed points are generated by introducing the local entropy value and gradient direction consistency joint modeling and combining the extreme point clustering with the topological connection, the regional seed points are generated by accurately describing the fine grain texture and edge direction characteristics in the image, and the information is enhanced in a diffusion mode to the regional boundary through characteristic transmission and attenuation control, so that the contrast ratio and the structural definition of the image in a defect region are effectively improved. The self-adaptive enhancement is realized by constructing a nonlinear mapping function, so that the problem that the traditional image enhancement cuts the whole image once is avoided, the recognition sensitivity to low-contrast, small-area or boundary fuzzy defects is improved, a high-quality data base is laid for subsequent feature extraction and defect positioning, and the detection accuracy and robustness are improved.

In an alternative embodiment, extracting structural features of the enhanced image data, establishing a feature transfer sequence includes:

Illustratively, when constructing a pyramid hierarchy from enhanced image data, the original image is downsampled to a scale of 1/2, typically forming a pyramid structure that gradually increases from top to bottom. For example, for a 1024×1024 pixel image, a pyramid structure containing 5 layers can be constructed, where layer 0 is the original size 1024×1024, layer 1 is 512×512, layer 2 is 256×256, layer 3 is 128×128, and layer 4 is 64×64. In dividing overlapping sub-regions in each layer structure, an overlapping proportion of 30% to 50% is typically used, for example, for a layer 1 512×512 image, it may be divided into a plurality of 256×256 sub-regions, overlapping 128 pixels between adjacent sub-regions, ensuring feature continuity and smooth transitions.

When the local gradient distribution of the overlapped subareas is extracted, the gradient amplitude and direction of the pixels are calculated for each subarea. In particular, the gradient of each pixel point can be calculated using a difference operator in the horizontal and vertical directions. For example, for a pixel at a position (x, y), a horizontal direction gradient Gx and a vertical direction gradient Gy thereof are calculated, and then a gradient magnitude G and a direction θ are synthesized. The gradient magnitude G is equal to the square root of the sum of the squares of the horizontal and vertical gradients, and the gradient direction θ is equal to the arctangent of the vertical and horizontal gradients. The distribution of gradient directions in the subarea is counted, 360 degrees are generally divided into 8 or 16 direction intervals, and the accumulated intensity of the gradient in each direction interval is calculated to obtain a gradient direction histogram of the subarea.

When the direction parameters of the filter are adjusted based on the local gradient distribution, the direction sensitivity of the filter is adjusted according to the main direction in the gradient direction histogram. For example, if the gradient of a sub-region is distributed mainly in 45 degree and 225 degree directions, the filter parameters are adjusted to be more sensitive to these directions. Specifically, the center direction of the directional filter may be adjusted to θmax according to the direction angle θmax corresponding to the highest peak in the gradient direction histogram, and the direction bandwidth of the filter may be adjusted according to the degree of dispersion of the histogram. When the gradient direction distribution is more concentrated, a narrower direction bandwidth (such as 15 degrees) is set, and when the direction distribution is more dispersed, a wider direction bandwidth (such as 45 degrees) is set.

When the structure features of the overlapped subareas are extracted by adopting the adjusted filters, the adjusted directional filters are applied to the corresponding subareas, and the structure information such as edges, textures and the like is extracted. For example, for a sub-region with a dominant direction of 45 degrees, a convolution operation is performed using a direction filter with a center direction of 45 degrees and a bandwidth of 30 degrees, and the structural feature in the direction is extracted. For each sub-region, structural features in multiple directions may be extracted, forming feature vectors. In practical application, 8 or 16 directions of features can be extracted, and each direction corresponds to a feature channel.

When the structural features are mapped in the overlapping areas of adjacent layers, the dimensional change between different layers needs to be considered. For example, a 256×256 sub-region in layer 1 corresponds to a 512×512 region in layer 0, and a mapping is established by the positional correspondence. In specific implementation, a bilinear interpolation method can be adopted to up-sample the features with smaller scale to larger scale or down-sample the features with larger scale to smaller scale, then a similarity matrix between two feature graphs is calculated, and cosine similarity or Euclidean distance can be used for similarity calculation. And establishing a corresponding mapping relation for the feature point pairs with similarity exceeding a threshold value (such as 0.7).

And when the feature fusion sequence is determined according to the corresponding mapping, analyzing the correlation strength among the features of different levels. In specific implementation, the average similarity of the feature mapping point pairs of the adjacent layers can be calculated, and the higher the similarity is, the higher the fusion priority is. For example, if the average mapping similarity between the 1 st layer and the 0 th layer is 0.85 and the average mapping similarity between the 2 nd layer and the 1 st layer is 0.76, the feature fusion sequence is that the 1 st layer and the 0 th layer are fused preferentially, and then fused with the 2 nd layer.

And combining the structural features of each layer according to the feature fusion sequence, and adopting a weighted fusion mode when the multi-layer structural features are obtained. For example, if the determined fusion order is that layer 1 and layer 0 are fused first, and the weights are 0.4 and 0.6, respectively, then the fused feature F01 is equal to 0.4 times the layer 1 feature F1 plus 0.6 times the layer 0 feature F0. And then fusing F01 and the layer 2 feature F2 according to weights of 0.7 and 0.3 to obtain F012, and the like to finish feature fusion of all layers.

When the hierarchical transfer tree is constructed based on the feature fusion sequence, the feature nodes of each layer are connected according to the fusion sequence to form a tree structure. For example, if the fusion order is to fuse the 0 th layer and the 1 st layer first and fuse the 2 nd layer later, the root node of the tree is F012 after fusion, the lower nodes are F01 and F2, and the lower nodes are F0 and F1. When the feature transfer link is established in the hierarchical transfer tree, the feature nodes of adjacent hierarchies form a transfer path through the connecting edges.

And when the spatial structure distribution and the direction consistency between adjacent transfer nodes are analyzed, calculating the spatial distribution similarity and the direction consistency score of the node characteristics. Spatial distribution similarity can be measured by the absolute value of the difference between the spatial distribution entropies of the feature map, the closer the entropic values are, the more similar the distribution is. The direction consistency is measured by calculating the angle between the main directions of the two nodes, and the smaller the angle is, the higher the consistency is. The node delivery priority is determined based on spatial distribution similarity and directional consistency, and a weighted sum approach may be used, for example, the priority score is equal to 0.6 times spatial distribution similarity plus 0.4 times directional consistency score.

When the characteristic transfer sequence is generated based on the node transfer priority, all the transfer nodes are ordered from high to low according to the priority to form the transfer sequence. For example, if the priority score of the node pair (F0, F1) is 0.92 and the priority score of the node pair (F01, F2) is 0.85, the transfer sequence is to transfer F0 to F1 first and then transfer F01 to F2. The finally generated feature transfer sequence can guide the effective fusion and transfer of the multi-level features, and the accuracy and the robustness of the extraction of the image structural features are improved.

In the embodiment, the pyramid layered structure is constructed, overlapped subarea division is introduced, the direction parameters of the filter are dynamically adjusted by combining local gradient distribution, so that the structural features are extracted to have more direction adaptability and multiscale sensing capability, the multi-level structural features are constructed through mapping and fusion of adjacent level structural features, the expression capability of complex textures and fine deformation is enhanced, spatial structure distribution and direction consistency analysis between a level transmission tree and nodes is introduced, the feature transmission process is ensured to have structural continuity and semantic relevance, and therefore a feature transmission sequence with layering and priority control is generated, and stable and high-quality structural information support is provided for subsequent feature fusion and defect area identification.

In an alternative embodiment, calculating a relevance matrix between features according to a feature transfer sequence, establishing a feature optimization path based on the relevance matrix, and performing progressive fusion on the features along the feature optimization path to obtain fused feature data, wherein the step of obtaining the fused feature data comprises the following steps:

Illustratively, when calculating feature correlation strengths between adjacent delivery nodes of a feature delivery sequence, the feature correlation strengths are determined by calculating the similarity of their feature vectors for any two adjacent nodes. The similarity calculation uses a cosine distance measure, taking the product of the dot product of the two eigenvectors divided by the product of the respective modulo lengths. When the extracted two node features are 128-dimensional vectors, if the included angle of the two vectors is 30 degrees, the calculated association strength is about 0.866, and if the included angle is 60 degrees, the association strength is about 0.5. The association strength threshold is typically set to 0.75, and pairs of nodes below this threshold are considered weak associations.

Progressive transmission of feature correlation strengths in feature transmission sequences employs a cumulative decay mechanism. From the start node, the association strength decays by a proportion every time a node passes as it passes back along the transmission sequence. The attenuation coefficient is dynamically adjusted according to the distance between the nodes, and the attenuation coefficient of the adjacent node with the distance of 1 is usually 0.9, the attenuation coefficient of the node with the distance of 2 is 0.8, and the attenuation coefficient of the node with the distance of 3 is 0.7. Specifically, if the association strength of the node a and the node b is 0.85 and the association strength of the node b and the node c is 0.8, the association strength of the node a transmitted to the node c through the node b is 0.85×0.8×0.9=0.612.

When the association degree matrix is constructed, the direct association strength and the indirect association strength are calculated for each pair of nodes in the characteristic transfer sequence, and the maximum value of the direct association strength and the indirect association strength is taken as the final association strength. Each element in the matrix represents the strength of association between different nodes, forming a symmetric matrix. A sequence of 5 transfer nodes may generate an association matrix of 0.89 for the first and second nodes, 0.76 for the second and third nodes, 0.82 for the third and fourth nodes, 0.65 for the fourth and fifth nodes, 0.68 for the first and third nodes, 0.62 for the second and fourth nodes, 0.53 for the third and fifth nodes, 0.56 for the first and fourth nodes, 0.48 for the second and fifth nodes, and 0.36 for the first and fifth nodes.

When the initial transfer node is determined in the association degree matrix, the node with the highest average association strength in the matrix is selected. The specific calculation method is to sum elements of each row in the matrix and select a node corresponding to the largest row. If the sum of the association strengths of the second node and all other nodes is 2.75 and is higher than the sum of the association strengths of other nodes, the second node is selected as the initial transfer node.

And when the next transfer node is expanded step by step from the initial transfer node based on the characteristic association strength, selecting the non-access node with the highest association strength with the current node by adopting a greedy strategy. If the initial node is the second node, and the association strength with the first, third, fourth and fifth nodes is 0.89, 0.76, 0.62 and 0.48 respectively, the next transfer node selects the first node. And updating the accessed node set every time a new node is selected, and continuing to select the non-accessed node with the highest association strength until all the nodes are accessed.

When the transfer connection between the nodes is established, the selected node sequence is converted into the directed connection. If the selected node sequence is [ second, first, third, fourth and fifth ], the established transfer connection is from second to first, from first to third, from third to fourth and from fourth to fifth, and each connection is marked with a corresponding association strength. The transmission connection is connected in series according to the sequence from high to low of the characteristic association strength, and a characteristic optimization path is formed.

When weighting fusion is carried out along the characteristic optimization path, the characteristic optimization path is gradually fused with the next node from the initial transfer node. The fusion weight is determined by the characteristic association strength of the corresponding transfer connection. If the association strength between the second node and the first node is 0.89, the fusion weight is set to be 0.89 to 0.11, that is, the characteristics of the second node account for 89% and the characteristics of the first node account for 11%. The fusion operation uses a weighted average, and for each feature dimension, the fused value is equal to the weighted average of the two node feature values.

In specific implementation, for two 128-dimensional feature vectors, the association strength is 0.89, and the fusion result is calculated in a manner that fusion feature=0.89×second node feature+0.11×first node feature. And taking the fused characteristic as a new current node characteristic, and continuing to fuse with the characteristic of the third node of the next transfer node. If the association strength between the fusion feature and the third node is 0.76, the fusion weight is 0.76 to 0.24, and the fusion result=0.76×the fusion feature+0.24×the third node feature.

The progressive fusion process specifically comprises the following steps of firstly taking the characteristic of a second node of an initial transmission node as a current characteristic, calculating the association strength of the current characteristic and the characteristic of a first node of a next transmission node to be 0.89, setting a fusion weight according to the association strength, carrying out weighted average on the current characteristic and the characteristic of the next node to obtain a new current characteristic, calculating the association strength of the new current characteristic and the characteristic of a third node of the next transmission node to be 0.76, continuing to set the fusion weight according to the association strength, carrying out weighted average, updating the current characteristic, and the like until all transmission nodes participate in fusion to obtain final fusion characteristic data. By means of the progressive fusion mode, the features of all nodes are gradually fused along the feature optimization path, and finally fusion feature data containing complete information is obtained. The final fusion characteristics can effectively express the characteristics of various types of defects on the surface of the product, and the dimension of the fusion characteristics is the same as that of the original characteristics, but the fusion characteristics contain multi-scale and multi-directional comprehensive information, so that the accuracy of defect detection and classification is remarkably improved.

In the embodiment, the method comprises the steps of constructing a relevance matrix which accurately reflects similarity and dependency relationship between features by quantifying feature association strength between adjacent nodes in a feature transfer sequence, avoiding the problem of unordered or equal-weight superposition in traditional feature fusion, constructing an orderly and clear-priority feature optimization path based on the relevance matrix, enabling the feature fusion process to follow the principle of strong association priority, realizing progressive fusion strategy from local to whole and from shallow to deep, integrating the features of each transfer node step by step in a dynamic weighting mode, retaining key structure information, inhibiting redundant interference, improving discrimination and stability of fusion features, and providing higher-quality feature expression for defect identification and classification.

In an alternative embodiment, generating a probability distribution map of the defect region according to the fused feature data, correcting the probability distribution map of the defect region in combination with the region enhancement weight, and determining the defect region includes:

In practicing the present invention, local statistics of fused feature data are first calculated, including calculating the mean, variance, skewness, and kurtosis of each local region in a sliding window manner in feature space. In the present embodiment, the above statistics are calculated for the feature values within each window using a local window of 7×7 pixels with a sliding step size of 1. For example, for a region, local statistics with a mean of 0.65, a variance of 0.23, a skewness of 0.08, and a kurtosis of 3.12 may be obtained.

And constructing a self-adaptive kernel function based on the calculated local statistic, wherein the kernel function can self-adaptively adjust the shape and the scale according to the local characteristic distribution condition. In this embodiment, the bandwidth matrix of the kernel function is determined by the covariance matrix of the local area, and when the feature variance is large, the expandability of the kernel function in the direction is stronger. For example, for a complex texture region, the kernel exhibits anisotropy with a major axis direction coincident with the texture direction and a minor axis to major axis ratio of 1:3. And decomposing the self-adaptive kernel function by adopting orthogonal transformation to obtain a group of basis functions. In this embodiment, the kernel function is decomposed into 5 orthogonal basis functions, each capturing characteristic information of different directions and scales, using a singular value decomposition technique. The weights of these basis functions are 0.45, 0.25, 0.15, 0.10 and 0.05, respectively, representing the importance of each in the feature expression.

And carrying out convolution calculation on the basis function set and the fusion characteristic data to obtain a characteristic response diagram. In this embodiment, the convolution operation is accelerated by using a fast fourier transform, a response chart corresponding to each basis function is calculated, and weighted combination is performed according to the weights of the basis functions, so as to form a final characteristic response chart. For example, for a 512 x 512 pixel image, a corresponding size feature response map is generated in which the response value of the defect region is significantly higher than the background region, typically the response value range of the defect region is [0.75,0.95], and the background region response value range is [0.05,0.30].

And carrying out gradient diffusion on the characteristic response graph to obtain a density flow field. In this embodiment, an anisotropic diffusion method is used, the diffusion time is 10 iteration steps, the diffusion coefficient is adaptively adjusted according to the gradient amplitude, the diffusion of the region with large gradient is weak, and the diffusion of the region with small gradient is strong. For example, the gradient threshold is set to 0.2, the diffusion coefficient is 0.8 when the gradient magnitude is less than the threshold, otherwise 0.2× (1-gradient magnitude). Extracting gradient tracks from the density flow field, and tracking the gradient tracks along the gradient direction by randomly selecting a starting point in the flow field. In this embodiment, 1000 random starting points are selected, the maximum tracking step number of each track is 100, the step length is 0.5 pixel, and tracking is stopped when the distance between track points is less than 0.1 pixel or exceeds the image boundary.

And calculating regional cluster characteristics according to the convergence points and the convergence directions of the gradient tracks. In this embodiment, a density clustering algorithm is used to classify the convergence points with a distance less than 5 pixels into one class, and calculate the center position, size, shape and direction of each cluster. For example, for a typical defect, 3-5 convergence zones may be formed, each zone containing 50-200 convergence points. And generating a defect region probability distribution map based on the region clustering features. And calculating the distance and the direction similarity of each pixel point to the nearest convergence region, and converting the distance and the direction similarity into probability values. For example, the distance function uses a gaussian decay model with a standard deviation of 15 pixels, a probability of 1 when the point-to-nearest convergence region distance is 0, and a probability of 0.1 when the distance is 30 pixels.

A region enhanced weight map is constructed that takes into account edge information, texture complexity and local contrast of the original image. In this embodiment, the edge weights are extracted by Canny operator, the threshold is set to 50 and 150, the texture complexity is calculated by local entropy, the window size is 9×9, and the contrast is calculated by dividing the local standard deviation by the local mean. The final weight is the weighted sum of the three, and the weights are respectively 0.4, 0.35 and 0.25.

And carrying out self-adaptive weighted fusion on the region enhancement weight and the probability distribution map of the defect region to obtain a corrected probability map. In this embodiment, the fusion formula is that the modified probability=original probability× (1+α×enhancement weight), where α is an adaptive coefficient, determined statistically from the global probability distribution, and typically has a value of 0.4. And extracting a boundary point set from the corrected probability map, and extracting the boundary point set after binarizing the boundary point set through a probability threshold (such as 0.65). The multi-order moment characteristics of the boundary point set are calculated, including centroid, principal axis direction, eccentricity and the like. For example, for a typical crack defect, its second order center moment may show a major axis direction of 32 degrees with a 5:1 ratio of major to minor axes. A level set function is constructed for the modified probability map, with the initial level set function being the modified probability map minus a threshold (e.g., 0.5). The evolution speed of the level set function is adaptively adjusted based on the multi-order moment characteristics, the evolution speed is faster in the direction of the main shaft, and the speed is slower in the direction perpendicular to the main shaft. In the present embodiment, the velocity coefficient in the main axis direction is 1.5, and the vertical direction is 0.8. And obtaining a final defect area through zero-level segmentation of the level set function. The level set evolution adopts an iteration mode, the maximum iteration number is 100, and the level set evolution is stopped when the boundary change of continuous 5 iterations is smaller than 0.5 pixel. For example, for a defect with an area of about 500 square pixels, the convergence is achieved after about 35 iterations, resulting in an accurate defect area profile.

According to the technical scheme, the sensitivity of the feature response map to different scales and morphological defects is improved by calculating local statistics on the fusion feature data and constructing a self-adaptive kernel function, a gradient diffusion generation density flow field is introduced, gradient tracks are extracted, the space aggregation trend of defect features can be effectively identified, the accuracy of defect region positioning is improved, the region enhancement weight is combined for carrying out weighted correction, the self-adaptive enhancement of defects with fuzzy boundaries or lower contrast is realized, the level set function evolution process is guided through the multi-order moment features, the defect segmentation boundary is finer and smoother, the high-precision extraction of defect regions is finally realized, and the robustness and segmentation precision of detection are enhanced.

Fig. 2 is a schematic diagram showing performance comparison of different defect detection methods according to an embodiment of the present invention, and as shown in fig. 2, the diagram shows detection accuracy of three different defect detection algorithms (adaptive kernel function method, conventional convolutional neural network and level set segmentation method) under four different defect types (crack detection, surface dishing, texture anomaly and complex background) scenes. As apparent from graph data, the adaptive kernel function method is best in all test scenes, the accuracy is between 85.2% and 92.7%, and the traditional convolutional neural network is weaker in overall performance. Particularly, under the complex background condition, the accuracy (85.2%) of the self-adaptive kernel function method is far higher than that of the traditional method (71.8% and 68.5%), and the advantages of the algorithm in processing complex scenes are reflected.

In an alternative embodiment, extracting topological structure features of the defect area, establishing defect feature descriptions, classifying defects according to the defect feature descriptions, and outputting defect types and confidence scores comprises:

Illustratively, an adaptive threshold boundary tracking algorithm is employed in extracting the closed contour of the defect region. The algorithm firstly carries out gray scale processing on the image, a threshold value is determined by calculating the gray scale mean value and standard deviation of the local area, and the threshold value is set to be the mean value minus 0.5 times of the standard deviation. For example, for an 8-bit gray scale image, if the gray scale average value of a region is 120 and the standard deviation is 40, the threshold value of the region is 120-0.5x40=100. And carrying out binarization processing on the image based on the threshold value to obtain an initial boundary of the defect area. The boundary tracking process uses the 8-connectivity criteria, starting from the first boundary point in the upper left corner, exploring the neighboring points in a clockwise direction, recording the coordinates of all boundary points until returning to the starting point to complete the extraction of the closed contour.

When the noise reduction and smoothing treatment are carried out on the extracted closed contour, a sliding window average method is adopted, and the window size is set to 7 pixel points. For each contour point, taking 3 points before and after the contour point, and calculating the coordinate average value of the 7 points as the new position of the current point. If the original contour point coordinates are [ (10, 15), (12, 15), (14, 16), (16, 17), (18, 18), (20, 20), (22, 21) ], the new coordinates of the intermediate point (16, 17) after the smoothing process are (16,17.4). After all points are processed, a smooth continuous sequence of boundary and contour points is obtained.

When constructing the distance transform field, the shortest distance from each point in the image to the boundary is calculated. The specific implementation uses a two-dimensional array to store the distance value, the initialized boundary point distance is 0, and the non-boundary point distance is infinity. The distance value of each point is calculated by two scans (forward and reverse). The forward direction scans from the upper left corner to the lower right corner, the reverse direction scans from the lower right corner to the upper left corner, and the distance value of each scanning update point is the minimum value of the current value and the distance between adjacent points plus 1. For example, if a current distance value of a point is 5 and a distance between adjacent points above the current distance value is 3, the point is updated to min (5, 3+1) =4.

When extracting the center line from the distance conversion field by the gradient descent method, the motion is iterated along the gradient direction starting from the local maximum point of the distance field. For each point, calculating the distance value gradient of the 8 neighborhood points, and selecting the direction with the maximum gradient to advance until reaching the local extreme point or the accessed central line point. For example, if the 8-neighborhood distance value of a point is [5,6,5,4,3,4,5,6], then the gradient direction is the direction pointing to the second value (value is 6). And recording all access point coordinates in the iterative process to form a center point sequence. When there are multiple starting points, they are combined into a complete centerline by analyzing the connection between the sequences.

When the contour point sequence and the center point sequence are mapped to the same coordinate space, the original image coordinate system is kept unchanged, and the points of the two sequences are combined. And merging to form a topological structure representation of the defect area, and calculating the number of central line points in the neighborhood of each point on the central line when branch nodes and crossing points in the topological structure are extracted. The neighborhood radius is set to 3 pixels. If there are 3 or more centerline segments in different directions within the neighborhood, the point is an intersection point, and if there are more than 2 centerline points but only 2 different directions, the point is a branch node. For example, if there are 5 centerline points in a neighborhood of a point, each from 3 line segments of different directions, then the point is marked as an intersection.

When the connection relation between the adjacent nodes is calculated, starting from each node, tracking along the central line until reaching the other node, and recording the distance, angle and central line width change between the two nodes. For example, the distance between nodes A and B is 30 pixels, the average width is 5 pixels, and the standard deviation of the width is 0.8 pixels. Based on these connection relations, a topology tree is constructed, each node representing a branch node or intersection, and the edges representing the connection relations between the nodes.

When the structural characteristics of the topological tree are extracted, indexes such as the number of nodes, the number of leaf nodes, the depth of the tree, the branch density and the like are calculated. For example, a defective topology tree has 15 nodes, 8 leaf nodes, a tree depth of 4, and a branch density of 0.6. These indices constitute the topological feature vector of the defect area.

And carrying out normalization processing on the topological structure characteristics by adopting a maximum and minimum normalization method. For example, if the minimum value of the number of nodes in a sample is 5, the maximum value is 25, and the number of nodes in a certain sample is 15, the normalized value is (15-5)/(25-5) =0.5. And performing similar processing on all the characteristic components to obtain normalized characteristic vectors. The pearson correlation coefficient between the features is used when calculating the correlation coefficient between the feature components. For example, a correlation coefficient of 0.85 for the number of nodes and the number of leaf nodes indicates a high correlation. All the correlation coefficients form a characteristic correlation matrix. The distinguishing capability of each feature component is calculated based on the feature correlation matrix, and an information gain method is used. For each feature, its information gain value in the classification task is calculated, the higher this value is indicative of the more discriminative power. For example, a branch density feature with an information gain of 0.75 indicates that the feature has a strong discrimination capability.

And normalizing the information gain value of each feature to be used as the weight of the corresponding feature. For example, in a defect classification task, the topological structure features include four features of node number, leaf node number, tree depth and branch density, and weights of the four features are respectively 0.2, 0.15, 0.25 and 0.4. The feature components are weighted and combined using these weights to form a defect feature description. And matching the defect characteristic description with a preset multi-type defect sample, and calculating Euclidean distance and cosine similarity between the sample to be detected and the characteristic description of each type of defect sample. For example, the Euclidean distance between the sample to be tested and the class A defect is 0.25, the cosine similarity is 0.92, the Euclidean distance between the sample to be tested and the class B defect is 0.47, and the cosine similarity is 0.81. These distance and similarity values form a feature matching matrix.

And when the matching degree of the sample to be detected and the various types of defect samples in the feature matching matrix is analyzed, comprehensively considering the distance and the similarity, and calculating the comprehensive matching score. For example, the composite score for a class A defect is 0.85 and the class B defect is 0.72. And selecting the type with the highest score as the defect type, and taking the matching score corresponding to the defect type as the confidence score. In the above example, the system outputs the recognition result as "defect type: class A; confidence: 0.85".

Based on the technical scheme, the structural modeling and accurate classification of the complex defect area can be realized. The method comprises the steps of constructing a topological structure by extracting a closed contour and a central line, reserving the connection relation between geometric features and space of a defect region, improving the comprehensiveness and the separability of defect characterization, extracting key node relations based on the topological tree structure, quantifying the features, helping to distinguish defects with similar forms and obvious structural differences, introducing correlation analysis and weighted combination among feature components, constructing defect feature description with higher discrimination, improving the classification accuracy and robustness in feature matching, and supporting intelligentized and interpretable defect identification and subsequent quality control by finally outputting defect types and confidence scores.

In an alternative embodiment, constructing a dynamic decision matrix based on the region enhancement weights and the confidence scores, calculating a composite score for each defect region in the dynamic decision matrix, grading the defects according to the composite scores, generating a defect severity quantization index, and outputting a defect detection result includes:

As shown in fig. 3, a schematic diagram of the intelligent defect detection and comprehensive rating flow in this embodiment is shown.

In this embodiment, when calculating the distribution uniformity of the enhancement weights of the region, it is necessary to analyze the distribution of the enhancement weights in the defect region. The uniformity of distribution is represented by calculating the coefficient of variation of the enhancement weights within the region, which is the standard deviation divided by the average value. For a detected electronic component solder joint defect area, if the average value of the enhancement weight is 0.75 and the standard deviation is 0.15, the variation coefficient is 0.2, which indicates that the distribution is relatively uniform, and if the standard deviation is 0.3, the variation coefficient is 0.4, which indicates that the distribution is non-uniform. The higher the distribution uniformity is, the larger the corresponding scoring weight coefficient is, and the value range is usually 0.6 to 0.9.

The fluctuation range of the confidence score is determined by analyzing the difference between the maximum value and the minimum value of the confidence scores of all pixel points in the defect area. For a detected microcrack defect, the range of fluctuation is 0.14 if the confidence score in the region is 0.92 at maximum and 0.78 at minimum, and 0.4 if the confidence score is 0.95 at maximum and 0.55 at minimum. The smaller the fluctuation range is, the more stable the confidence score is, and the larger the corresponding scoring weight coefficient is, and the value range is usually 0.65 to 0.95.

And taking the distribution uniformity and the fluctuation range as scoring weight coefficients, and carrying out weighted fusion on the region enhancement weights and the confidence scores to generate a scoring matrix. Specifically, for each defective region, the product of the distribution uniformity weight and the region enhancement weight is calculated, and the product of the fluctuation range weight and the confidence score is added to form the element value of the scoring matrix. For a product surface scratch defect, if the distribution uniformity weight is 0.85, the region enhancement weight is 0.78, the fluctuation range weight is 0.75, and the confidence score is 0.83, the corresponding element value of the scoring matrix is 0.85×0.78+0.75×0.83=1.29.

When the change rule of the regional enhancement weight along with the position is extracted to obtain the regional change characteristics, a gradient analysis method is adopted to calculate the change rate of the enhancement weight in the horizontal and vertical directions. For a metal surface dishing defect, if the enhancement weight in the horizontal direction changes from left to right by 0.65, 0.72, 0.85, 0.79 and 0.68, the difference between adjacent positions can be calculated to be 0.07, 0.13, -0.06 and-0.11, the change rule is shown to be increased and then reduced to form peak characteristics, if the enhancement weight in the vertical direction shows similar change trend, the region change characteristic is considered to be 'center protruding', higher dimension weight is given to 0.8, and if the enhancement weight shows monotonous change from one side to the other side, the region change characteristic is considered to be 'gradual change', and medium dimension weight is given to 0.6.

And when the stability of the confidence score is extracted to obtain the reliability characteristic, analyzing the local consistency and the cross-region consistency of the confidence score. Local coherence refers to the difference in confidence scores of adjacent pixels, typically measured using local variance, and cross-regional coherence refers to the difference in confidence scores between different sub-regions, typically measured using inter-region variance. If the local variance is 0.02 and the inter-region variance is 0.05, the reliability feature value is 0.93, which indicates high confidence stability, and if the local variance is 0.08 and the inter-region variance is 0.12, the reliability feature value is 0.8, which indicates medium confidence stability.

And calculating the dimension weight of the scoring matrix based on the region variation characteristics and the reliability characteristics, and constructing a dynamic decision matrix according to the dimension weight. The dimension weight calculation adopts a weighted average method, and the ratio of the regional variation characteristic weight to the reliability characteristic weight is usually 7:3. If the region change feature weight is 0.75 and the reliability feature weight is 0.88, the dimension weight is 0.75x0.7+0.88x0.3=0.789. The dynamic decision matrix is constructed by multiplying each element of the scoring matrix by a corresponding dimension weight.

And calculating the relative importance degree among the dimensions in the dynamic decision matrix, and generating an importance weight vector. The relative importance degree among the dimensions is determined based on a defect type feature library, and different importance weights are given to different types of defects. The importance of the region enhancement weight is generally higher than the confidence score, which may be [0.65,0.35] for a solder joint deficiency on the surface of the electronic component, and the importance of the confidence score is generally higher than the region enhancement weight, which may be [0.4,0.6]. Multiplying the importance weight vector by a scoring matrix to obtain a comprehensive score for each defect region. For a fine crack defect on a glass surface, if the corresponding element of the scoring matrix is [1.25,1.35], the importance weight vector is [0.55,0.45], the comprehensive score is 1.25×0.55+1.35×0.45=1.29.

And constructing a grading evaluation standard for the comprehensive scores, and calculating severity distribution according to the grading evaluation standard. The grading evaluation criteria are typically determined based on historical data and expert experience, for example, for automotive part surface defects, a composite score of less than 0.8 is a minor defect, 0.8 to 1.2 is a general defect, 1.2 to 1.6 is a severe defect, and greater than 1.6 is a fatal defect. The severity distribution is obtained by calculating the proportion of defective areas in each level range.

The severity distribution is converted to a severity quantization index using a membership function. The membership function is a piecewise linear function, mapping the composite score to a severity quantization index range of 0 to 10. For example, if the comprehensive score of the pit defect on the surface of a precise bearing is 1.29, the pit defect falls in a serious defect zone, and the severity quantization index is calculated to be 7.2 points through membership function. And dividing the defect level according to the severity quantization index and outputting a detection result. The defect classification criteria are, for example, 0 to 3 for primary defects, 3 to 6 for secondary defects, 6 to 8 for tertiary defects, and 8 to 10 for quaternary defects. And judging the defect with the severity quantization index of 7.2 minutes as a three-level defect, marking the defect as a red warning in a detection result, providing detailed information such as coordinates, area, shape characteristics and the like of a defect area, and providing decision basis for subsequent processing.

Based on the technical scheme, quantitative evaluation and intelligent classification of the defect detection result can be realized. The method comprises the steps of establishing a scoring matrix and a dynamic decision matrix by fusing region enhancement weights and confidence scores, effectively describing importance and detection stability of a defect region, introducing dimension weights by further combining region change features and reliability features, improving accuracy and interpretability of scoring results, calculating comprehensive scores by adopting relative importance analysis and weighted fusion, enhancing sensitivity to distinguishing different types of defects, and finally realizing continuous quantitative expression of defect grades by severity distribution and membership functions to provide accurate support for defect management and quality assessment.

In a second aspect of the embodiments of the present invention, there is provided a machine vision defect real-time detection and classification system based on deep learning, the system comprising:

A processor;

A memory for storing processor-executable instructions;

The present invention may be a method, apparatus, system, and/or computer program product. The computer program product may include a computer readable storage medium having computer readable program instructions embodied thereon for performing various aspects of the present invention.

It should be noted that the above embodiments are merely for illustrating the technical solution of the present invention and not for limiting the same, and although the present invention has been described in detail with reference to the above embodiments, it should be understood by those skilled in the art that the technical solution described in the above embodiments may be modified or some or all of the technical features may be equivalently replaced, and these modifications or substitutions do not make the essence of the corresponding technical solution deviate from the scope of the technical solution of the embodiments of the present invention.

Claims

1. A machine vision defect real-time detection and classification method based on deep learning, characterized in that it includes:

Acquire image data of the surface of industrial products, and perform region segmentation on the image data along a preset direction sequence to obtain multiple region sub-images;

Within each sub-image region, a correlation strength matrix between a pixel and its neighborhood is constructed. Based on the correlation strength matrix, the local entropy value of each pixel is calculated. The gradient vector of each pixel is extracted and projected in different directions. The gradient direction consistency of each pixel is calculated based on the projection components. The local entropy value and gradient direction consistency are combined to generate regional characteristic data.

Extract the extreme points in the regional characteristic data where the local entropy value of each pixel is consistent with the gradient direction, cluster the extreme points that meet the preset extreme value range to obtain regional seed points, establish the topological connection relationship between regional seed points, calculate the characteristic transfer coefficient, and generate regional growth path;

The local entropy value and gradient direction consistency of the region seed point are propagated outward along the region growth path. The attenuation factor is calculated based on the distance from the pixel to the region seed point. The product of the local entropy value, gradient direction consistency and attenuation factor is used as the region enhancement weight. Based on the region enhancement weight, the image data is subjected to regional adaptive enhancement to obtain the enhanced image data.

Structural features of the enhanced image data are extracted, a feature transfer sequence is established, the correlation matrix between features is calculated based on the feature transfer sequence, a feature optimization path is established based on the correlation matrix, and features are progressively fused along the feature optimization path to obtain fused feature data.

A probability distribution map of defect areas is generated based on the fused feature data. The probability distribution map of defect areas is then corrected by combining the region enhancement weights to identify defect areas. The topological features of the defect areas are extracted to establish defect feature descriptions. Defects are classified based on the defect feature descriptions, and the defect type and confidence score are output.

A dynamic decision matrix is constructed based on regional enhancement weights and confidence scores. A comprehensive score for each defect region is calculated in the dynamic decision matrix. Defects are classified according to the comprehensive scores, a quantitative index of defect severity is generated, and the defect detection results are output.

2. The method according to claim 1, characterized in that, the image data is subjected to region-specific adaptive enhancement based on region enhancement weights to obtain enhanced image data, comprising:

The enhancement coefficient is obtained by weighting the regional characteristic data based on the regional enhancement weight.

A nonlinear mapping function is constructed based on the enhancement coefficient to enhance each region sub-image; a characteristic transfer coefficient is introduced at the region boundary to adjust the enhancement intensity, and the enhanced region sub-images are recombined to obtain the enhanced image data.

3. The method according to claim 1, characterized in that, extracting the structural features of the enhanced image data and establishing a feature transfer sequence includes:

A pyramid hierarchical structure is constructed for the enhanced image data. Overlapping sub-regions are divided in each layer. The local gradient distribution of the overlapping sub-regions is extracted, and the filter direction parameters are adjusted based on the local gradient distribution.

The structural features of the overlapping sub-regions are extracted using an adjusted filter. Corresponding mappings are established between the structural features in the overlapping regions of adjacent layers. The feature fusion order is determined based on the corresponding mappings. The structural features of each layer are combined according to the feature fusion order to obtain multi-level structural features.

A hierarchical transmission tree is constructed based on the feature fusion order. Feature transmission links are established in the hierarchical transmission tree. The feature transmission links include transmission nodes between adjacent features. The spatial structure distribution and directional consistency between adjacent transmission nodes are analyzed. The node transmission priority is determined based on the spatial structure distribution and directional consistency. A feature transmission sequence is generated based on the node transmission priority.

4. The method according to claim 1, characterized in that, calculating the correlation matrix between features based on the feature transfer sequence, establishing a feature optimization path based on the correlation matrix, and progressively fusing features along the feature optimization path to obtain fused feature data includes:

Calculate the feature association strength between adjacent transmission nodes in the feature transmission sequence, and transmit the feature association strength step by step in the feature transmission sequence to construct the association degree matrix;

In the correlation matrix, an initial transmission node is determined. Based on the feature correlation strength, the next transmission node is expanded step by step from the initial transmission node to establish transmission connections between nodes. The transmission connections are connected in series in order of feature correlation strength from high to low to form a feature optimization path.

Along the feature optimization path, starting from the initial transmission node, the features of the current transmission node are weighted and fused with the features of the next transmission node. The weight of the weighted fusion is determined by the feature association strength of the corresponding transmission connection. The fused feature is used as the new current node feature and continues to be fused with the features of the next transmission node. By progressively fusing the features of all transmission nodes, the final fused feature data is obtained.

5. The method according to claim 1, characterized in that, a probability distribution map of defect areas is generated based on the fused feature data, and the probability distribution map of defect areas is corrected by combining region enhancement weights, thereby determining the defect areas including:

Calculate local statistics on the fused feature data, construct an adaptive kernel function based on the local statistics, decompose the adaptive kernel function using orthogonal transformation to obtain a basis function set, and convolve the basis function set with the fused feature data to obtain a feature response map;

Gradient diffusion is performed on the feature response map to obtain the density flow field. Gradient trajectories in the density flow field are extracted. Regional clustering features are calculated based on the convergence points and convergence directions of the gradient trajectories. A probability distribution map of defect areas is generated based on the regional clustering features.

The region enhancement weights are adaptively weighted and fused with the probability distribution map of the defect region to obtain a modified probability map. The boundary point set of the modified probability map is extracted, the multi-moment features of the boundary point set are calculated, a level set function is constructed on the modified probability map, and the evolution speed of the level set function is adaptively adjusted based on the multi-moment features. The defect region is obtained by segmenting the zero level plane of the level set function.

6. The method according to claim 1, characterized in that, extracting the topological features of the defect region, establishing a defect feature description, classifying defects based on the defect feature description, and outputting the defect type and confidence score include:

An adaptive threshold boundary tracking algorithm is used to extract the closed contour of the defect region. The closed contour is then denoised and smoothed to obtain a continuous boundary. A contour point sequence is established, and a distance transformation field is constructed based on the contour point sequence. The center line is extracted from the distance transformation field using the gradient descent method, and a center point sequence is constructed.

The contour point sequence and the center point sequence are mapped to the same coordinate space to construct the topology of the defect region. The branch nodes and intersections in the topology are extracted, the connection relationship between adjacent nodes is calculated, a topology tree is constructed based on the connection relationship, and the structural features of the topology tree are extracted as the topology features of the defect region.

Normalize each feature component of the topological structure feature, calculate the correlation coefficient between the normalized feature components, construct a feature correlation matrix, calculate the discrimination ability of each feature component based on the feature correlation matrix, determine the feature weight according to the discrimination ability, and use the feature weight to weight and combine the feature components to establish a defect feature description.

The defect feature descriptions are matched with preset multi-type defect samples, the distance and similarity between the feature descriptions are calculated, a feature matching matrix is constructed, the matching degree between the test sample and each type of defect sample in the feature matching matrix is analyzed, the type with the highest matching degree is determined as the defect type, and its corresponding matching degree is used as the confidence score.

7. The method according to claim 1, characterized in that a dynamic decision matrix is constructed based on regional enhancement weights and confidence scores, a comprehensive score for each defect region is calculated in the dynamic decision matrix, defects are classified according to the comprehensive scores, a quantitative index of defect severity is generated, and the defect detection results are output, including:

The uniformity of the distribution of the region enhancement weights and the fluctuation range of the confidence scores are calculated. The uniformity of the distribution and the fluctuation range are used as scoring weight coefficients to perform weighted fusion of the region enhancement weights and the confidence scores to generate a scoring matrix.

The variation pattern of the region enhancement weight with location is extracted to obtain the region variation feature, and the stability of the confidence score is extracted to obtain the reliability feature. The dimension weights of the scoring matrix are calculated based on the region variation feature and the reliability feature, and a dynamic decision matrix is constructed based on the dimension weights.

The relative importance between dimensions is calculated in the dynamic decision matrix to generate an importance weight vector. The importance weight vector is then multiplied by the scoring matrix to obtain a comprehensive score for each defect area.

A graded evaluation standard is constructed based on the comprehensive score. The severity distribution is calculated based on the graded evaluation standard. The severity distribution is transformed into a severity quantification index using a membership function. The defect level is classified according to the severity quantification index and the detection results are output.

8. A machine vision defect real-time detection and classification system based on deep learning, used to implement the method of any one of claims 1-7, characterized in that it comprises:

The first unit is used to acquire image data of the surface of industrial products, calculate the local entropy value and gradient direction consistency of each pixel in the image data, determine the region enhancement weight based on the local entropy value and gradient direction consistency, and perform regional adaptive enhancement on the image data based on the region enhancement weight to obtain the enhanced image data.

The second unit is used to extract the structural features of the enhanced image data, establish a feature transfer sequence, calculate the correlation matrix between features based on the feature transfer sequence, establish a feature optimization path based on the correlation matrix, and perform progressive fusion of features along the feature optimization path to obtain fused feature data.

The third unit is used to generate a probability distribution map of defect areas based on the fused feature data, correct the probability distribution map of defect areas by combining the region enhancement weight, determine the defect areas, extract the topological structure features of the defect areas, establish defect feature descriptions, classify defects based on the defect feature descriptions, and output defect types and confidence scores.

The fourth unit is used to construct a dynamic decision matrix based on regional enhancement weights and confidence scores. It calculates the comprehensive score for each defect region in the dynamic decision matrix, classifies the defects according to the comprehensive score, generates a quantitative index of defect severity, and outputs the defect detection results.

9. An electronic device, characterized in that it comprises:

processor;

Memory used to store processor-executable instructions;

The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon computer program instructions, characterized in that, when executed by a processor, the computer program instructions implement the method described in any one of claims 1 to 7.