CN115661129A - Visual workpiece mistaken and neglected loading detection system and method based on cooperative motion - Google Patents
Visual workpiece mistaken and neglected loading detection system and method based on cooperative motion Download PDFInfo
- Publication number
- CN115661129A CN115661129A CN202211452655.3A CN202211452655A CN115661129A CN 115661129 A CN115661129 A CN 115661129A CN 202211452655 A CN202211452655 A CN 202211452655A CN 115661129 A CN115661129 A CN 115661129A
- Authority
- CN
- China
- Prior art keywords
- detection
- workpiece
- category
- module
- pictures
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 205
- 230000000007 visual effect Effects 0.000 title claims abstract description 33
- 238000000034 method Methods 0.000 title claims abstract description 30
- 238000012800 visualization Methods 0.000 claims abstract description 16
- 238000010191 image analysis Methods 0.000 claims abstract description 8
- 238000013527 convolutional neural network Methods 0.000 claims description 16
- 238000010586 diagram Methods 0.000 claims description 11
- 230000006870 function Effects 0.000 claims description 11
- 230000004927 fusion Effects 0.000 claims description 9
- 238000012216 screening Methods 0.000 claims description 6
- 238000012549 training Methods 0.000 claims description 6
- 239000013598 vector Substances 0.000 claims description 6
- 238000004364 calculation method Methods 0.000 claims description 4
- 239000000284 extract Substances 0.000 claims description 4
- 238000007405 data analysis Methods 0.000 claims description 3
- 239000011159 matrix material Substances 0.000 claims description 3
- 238000013507 mapping Methods 0.000 claims description 2
- 238000011897 real-time detection Methods 0.000 abstract 1
- 230000007547 defect Effects 0.000 description 10
- 238000007689 inspection Methods 0.000 description 7
- 238000009434 installation Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 2
- 238000011179 visual inspection Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000002950 deficient Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 230000004438 eyesight Effects 0.000 description 1
- 239000000446 fuel Substances 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000003801 milling Methods 0.000 description 1
- 238000013450 outlier detection Methods 0.000 description 1
- 239000003208 petroleum Substances 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000003908 quality control method Methods 0.000 description 1
- 230000003014 reinforcing effect Effects 0.000 description 1
- 230000000153 supplemental effect Effects 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 230000004304 visual acuity Effects 0.000 description 1
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 1
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Landscapes
- Image Analysis (AREA)
Abstract
A visual workpiece mistaken and neglected loading detection system and method based on cooperative motion are disclosed, wherein the detection system comprises: the device comprises an image acquisition unit, a shared memory and an image analysis unit; the image acquisition unit consists of a movable camera set and a clamp table capable of axially rotating a workpiece; the image analysis unit includes: the device comprises a detection module, a visualization module and an output module. The invention also discloses a visual workpiece mistaken and missed loading detection method based on the cooperative motion. The camera and the workpiece of the system are flexible and movable in position, the omnibearing dead-corner-free coverage detection of the large-size workpiece is realized by a small number of cameras through cooperative motion, the dependence on indoor environment light is very low, the efficiency is high, and the cost is low. The method can realize omnibearing, full-automatic, high-speed, high-precision, easy-to-expand and online real-time detection.
Description
Technical Field
The invention relates to a visual workpiece wrong and neglected loading detection system and a visual workpiece wrong and neglected loading detection method, in particular to a visual workpiece wrong and neglected loading detection system and a visual workpiece wrong and neglected loading detection method based on cooperative motion.
Background
Under the influence of environmental protection and petroleum resource problems, the development of new energy automobile industry is accelerated in all countries around the world from 2011, new energy automobiles are enabled to rush into developed express lanes, and the yield of the new energy automobiles is increased by hundreds of times in eight years. There have been several national claims worldwide that prohibit the sale of traditional fuel vehicles.
In the production process of important parts of the new energy automobile, quality control needs to be carried out on each battery box, wherein assembly holes such as stamping holes, milling holes, threaded holes, water nozzle holes, studs, bolts, shaft sleeves, rivets, nuts and the like for reinforcing the battery box and fixing the battery pack subsequently or connecting other parts of the automobile body are strictly positioned and numerically standard, if wrong and neglected installation exists, the subsequent automobile assembly and other accessories are influenced, the parts are repaired lightly and scrapped heavily.
The difficulty of wrong and neglected installation detection of the battery box is as follows:
(1) The battery boxes are of various types, and the positions and the types of the assembly holes of the battery boxes of different types are different;
(2) The types and the number of the assembly holes on the battery box are numerous, the types exceed 20, and the number exceeds 500;
(3) The battery box has a large area (the length and the width reach or exceed 2 meters), and the assembly holes have a small area (the length and the width are generally less than 1 centimeter);
at present, the detection modes are mainly divided into two types: conventional manual inspection methods and general visual inspection methods:
(1) The manual detection method comprises the following steps: the type of the holes is mainly judged by human eyes, and workers are required to compare the types of the holes one by one. Due to the difficulty of wrong and missed battery box detection, the manual detection difficulty is very high, the detection efficiency is low, the detection quality is difficult to ensure, and the visual acuity of workers is greatly burdened;
(2) General visual inspection techniques: the method mainly combines a camera and some feature recognition to replace human eye judgment, improves the efficiency of manual detection to a great extent, and relieves the pressure of detection workers. However, these methods have extremely high requirements on the layout of the cameras, and generally more than 10 cameras can cover one surface of the battery box, and more than 50 cameras are needed to cover the whole battery box; at the same time, the requirements for ambient lighting are also very high, requiring uniform lighting across the entire surface or multiple surfaces of the battery compartment, which creates significant cost and installation difficulties.
CN113049600A discloses a method and a system for detecting wrong and neglected loading of engine parts based on visual detection, which is a visual identification technology for wrong and neglected loading of engine parts, when an engine reaches a preset position, pictures are taken, visual identification detection is carried out, and if the detection result information is qualified, the engine is released; and if the detection result information is unqualified, sending early warning information to a second preset station. However, the position of the camera is not flexible enough, images of all surfaces of the workpiece cannot be obtained, and the identification effect of the part at a specific angle is not ideal; and the detection range is small, and only the workpiece in the fixed view field of the camera can be detected.
CN114146939A discloses a method and a system for sorting multi-view visual parts based on outlier detection, which is a method for detecting surface defects of an object based on multi-view vision, and comprises the steps of firstly obtaining a current multi-view visual image of a part to be detected, and extracting surface defect characteristic information in the multi-view visual image; the surface defect characteristic information comprises a mean value and a standard deviation of defect pixel points; and calculating a probability density parameter based on the mean value and the standard deviation of the defective pixel points, and judging whether the part to be detected is qualified or not based on a comparison result of the probability density parameter and a preset parameter threshold. However, when the surface defects of the detected product have problems such as complex background texture (including regular and irregular), large scale variation of defect features, similarity of defect region features and background features, and the like, the mean value and the standard deviation cannot be well extracted or represent the defects.
Aiming at the problems existing in the methods, a system and a method for detecting the mistaken and missed loading of the visual workpiece based on the cooperative motion are urgently needed to be found, wherein the camera and the workpiece are flexible and movable in position, the full-angle coverage detection of the large-size workpiece is realized by a small number of cameras through the cooperative motion, the dependence on indoor environment light is very low, the efficiency is high, the cost is low, and the system and the method are all-directional, full-automatic, high-speed, high-precision, easy to expand and online in real time.
Disclosure of Invention
The invention aims to solve the technical problem of overcoming the defects in the prior art and provide a visual workpiece neglected loading detection system based on cooperative motion, which has the advantages of flexible and movable camera and workpiece positions, realization of omnibearing dead-angle-free coverage detection of large-size workpieces by a small number of cameras through cooperative motion, very low dependence on indoor environment light, high efficiency and low cost.
The invention further aims to solve the technical problem of overcoming the defects in the prior art and provide a visual workpiece misloading and neglected loading detection method based on cooperative motion, which is omnibearing, full-automatic, high-speed, high-precision, easy to expand and capable of detecting online in real time.
The technical scheme adopted by the invention for solving the technical problem is as follows: a visual workpiece mis-loading detection system based on coordinated motion comprises: the device comprises an image acquisition unit, a shared memory and an image analysis unit; the image acquisition unit consists of a movable camera set and a clamp table capable of axially rotating a workpiece; the image analysis unit includes: the device comprises a detection module, a visualization module and an output module;
extracting image features of workpiece detection points from pictures shot by a camera on the detection points on the workpiece by a convolutional neural network algorithm in the detection module, mapping the features to different types of spaces through a full connection layer to obtain one-dimensional vectors with the length being the number of the types, outputting the results in a probability form by using a softmax function to obtain the probability of the input pictures corresponding to each type, wherein the type with the maximum probability value is a prediction type; the method for extracting the image characteristics of the detection points of the workpiece comprises the following steps: the method comprises the steps of intercepting a fixed position area, wherein the area comprises detection points on a workpiece, namely an interested area, and taking an intercepted interested area image as the input of a convolutional neural network algorithm to obtain a classification result;
the visualization module is: comparing the prediction type and the real type detected by the detection module, visualizing the same or different comparison results to an original image, and visualizing the same or different comparison results to a workpiece schematic diagram in an OK or NG mode to indicate whether the detection points are mistakenly and neglected to be installed; the real category determination mode is as follows: after the movement track of the camera is determined, the positions of the detection points on the workpiece are relatively fixed during each shooting, the coordinates of the detection points can be determined after one-time complete shooting of the workpiece is completed, and the real categories, namely correct categories, of the detection points are marked; the comparison mode is as follows: when the workpiece detection point is photographed each time, the coordinate of the detection point and the real category of the detection point are known, and a prediction category label under the same detection point coordinate of a detection point area intercepted from a picture is compared with a real category label, so that whether the prediction category label is the same as the real category label is judged; when the prediction category is the same as the real category, it is visualized as OK, and when the prediction category is different from the real category, it is visualized as NG.
The workpiece is a large-sized workpiece, such as a battery case.
The convolutional neural network algorithm has the following advantages compared with other algorithms used in the system of the invention: the detection speed is high, 100 measuring points can be analyzed and judged every second, the accuracy is high and can reach 99.99%.
When the same workpiece in different batches is detected, the clamp table and the camera are relatively fixed at a detection beat point of pause photographing, so that the angle and the position of each photographed picture are consistent; even if the shooting angles of the detection points in the same category are different, the characteristics of the detection points are similar, and the detection points are far similar to different categories.
The visualization is to mark the comparison result on the schematic diagram, and finally the picture is presented on the software interface.
Preferably, the movable camera assembly is controlled by a robotic arm fitted with a supplemental light source.
Preferably, the number of cameras in the movable camera group is 1-2 cameras/m width.
Preferably, the shooting angle of the camera is 0-45 degrees with respect to the direction perpendicular to the detection point, and the mechanical arm rotates randomly.
PreferablyThe value range of the softmax function, namely the normalized exponential function is as follows: (∞, + ∞), range: (0,1) as follows: softmax =
=
Wherein z is i And C is the output value of the ith node, and the number of output nodes, namely the number of classified categories. The output value of the multi-classification can be converted into the range of [0, 1 ] through the softmax function]And a probability distribution of 1. For example, some model output: [ -0.3,0.1,0.6]After calculation of softmax: [0.2020,0.3013,0.4967]Which value is large in the predicted result represents a large probability of belonging to the category.
Preferably, the different class spaces are formed by extracting features common to each of the different class detection points through a model training summary by the convolutional neural network algorithm.
Preferably, the model training means: the convolution kernels of the convolution network traverse the standard detection point pictures to perform matrix operation, the data characteristics of the pictures are extracted, each convolution kernel and the picture operation can obtain one characteristic graph, and the plurality of convolution kernels can obtain a plurality of characteristic graphs, so that the characteristics of the plurality of dimensions of the input standard detection point pictures are learned. Some are natural features that can be intuitively felt, such as brightness, edges, texture, color, and the like; some of them are obtained by transformation or processing, such as moment, histogram, principal component, etc.; the same type of picture will have more identical and similar features.
Preferably, the different category spaces all have category labels, and the category labels are labels defined for each different type of detection points by classifying the detection points according to types after a data analysis engineer obtains a picture.
Preferably, the prediction classes are also output with corresponding class labels.
Preferably, after a detection point is shot for multiple times, an odd number of imaging images are firstly reserved by screening before the detection module detectsAfter the detection of the detection module, the voting criterion is used to determine whether the result of the measuring point is OK or NG by comparing the number of votes obtained with the same or different results, or after the softmax is carried out on the pictures of a plurality of visual angles at the same detecting point, the weighted and fused average value is used as the final judgment basis, and the calculation formula is replaced by: softmax =
=
+
+ … +
Wherein, V represents a viewing angle,Wvrepresenting a viewing anglevThe weight of (c). The effect of different viewing angles on the result is eliminated by weighted fusion.
The technical scheme adopted for further solving the technical problems is as follows: a visual workpiece misloading and neglected loading detection method based on cooperative motion comprises the following steps:
(1) Continuously changing the angle of a fixture table capable of axially rotating the workpiece, and taking pictures along the set shooting position and angle by the movable camera group when the fixture table rotates to an angle until all pictures can cover all detection points of the whole workpiece and are stored in a shared memory;
(2) In the photographing process, a convolutional neural network algorithm in a detection module acquires a picture from a shared memory in real time, extracts image characteristics of workpiece detection points, maps the characteristics to different types of spaces through a full connection layer to obtain one-dimensional vectors with the lengths of the types, outputs the results in a probability form by using a softmax function to obtain the probability of an input picture corresponding to each type, and outputs the type with the maximum probability value as a prediction type by using a type label;
(3) Comparing the predicted category label detected by the detection module with the real category label, visualizing the same or different comparison results to an original image by using a visualization module, and visualizing the same or different comparison results to a workpiece schematic diagram in an OK or NG mode to indicate whether the detection point is wrongly installed or not; after a detection point is shot for multiple times, before the detection of a detection module, an odd number of clear images are reserved by screening, after the detection of the detection module, a voting criterion is used to compare the number of votes obtained with the same or different results, or after the votetmax, the average value of the weighted fusion is used to determine whether the result of the detection point is OK or NG;
(4) When all the detection points are OK, the output module sends a normal workpiece outflow signal, and when NG occurs at the detection points, the output module sends a workpiece rework signal.
In the step (2), for different detection points, the number and the angle of the pictures are related to the type of the detection points, for example, one nut can be vertically shot, and the threaded hole takes four pictures from four directions at 45 degrees in a direction perpendicular to the threaded hole.
The invention has the following beneficial effects:
(1) Starting from the actual industrial application scene, the camera and the workpiece are flexible and movable, and the rotatable workpiece and the movable camera/lamp light realize the omnibearing dead-corner-free coverage detection of the large-size workpiece by a small number of cameras through cooperative motion;
(2) The invention can realize the shooting effect of dozens of cameras and stable indoor light sources by using a small number of cameras and one panel light, has very low dependence on indoor environment light, high efficiency and low cost;
(3) When the novel-type workpiece is detected, only the shooting position of the camera needs to be adjusted, other hardware equipment does not need to be replaced, and later expansion is very convenient;
(4) The detection model of the invention is combined with the most advanced deep learning network algorithm, the precision of the detection model after debugging is as high as 99.99%, the speed is as high as 100 measuring points/s, and the result is obtained by almost synchronizing with the shooting, thus realizing the online real-time wrong and neglected loading detection with omnibearing, full automation, high speed and high precision.
Drawings
FIG. 1 is a schematic diagram of detection of a detection hole 1 by a convolutional neural network algorithm in a detection module according to an embodiment of the present invention (left is an intercepted region-of-interest image);
FIG. 2 is a schematic diagram of detection of a detection hole 2 by a convolutional neural network algorithm in a detection module according to an embodiment of the present invention (left is a truncated region of interest image);
FIG. 3 is a schematic diagram of detection of a detection hole 3 by a convolutional neural network algorithm in a detection module according to an embodiment of the present invention (left is a truncated region of interest image);
FIG. 4 is a schematic diagram of detection of a detection hole 4 by a convolutional neural network algorithm in a detection module according to an embodiment of the present invention (left is a truncated region of interest image);
fig. 5 shows the visualization module visualizing the comparison result to the original drawing (the box indicates that the prediction category is the same as the real category).
Detailed Description
The invention is further illustrated by the following examples and figures.
The cell box used in this example is 2 m wide, and has a total of 352 and 8 types of stations.
Visual battery box wrong and neglected loading detection system embodiment based on cooperative motion
The visual battery box wrong and missed installation detection system based on cooperative motion comprises: the device comprises an image acquisition unit, a shared memory and an image analysis unit; the image acquisition unit consists of a movable camera set and a clamp table capable of axially rotating the battery box; the image analysis unit includes: the device comprises a detection module, a visualization module and an output module; the movable camera set is controlled by a mechanical arm provided with a supplementary light source; the number of cameras in the movable camera group is 3; the shooting angle of the camera rotates along with the mechanical arm and is 0-45 degrees with the direction vertical to the detection point.
In this embodiment, the analysis process of the obtained 4 detection holes is described in detail by taking the pictures of the detection holes as an example, and the analysis process of the pictures of the other detection holes is the same as that of the pictures of the other detection holes;
the convolution neural network algorithm in the detection module extracts the image characteristics of the detection points of the workpiece from the pictures of the detection points on the workpiece taken by the camera, and then the image characteristics are obtained through the convolution neural network algorithmThe full-connection layer maps the features to different category spaces to obtain one-dimensional vectors with the length being the number of categories, then a softmax function is used for outputting results in a probability mode to obtain the probability of each category corresponding to the input photo, and the category with the maximum probability value is the prediction category; softmax =
=
Wherein z is i Is the output value of the ith node, and C is the number of output nodes, namely the number of classified categories;
the method for extracting the image characteristics of the detection points of the workpiece comprises the following steps: the method comprises the steps of intercepting a fixed position area, wherein the area comprises detection points on a workpiece, namely an interested area, and taking an intercepted interested area image as the input of a convolutional neural network algorithm to obtain a classification result;
the different types of spaces are formed by extracting common features of detection points of different types through model training and summarizing of a convolutional neural network algorithm; the model training is as follows: the convolution kernels of the convolution network traverse the standard detection point pictures to perform matrix operation, the data characteristics of the pictures are extracted, each convolution kernel and the picture operation can obtain one characteristic graph, and a plurality of convolution kernels can obtain a plurality of characteristic graphs, so that the characteristics of a plurality of dimensions of the input standard detection point pictures are learned; the different category spaces are provided with category labels, and the category labels are labels defined for each different type of detection points by classifying the detection points according to types after a data analysis engineer obtains a picture; the prediction classes are also output with corresponding class labels.
Detection process of detection hole 1: as shown in FIG. 1, the category labels of the battery pack according to the embodiment of the present invention are: 1,2,3,4,5,6,7, 8, the prediction probability of the detection hole 1 corresponding to each type of label is as follows: 0.002,0.001,0.99,0.001,0.001,0.001,0.002,0.002, the prediction probability corresponding to battery box category 3 is the highest, and the category label of the prediction category of the detection hole 1 is obtained as follows: 3.
detection process of the detection hole 2: as shown in fig. 2, the category labels of the battery pack according to the embodiment of the present invention are: 1,2,3,4,5,6,7, 8, the prediction probability of the detection hole 2 corresponding to each label is as follows: 0.002,0.001,0.002,0.001,0.001,0.001,0.99,0.002, the predicted probability corresponding to battery pack type 7 is the highest, and the label of the predicted type of the detection hole 2 is obtained as follows: 7.
the visualization module is: comparing the prediction type and the real type detected by the detection module, visualizing the same or different comparison results to an original image, and visualizing the same or different comparison results to a workpiece schematic diagram in an OK or NG mode to indicate whether the detection points are mistakenly and neglected to be installed; the real category determination mode is as follows: after the movement track of the camera is determined, the positions of the detection points on the workpiece are relatively fixed during each shooting, the coordinates of the detection points can be determined after one-time complete shooting of the workpiece is completed, and the real categories, namely correct categories, of the detection points are marked; the comparison mode is as follows: when the workpiece detection point is photographed each time, the coordinate of the detection point and the real category of the detection point are known, and a prediction category label under the same detection point coordinate of a detection point area intercepted from a picture is compared with a real category label, so that whether the prediction category label is the same as the real category label is judged; when the prediction category is the same as the real category, it is visualized as OK, and when the prediction category is different from the real category, it is visualized as NG.
Visualization of well 1: the class labels for the true classes at inspection hole 1 are: the same results were obtained in the comparison, and the original image was visualized (see fig. 5).
Visualization of well 2: the class labels of the true classes at the inspection well 2 are: the same results were obtained in comparison 7, and the original image was visualized (see fig. 5).
After a detection point is shot for multiple times, before the detection of a detection module, odd number of clear images are reserved by screening, after the detection of the detection module, the result of the detection point is OK or NG by comparing the number of votes obtained with the same or different results by using a voting criterion, or after the pictures of multiple visual angles on the same detection point pass through softmax, the average of weighted fusion is carried outThe value is used as the final judgment basis, and the calculation formula is replaced by: softmax =
=
+
+ … +
Where, V represents a viewing angle,Wvrepresenting a viewing anglevThe weight of (c).
Detection process of the detection hole 3: as shown in fig. 3, the category labels of the battery pack according to the embodiment of the present invention are: 1,2,3,4,5,6,7, 8, the prediction probability of each type label corresponding to the pictures 1-3 of the inspection hole 3 is as follows: [0.003,0.001,0.33,0.001,0.001,0.002,0.002,0.66], [0.002,0.001,0.99,0.001,0.001,0.001,0.002,0.002], [0.002,0.001,0.99,0.001,0.001,0.001,0.002,0.002], weighted fusion of probability values for three views, e.g., for category 1 the weighted fusion value is: (1 + 0.003+1 + 0.002)/3 =0.0023 (the weight values of the three visual angles are all 1), and the other categories are weighted and fused to obtain: [0.0023,0.001,0.77,0.001,0.001,0.0013,0.002,0.2213] the prediction probability corresponding to battery pack type 3 is the highest, and the type label of the prediction type of the detection hole 3 is obtained as follows: 3.
visualization of well 3: the labels of the true categories at the inspection well 3 are: 3, the comparison results are the same; or by adopting a voting criterion, for the category 3, 2 comparison results are the same, 1 comparison result is different, the same number of votes obtained is large, the category result of the measuring point is determined to be 3, and the result is visualized to the original image (as shown in fig. 5).
Detection process of the detection hole 4: as shown in fig. 4, the category labels of the battery pack according to the embodiment of the present invention are: 1,2,3,4,5,6,7, 8, the prediction probability of each type label corresponding to the photos 1-3 of the inspection hole 4 is as follows: [0.003,0.001,0.001,0.001,0.66,0.002,0.002,0.33], [0.002,0.001,0.001,0.001,0.99,0.001,0.002,0.002], [0.002,0.001,0.001,0.001,0.99,0.001,0.002,0.002], weighted fusion of probability values for three views, e.g., for category 1 the weighted fusion value is: (1 + 0.003+1 + 0.002)/3 =0.0023 (the weight values of the three visual angles are all 1), and the other categories are weighted and fused to obtain: [0.0023,0.001,0.001,0.001,0.88,0.0013,0.002,0.334] the prediction probability corresponding to battery pack type 5 is the highest, and the type label of the prediction type of the detection hole 4 is obtained as: 5.
visualization of the detection well 4: the labels of the true categories at the inspection hole 4 are: 5, the comparison results are the same; or, by using a voting criterion, for category 5, 3 of the comparison results are the same, 0 of the comparison results are different, the same votes are obtained, the result of determining the measuring point is 5, and the result is visualized to the original image (as shown in fig. 5).
Embodiment of visual battery box wrong and neglected loading detection method based on cooperative motion
(1) Continuously changing the angle of a clamp table capable of axially rotating the battery box, and when the clamp table rotates to an angle, taking pictures along the set picture taking position and angle by the movable camera group until all pictures can cover all detection points of the whole battery box and are stored in a shared memory;
(2) In the photographing process, a convolutional neural network algorithm in a detection module acquires a picture from a shared memory in real time, extracts image characteristics of battery box detection points, maps the characteristics to different types of spaces through a full connection layer to obtain one-dimensional vectors with the lengths being the number of the types, outputs the results in a probability form by using a softmax function to obtain the probability of an input picture corresponding to each type, and outputs the type with the maximum probability value as a prediction type by using a type label;
(3) Comparing the predicted category label and the real category label detected by the detection module, visualizing the same or different comparison results to an original image by using a visualization module, and visualizing the original image to a battery box schematic diagram in an OK or NG mode to indicate whether the detection point is wrongly or incorrectly installed; after a detection point is shot for multiple times, before the detection of a detection module, an odd number of clear images are reserved by screening, after the detection of the detection module, a voting criterion is used to compare the number of votes obtained with the same or different results, or after the votetmax, the average value of the weighted fusion is used to determine whether the result of the detection point is OK or NG;
(4) When all the detection points are OK, the output module sends a normal battery box outflow signal, and when NG occurs at the detection points, the output module sends a battery box reworking signal.
Claims (6)
1. A visual workpiece mis-loading and mis-loading detection system based on coordinated motion is characterized by comprising: the device comprises an image acquisition unit, a shared memory and an image analysis unit; the image acquisition unit consists of a movable camera set and a clamp table capable of axially rotating a workpiece; the image analysis unit includes: the device comprises a detection module, a visualization module and an output module;
extracting image features of workpiece detection points from pictures shot by a camera on the detection points on the workpiece by a convolutional neural network algorithm in the detection module, mapping the features to different types of spaces through a full connection layer to obtain one-dimensional vectors with the length being the number of the types, outputting the results in a probability form by using a softmax function to obtain the probability of the input pictures corresponding to each type, wherein the type with the maximum probability value is a prediction type; the method for extracting the image characteristics of the detection points of the workpiece comprises the following steps: the method comprises the steps of intercepting a fixed position area, wherein the area comprises detection points on a workpiece, namely an interested area, and taking an intercepted interested area image as the input of a convolutional neural network algorithm to obtain a classification result;
the visualization module is: comparing the prediction type and the real type detected by the detection module, visualizing the same or different comparison results to an original image, and visualizing the same or different comparison results to a workpiece schematic diagram in an OK or NG mode to indicate whether the detection points are mistakenly and neglected to be installed; the real category determination mode is as follows: after the movement track of the camera is determined, the positions of the detection points on the workpiece are relatively fixed during each shooting, the coordinates of the detection points can be determined after one-time complete shooting of the workpiece is completed, and the real categories, namely correct categories, of the detection points are marked; the comparison mode is as follows: when the workpiece detection point is photographed each time, the coordinate of the detection point and the real category of the detection point are known, and a prediction category label under the same detection point coordinate of a detection point area intercepted from a picture is compared with a real category label, so that whether the prediction category label is the same as the real category label is judged; when the prediction category is the same as the real category, it is visualized as OK, and when the prediction category is different from the real category, it is visualized as NG.
2. A coordinated motion based visual workpiece misloading detection system as claimed in claim 1 wherein: the movable camera set is controlled by a mechanical arm provided with a supplementary light source; the number of the cameras in the movable camera group is 1-2 cameras/meter wide; the shooting angle of the camera rotates along with the mechanical arm and is 0-45 degrees with the direction vertical to the detection point.
3. A coordinated motion based visual workpiece mis-loading detection system as claimed in claim 1 or claim 2 wherein: the softmax function, namely the normalized exponential function, has a value range of: (∞, + ∞), range: (0,1) as follows: softmax =
= 
Wherein z is i And C is the output value of the ith node, and the number of output nodes, namely the number of classified categories.
4. A coordinated motion based visual workpiece mis-loading detection system as claimed in claim 1 or claim 2 wherein: the different types of spaces are formed by extracting common features of detection points of different types through model training and summarizing of a convolutional neural network algorithm; the model training is as follows: the convolution kernels of the convolution network traverse the standard detection point pictures to perform matrix operation, the data characteristics of the pictures are extracted, each convolution kernel and the picture operation can obtain one characteristic graph, and a plurality of convolution kernels can obtain a plurality of characteristic graphs, so that the characteristics of a plurality of dimensions of the input standard detection point pictures are learned; the different category spaces all have category labels, and the category labels are labels defined for each different type of detection points by classifying the detection points according to types after a data analysis engineer obtains a picture; the prediction classes are also output with corresponding class labels.
5. A coordinated motion based visual workpiece mis-loading detection system as claimed in claim 1 or claim 2 wherein: after a detection point is shot for multiple times, before the detection of a detection module, an odd number of clear images are reserved by screening, after the detection of the detection module, the result of the detection point is OK or NG by using a voting criterion and comparing the number of votes obtained with the same or different results, or after the pictures of multiple visual angles on the same detection point are subjected to softmax, the weighted and fused average value is used as the final judgment basis, and the calculation formula is replaced by: softmax =
=
+
+ … +
Where, V represents a viewing angle,Wvrepresenting a viewing anglevThe weight of (c).
6. A visual workpiece mis-loading detection method based on coordinated motion for use in a system according to any of claims 1 to 5, comprising the steps of:
(1) Continuously changing the angle of a fixture table capable of axially rotating the workpiece, and taking pictures along the set shooting position and angle by the movable camera group when the fixture table rotates to an angle until all pictures can cover all detection points of the whole workpiece and are stored in a shared memory;
(2) In the photographing process, a convolutional neural network algorithm in a detection module acquires a picture from a shared memory in real time, extracts image characteristics of workpiece detection points, maps the characteristics to different types of spaces through a full connection layer to obtain one-dimensional vectors with the lengths of the types, outputs the results in a probability form by using a softmax function to obtain the probability of an input picture corresponding to each type, and outputs the type with the maximum probability value as a prediction type by using a type label;
(3) Comparing the predicted category label detected by the detection module with the real category label, visualizing the same or different comparison results to an original image by using a visualization module, and visualizing the same or different comparison results to a workpiece schematic diagram in an OK or NG mode to indicate whether the detection point is wrongly installed or not; after a detection point is shot for multiple times, before the detection of a detection module, an odd number of clear images are reserved by screening, after the detection of the detection module, a voting criterion is used to compare the number of votes obtained with the same or different results, or after the votetmax, the average value of the weighted fusion is used to determine whether the result of the detection point is OK or NG;
(4) When all the detection points are OK, the output module sends a normal workpiece outflow signal, and when NG occurs at the detection points, the output module sends a workpiece rework signal.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211452655.3A CN115661129A (en) | 2022-11-21 | 2022-11-21 | Visual workpiece mistaken and neglected loading detection system and method based on cooperative motion |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211452655.3A CN115661129A (en) | 2022-11-21 | 2022-11-21 | Visual workpiece mistaken and neglected loading detection system and method based on cooperative motion |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115661129A true CN115661129A (en) | 2023-01-31 |
Family
ID=85017413
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211452655.3A Pending CN115661129A (en) | 2022-11-21 | 2022-11-21 | Visual workpiece mistaken and neglected loading detection system and method based on cooperative motion |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115661129A (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106547885A (en) * | 2016-10-27 | 2017-03-29 | 桂林电子科技大学 | A kind of Text Classification System and method |
| US20210398271A1 (en) * | 2020-06-19 | 2021-12-23 | Hyundai Motor Company | Inspection system and method for vehicle underbody |
| CN115050023A (en) * | 2022-06-06 | 2022-09-13 | 哈尔滨工业大学（深圳） | Water inlet risk type identification method based on convolutional neural network |
-
2022
- 2022-11-21 CN CN202211452655.3A patent/CN115661129A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106547885A (en) * | 2016-10-27 | 2017-03-29 | 桂林电子科技大学 | A kind of Text Classification System and method |
| US20210398271A1 (en) * | 2020-06-19 | 2021-12-23 | Hyundai Motor Company | Inspection system and method for vehicle underbody |
| CN115050023A (en) * | 2022-06-06 | 2022-09-13 | 哈尔滨工业大学（深圳） | Water inlet risk type identification method based on convolutional neural network |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20250391009A1 (en) | Method for monitoring manufacture of assembly units | |
| CN110142785A (en) | A kind of crusing robot visual servo method based on target detection | |
| CN115965960A (en) | Weld joint identification method based on deep learning and 3D point cloud | |
| CN110135521A (en) | Pole-piece pole-ear defects detection model, detection method and system based on convolutional neural networks | |
| CN115131268A (en) | Automatic welding system based on image feature extraction and three-dimensional model matching | |
| US20020169586A1 (en) | Automated CAD guided sensor planning process | |
| CN112488022B (en) | Method, device and system for monitoring panoramic view | |
| CN110910350A (en) | A kind of nut loose detection method for wind power tower | |
| CN115861448A (en) | System calibration method and system based on angular point detection and characteristic point extraction | |
| CN119540724B (en) | A light guide plate surface defect detection method based on GCC-YOLOv8 network model | |
| CN113727022A (en) | Inspection image acquisition method and device, electronic equipment and storage medium | |
| CN111462310A (en) | A spatial location method of bolt defects based on multi-view geometry | |
| CN111815560A (en) | Photovoltaic power station fault detection method and device, portable detection equipment and storage medium | |
| CN119715554A (en) | Electric tricycle body spraying defect detection method and system based on machine vision | |
| Rio-Torto et al. | Automatic quality inspection in the automotive industry: a hierarchical approach using simulated data | |
| CN120765958B (en) | Intelligent inspection method and equipment based on rapid matching of images and point cloud | |
| CN115661129A (en) | Visual workpiece mistaken and neglected loading detection system and method based on cooperative motion | |
| CN118644468B (en) | A battery surface defect identification method, system, terminal and storage medium | |
| Kozák et al. | Visual localization via semantic structures in autonomous photovoltaic power plant inspection | |
| Ye et al. | Automatic optical apparatus for inspecting bearing assembly defects | |
| Klarák et al. | Processing 3D data from laser sensor into visual content using pattern recognition | |
| Shanbin et al. | Electrical cabinet wiring detection method based on improved yolov5 and pp-ocrv3 | |
| Bao et al. | Flexible Visual Measurement System for Dimension Measurement of Profiled Springs | |
| CN118657777B (en) | Fuse detection method and system based on projection technology | |
| CN118823764B (en) | A method and device for identifying blind spots in power equipment inspection |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20230131 |
|
| RJ01 | Rejection of invention patent application after publication |