CN115719380A - Visual robot fusion method - Google Patents

Abstract

The invention discloses a visual robot fusion method, aiming at overcoming the problem of inaccurate positioning in the prior art, which includes the establishment process of workpiece coordinate system and tool coordinate system, N point calibration process, template production process, feature image processing process, Attitude adaptive process and height automatic adjustment process.

Description

A visual robot fusion method

technical field

The invention belongs to machine vision technology, in particular to a visual robot fusion method.

Background technique

Robot positioning accuracy is an important index to measure its working performance. At present, due to factors such as manufacturing and installation, most of the visual robots produced by domestic and foreign manufacturers have low positioning accuracy and cannot meet the needs of high-precision processing. It has become the core content of machine vision technology to analyze various factors and improve the absolute positioning accuracy of robots as much as possible.

At present, the commonly used robot calibration methods at home and abroad are usually completed with the help of external advanced measuring equipment, but this leads to problems such as high cost, complicated measurement process, and the need for professional operation. At the same time, it is easy to introduce coordinate system The conversion error also leads to the problem of large measurement error.

In view of the above current situation, the present invention proposes a visual robot fusion method to solve the problem of positioning and calibration.

Contents of the invention

In order to overcome the deficiencies and existing problems of the prior art, the present invention provides a visual robot system and a fusion method thereof.

To achieve the above object, the present invention adopts the following technical solutions:

On the one hand, the present invention provides a visual robot system, which is used to execute the establishment process of the workpiece coordinate system and the tool coordinate system, execute the template production process, the template production process, the feature point position approximation process, the feature image processing process, and the attitude self-adaptation process And height automatic adjustment process, the system includes:

a robot including an actuator;

a two-dimensional camera, which is arranged at the end of the actuator;

a workbench on which the workpiece is positioned;

A feature pattern, which includes feature points and feature line segments.

Preferably, the process of establishing the workpiece coordinate system and the tool coordinate system includes the process of establishing the workpiece coordinate system and the tool coordinate system by the vision robot system, wherein the tool coordinate system includes the TCP position.

Preferably, the N-point calibration process includes the visual robot system establishing a transformation matrix between the image coordinate system and the robot physical coordinate system, wherein N is a positive integer.

Preferably, the template production process includes the visual robot system adjusting the shooting pose of the two-dimensional camera to the characteristic pattern, triggering the two-dimensional camera to take pictures and acquiring the image, processing the image to obtain the pixel outline of the characteristic pattern, and converting the pixel outline of the characteristic pattern As the positioning template, the positioning template is calculated to obtain the physical geometry parameters of the positioning template, wherein the feature pattern includes feature points and feature line segments.

Preferably, the process of approaching the position of the feature point includes setting the feature pattern on the workpiece by the visual robot system, triggering the camera to take pictures and acquiring the image, executing the feature image processing process on the image to obtain the physical geometric parameters of the feature contour, and controlling the camera to be in the robot physical coordinate system The translation on the X-axis and the Y-axis makes the physical geometric parameters of the feature contour approach the physical geometric parameters of the positioning template.

Preferably, the characteristic image processing flow includes processing the image by the visual robot system to obtain the contour of the characteristic pattern, judging whether the contour of the characteristic pattern matches the positioning template, and if so, using the contour of the characteristic pattern as the characteristic contour and calculating according to the characteristic pattern Obtain the physical geometry parameters of the feature profile, if not, readjust the shooting pose of the 2D camera to the feature pattern, trigger the 2D camera to take pictures and obtain an image, wherein the feature profile is the pixel profile of the feature pattern.

Preferably, the posture adaptation process includes the visual robot system controlling the rotation of the two-dimensional camera around the X-axis, Y-axis and Z-axis of the robot's physical coordinate system so that the physical geometric parameters of the feature contour approach the physical geometric parameters of the positioning template.

Preferably, the height automatic adjustment process includes the visual robot system controlling the two-dimensional camera to translate on the Z-axis of the robot's physical coordinate system so that the physical geometric parameters of the feature contour approach the physical geometric parameters of the positioning template.

On the other hand, the present invention also provides a visual robot fusion method, which is implemented by using the above-mentioned visual robot system, including the following steps:

Workpiece coordinate system and tool coordinate system establishment process: establish the workpiece coordinate system and tool coordinate system process, wherein the tool coordinate system includes the TCP position;

N-point calibration process: establish a conversion matrix between the image coordinate system and the robot physical coordinate system, where N is a positive integer;

Template production process: adjust the shooting pose of the two-dimensional camera for the feature pattern, trigger the two-dimensional camera to take pictures and obtain the image, process the image to obtain the pixel outline of the feature pattern, use the pixel outline of the feature pattern as a positioning template, and carry out the positioning template Calculate and obtain the physical geometry parameters of the positioning template, wherein the feature pattern includes feature points and feature line segments;

Feature point position approximation process: set the feature pattern on the workpiece, trigger the camera to take pictures and acquire the image, perform feature image processing on the image to obtain the physical geometry parameters of the feature outline, and control the camera to translate on the X-axis and Y-axis of the robot's physical coordinate system so that The physical geometric parameters of the feature contour approximate the physical geometric parameters of the positioning template;

Feature image processing flow: process the image to obtain the outline of the feature pattern, judge whether the outline of the feature pattern matches the positioning template, if so, use the outline of the feature pattern as the feature outline and calculate the physical geometry parameters of the feature outline according to the feature outline, if Otherwise, readjust the shooting pose of the two-dimensional camera to the characteristic pattern, trigger the two-dimensional camera to take pictures and obtain the image, wherein the characteristic contour is the pixel contour of the characteristic pattern;

Attitude adaptive process: respectively control the two-dimensional camera to rotate around the X-axis, Y-axis and Z-axis of the robot's physical coordinate system so that the physical geometric parameters of the feature contour approach the physical geometric parameters of the positioning template;

Height automatic adjustment process: Control the translation of the two-dimensional camera on the Z-axis of the robot's physical coordinate system so that the physical geometric parameters of the feature contour approach the physical geometric parameters of the positioning template.

Preferably, the process for establishing the workpiece coordinate system and the tool coordinate system specifically includes the following steps:

Step 111: establish the tool coordinate system and determine the TCP position, and set the TCP position on the visual center point, wherein the visual center refers to the center of the lower end surface of the two-dimensional camera lens;

Step 112: Determine the working surface of the robot through the workpiece coordinate system. The XY plane of the workpiece coordinate system is established on the processing surface, wherein the processing surface is a tangent surface on the feature pattern, and the common point of the processing surface and the feature pattern is a feature point.

As preferably, the N-point calibration process specifically includes the following process:

Step 121: set the TCP position on the visual center;

Step 122: Control the two-dimensional camera to face the processing surface, and control the object distance to reach the set value;

Step 123: Set N calibration points on the processing plane, the physical coordinates of the N calibration points are (X ₁ , Y ₁ , Z ₁ ), (X ₂ , Y ₂ , Z ₂ )...(X _N , Y _N , Z _N ), control TCP to move to N calibration points and obtain calibration images, determine the pixel coordinates of TCP on N calibration points according to the calibration images, and the pixel coordinates of TCP on N calibration points are (x ₁ , y ₁ ), (x ₂ ,y ₂ )…(x _N ,y _N );

Step 124: Substituting the following formula according to the physical coordinates of the N calibration points and the pixel coordinates of the TCP on the N calibration points and calculating the conversion matrix according to the least square method:

a、d、b和e分别为旋转分量，c和f分别为平移分量。In the formula, the transformation matrix is

a, d, b, and e are rotation components, respectively, and c and f are translation components, respectively.

Preferably, the template making process specifically includes the following steps:

Step 131: Adjust the shooting pose of the two-dimensional camera to the feature pattern, trigger the two-dimensional camera to take pictures and acquire images, wherein the feature pattern includes feature points and feature line segments; Step 132: Get the pixel outline of the feature pattern, and convert the pixel profile of the feature pattern The outline is used as a positioning template; Step 133: Calculate the pixel geometric parameters of the positioning template according to the positioning template, and convert the pixel geometric parameters of the positioning template into the physical geometric parameters of the positioning template through the transformation matrix, wherein the positioning template geometric parameters include the position parameters of feature points and feature points The size parameter of the line segment.

Preferably, the feature point position approximation process specifically includes the following steps:

Step 211: setting the feature pattern on the workpiece, and positioning the workpiece on the workbench;

Step 212: trigger the two-dimensional camera to take pictures and acquire images;

Step 213: Execute the characteristic image processing flow on the image to obtain the physical geometry parameters of the characteristic contour;

Step 214: Calculate the offset according to the physical geometric parameters of the positioning template and the physical geometric parameters of the feature contour, wherein the offset includes ΔT _X and ΔT _Y , and ΔT _X is the relative positioning template on the feature contour on the X axis of the robot physical coordinate system ΔT _Y is the offset of the feature contour relative to the positioning template on the Y axis of the robot's physical coordinate system;

若ΔT_X超过阈值

则控制二维相机在机器人物理坐标系X轴上平移至ΔT_X不超过阈值

若ΔT_X不超过阈值

则控制二维相机在机器人物理坐标系X轴上迭代平移

的距离直至ΔT_X小于

若ΔT_X小于

时则控制二维相机在机器人物理坐标系X轴上平移ΔT_X的距离，其中，阈值

大于4倍的阈值ΔX_i；Step 215: Judging whether ΔT _X is less than the threshold ΔX _i , if ΔT _X is not less than the threshold ΔX _i , then judging whether ΔT _X exceeds the threshold

If ΔT _X exceeds the threshold

Then control the two-dimensional camera to translate on the X-axis of the robot's physical coordinate system until ΔT _X does not exceed the threshold

If ΔT _X does not exceed the threshold

Then control the two-dimensional camera to iteratively translate on the X-axis of the robot's physical coordinate system

The distance until ΔT _X is less than

If ΔT _X is less than

When , the two-dimensional camera is controlled to translate the distance of ΔT _X on the X-axis of the robot's physical coordinate system, where the threshold

Greater than 4 times the threshold value ΔX _i ;

若ΔT_Y超过阈值

则控制二维相机在机器人物理坐标系Y轴上平移至ΔT_Y不超过阈值

若ΔT_Y不超过阈值

则控制二维相机在机器人物理坐标系Y轴上迭代平移

的距离直至ΔT_Y小于

若ΔT_Y小于

时则控制二维相机在机器人物理坐标系Y轴上平移ΔT_Y的距离，其中，阈值

大于4倍的阈值ΔY_i。Step 216: If ΔT _X is less than the threshold ΔX _i , judge whether ΔT _Y is less than the threshold ΔY _i , and if ΔT _Y is not less than the threshold ΔY _i , then judge whether ΔT _Y exceeds the threshold

If ΔT _Y exceeds the threshold

Then control the two-dimensional camera to translate on the Y axis of the robot's physical coordinate system until ΔT _Y does not exceed the threshold

If ΔT _Y does not exceed the threshold

Then control the two-dimensional camera to iteratively translate on the Y axis of the robot's physical coordinate system

The distance until ΔT _Y is less than

If ΔT _Y is less than

Then control the two-dimensional camera to translate the distance of ΔT _Y on the Y axis of the robot's physical coordinate system, where the threshold

Greater than 4 times the threshold value ΔY _i .

Preferably, the feature image processing flow specifically includes the following steps:

S11: The visual robot system extracts the edge points of the pattern in the image according to the Canny algorithm;

S12: The visual robot system generates a pattern outline according to the extracted edge points of the pattern, wherein the pattern outline is composed of several edge points;

S13: The visual robot system calculates the number of edge points and the aspect ratio of the pattern outline according to the pattern outline, and judges whether the number of edge points of the pattern outline is within the safe range and whether the aspect ratio of the pattern outline is within the safe aspect ratio range, If so, keep the corresponding pattern outline, otherwise remove the corresponding pattern outline;

S14: When the visual robot system judges whether the difference between an edge point of the retained pattern outline and an adjacent edge point is at the gradient change threshold, if so, it is determined that the edge point is the intersection point of the outline, and whether the intersection point of the outline of the pattern outline is judged to be Located within the range of safe intersection points, if so, reconstruct the outline of the feature pattern according to the outline intersection of the pattern outline;

S15: The visual robot system calculates the pixel geometric parameters of the feature pattern according to the outline of the feature pattern, converts the pixel geometric parameters of the feature pattern into the physical geometry parameters of the feature pattern according to the transformation matrix, and judges whether the pixel geometry parameters of the feature pattern and the physical geometry parameters of the positioning template are similar relationship, if it is determined that the matching between the feature pattern and the positioning template is successful and the contour of the feature pattern is used as the feature contour, otherwise it is judged that the matching between the feature pattern and the positioning template is unsuccessful and the two-dimensional camera is readjusted to the shooting pose of the feature pattern and the image is obtained.

As preferably, the posture adaptation process specifically includes the following steps:

Step 221: trigger the two-dimensional camera to take pictures and acquire images;

Step 222: Execute the characteristic image processing flow on the image to obtain the physical geometry parameters of the characteristic contour;

Step 223: Calculate the offset according to the physical geometric parameters of the feature contour and the physical geometric parameters of the positioning template. The offset includes θ _Z1 , θ _X1 and θ _Y1 , and θ _Z1 is the relative positioning template of the feature profile on the Z-axis of the robot's physical coordinate system The offset angle of , θ _X1 is the offset angle of the feature profile relative to the positioning template on the X axis of the robot physical coordinate system, and θ _Y1 is the offset angle of the feature profile relative to the positioning template on the Y axis of the robot physical coordinate system;

Step 224: Determine whether θ _Z1 is smaller than the threshold θ _Z , if not, control the two-dimensional camera to rotate around the Z-axis of the robot's physical coordinate system until it is smaller than the threshold θ _Z ;

若θ_X1大于阈值

则控制二维相机绕机器人物理坐标系X轴旋转至θ_X1不大于阈值

若θ_X1不大于阈值

则控制二维相机绕机器人物理坐标系X轴迭代旋转

的角度直至θ_X1小于

若θ_Z1小于

则控制二维相机绕机器人物理坐标系X轴旋转θ_X1的角度，其中，阈值θ_X大于4倍的阈值

Step 225: If θ _Z1 is smaller than the threshold θ _Z , judge whether θ _X1 is smaller than the threshold θ _X , and if θ _X1 is not smaller than the threshold θ _X , then judge whether θ _X1 is greater than the threshold

If θ _X1 is greater than the threshold

Then control the two-dimensional camera to rotate around the X axis of the robot's physical coordinate system until θ _X1 is not greater than the threshold

If θ _X1 is not greater than the threshold

Then control the iterative rotation of the two-dimensional camera around the X-axis of the robot's physical coordinate system

The angle until θ _X1 is less than

If θ _Z1 is less than

Then control the two-dimensional camera to rotate around the X-axis of the robot's physical coordinate system by an angle of θ _X1 , where the threshold θ _X is greater than 4 times the threshold

若θ_Y1大于阈值

则控制二维相机绕机器人物理坐标系Y轴旋转至θ_Y1不大于阈值

若θ_Y1不大于阈值

则控制二维相机绕机器人物理坐标系Y轴迭代旋转

的角度直至θ_Y1小于

若θ_Y1小于

则控制二维相机绕机器人物理坐标系Y轴旋转θ_Y1的角度，其中，阈值θ_Y大于4倍的阈值

Step 226: If θ _X1 is smaller than the threshold θ _X , execute the feature point approximation process, and then judge whether θ _Y1 is smaller than the threshold θ _Y , if θ _Y1 is not smaller than the threshold θ Y, then judge whether _{θ Y1 is} greater than the threshold

If θ _Y1 is greater than the threshold

Then control the two-dimensional camera to rotate around the Y axis of the robot's physical coordinate system until θ _Y1 is not greater than the threshold

If θ _Y1 is not greater than the threshold

Then control the iterative rotation of the two-dimensional camera around the Y axis of the robot's physical coordinate system

The angle until θ _Y1 is less than

If θ _Y1 is less than

Then control the two-dimensional camera to rotate around the Y axis of the robot's physical coordinate system by an angle of θ _Y1 , where the threshold θ _Y is greater than 4 times the threshold

Step 227: If θ _Y1 is less than the threshold θ _Y , execute the feature point position approximation process, and record the current posture information of the robot.

As preferably, the automatic height adjustment procedure includes the steps of:

Step 241: trigger the two-dimensional camera to take pictures and acquire images;

Step 242: Execute the characteristic image processing flow on the image to obtain the physical geometry parameters of the characteristic contour;

Step 243: Calculate the offset according to the physical geometric parameters of the feature profile and the physical geometric parameters of the positioning template, the offset includes ΔH, and ΔH is the height offset of the feature profile relative to the positioning template;

的距离直至ΔH小于

若ΔH小于

则控制二维相机在机器人物理坐标系Z轴上平移ΔH的距离，其中，阈值ΔH2大于4倍的阈值ΔH1。Step 244: Determine whether ΔH is less than threshold ΔH1, if ΔH is less than threshold ΔH1, then determine whether ΔH is greater than threshold ΔH2, if ΔH is greater than threshold ΔH2, control the two-dimensional camera to translate on the Z-axis of the robot's physical coordinate system until ΔH is not greater than threshold ΔH2, if If ΔH is not greater than the threshold ΔH2, the two-dimensional camera is controlled to iteratively translate on the Z-axis of the robot's physical coordinate system

The distance until ΔH is less than

If ΔH is less than

Then control the two-dimensional camera to translate the distance of ΔH on the Z-axis of the robot's physical coordinate system, wherein the threshold ΔH2 is greater than 4 times the threshold ΔH1.

Compared with the prior art, the present invention has outstanding and beneficial technical effects as follows:

(1) In the present invention, the image captured by the two-dimensional camera is processed to realize the position adjustment of the two-dimensional camera, the position of the workpiece and the adjustment of the robot posture, thereby realizing the effect of positioning and processing the workpiece by the actuator at the characteristic position , suitable for positioning processing of workpieces with complex surfaces, so the invention has the advantages of high positioning efficiency and accurate positioning.

(2) In the present invention, in the positioning process, the combination of successive approximation and coarse adjustment and fine adjustment is used to realize the gradual approach of the pixel contour to the positioning template. On the one hand, the efficiency of two-dimensional camera position adjustment is improved, and on the other hand, the 2D camera positioning error.

(3) In the present invention, the postures of the robot and the actuator are strictly positioned to avoid abnormal postures from interfering with the subsequent processing of the workpiece by the robot, thereby ensuring the processing accuracy and quality of the workpiece.

Description of drawings

Fig. 1 is the structural representation of vision robot system of the present invention;

Fig. 2 is a structural schematic diagram of a characteristic pattern of the present invention;

Fig. 3 is a schematic diagram of the overall flow of the visual robot fusion method of the present invention;

Fig. 4 is a schematic diagram of the distribution of 9 calibration points on the workpiece in the visual robot fusion method of the present invention;

Fig. 5 is a schematic diagram of the template making process in the visual robot fusion method of the present invention;

6 is a schematic diagram of a feature point position approximation process in the visual robot fusion method of the present invention;

Fig. 7 is a schematic diagram of the positions of feature points in the rough adjustment and fine adjustment processes in the X direction of the present invention;

Fig. 8 is a schematic diagram of the posture adaptive process in the visual robot fusion method of the present invention;

Fig. 9 is a schematic diagram of the height automatic adjustment process in the visual robot fusion method of the present invention;

In the figure: 1-robot, 2-two-dimensional camera, 3-table, 4-workpiece, 5-characteristic pattern, 6-tool, 11-executing mechanism.

Detailed ways

In order to facilitate the understanding of those skilled in the art, the present invention will be further described below in conjunction with the drawings and specific embodiments.

As shown in Figure 1, a visual robot system includes a robot, a two-dimensional camera, a workbench, a workpiece, a feature pattern, a photoelectric switch, and a touch screen.

The robot includes an actuator. The actuator is used to convert the control signal of the robot into a mechanical arm corresponding to the posture action. It is the main entity for the robot to complete the task. A series of connecting rods, mechanical joints, pose sensors, etc. constitute the robotic arm. A base is installed at the bottom of the actuator, and a cutter is installed at the end of the actuator. The cutter is used as a main tool for processing workpieces during work. In this embodiment, the cutter is a cutting tool.

The two-dimensional camera is used to take images. The robot generates control signals for controlling the actuator by processing the image. The two-dimensional camera is also installed on the end of the actuator. The end of the actuator has an end flange. The tool and the two-dimensional camera are respectively set On the end flange, and the axial direction of the tool and the shooting direction of the two-dimensional camera are parallel to each other, so as to facilitate subsequent positioning and calibration.

The workbench is used to place and clamp the workpiece, and it plays a role in positioning the workpiece. The workbench is set near the robot. In this embodiment, the workbench includes several movable positioning blocks. In actual use, if the workpiece is correctly placed on the workbench, the positioning blocks move to abut against the workpiece, and the positioning blocks position the workpiece on the workbench. superior.

The photoelectric switch (not shown in the figure) is used to detect the position of the workpiece on the workbench, so as to help the workpiece to be correctly positioned on the workbench. The photoelectric switch is electrically connected to the robot, the photoelectric switch is set on the workbench, and the detection range of the photoelectric switch is set on the workbench, so that the photoelectric switch can detect the position of the workpiece on the workbench.

The touch screen (not shown in the figure) is used to realize human-computer interaction. In actual use, the touch screen can display the current pose state of the actuator in real time.

The workpiece is the part that needs to be processed by the robot system. The workpiece has a machining surface on which the tool can start machining. In this embodiment, the workpiece is a car, and the processing surface is a plane tangent to the feature pattern.

The characteristic pattern is used to attach to the workpiece. In actual use, the characteristic pattern is attached to a specific position of the workpiece. The specific position of the workpiece can be determined in advance and manually. The camera takes pictures of the characteristic pattern, and the robot passes through the characteristic pattern Carry out identification and positioning, thereby realizing the identification and positioning of the workpiece. In this embodiment, the specific position of the workpiece may be the processing surface of the workpiece.

In this embodiment, the feature pattern is a soft film with a specific pattern on its surface. As shown in Figure 2, the specific pattern includes rectangles and regular triangles that intersect together, and one endpoint of the regular triangle is set on the center of the rectangle. The characteristic pattern includes feature points and feature line segments. The endpoints of the feature line segments are feature points, and the feature points use g , h, i, j, and k, the feature line segment is represented by A1, A2, B1, and B2, the feature point g is located at the intersection of the regular triangle and the rectangle, and is also located at the midpoint of the side of the regular triangle, and the feature points h and k are located in the rectangle feature points i and j are located at the corners of a regular triangle, feature line segment A1 is a line segment between feature points i and g, feature line segment A2 is a line segment between feature points g and j, feature line segment B1 is feature point h and g, the feature line segment B2 is the line segment between feature points g and k. In actual use, the characteristic pattern can be fully attached to the processing surface of the workpiece, so that it can also have a precise positioning effect on the curved surface.

The visual robot system is used for the establishment process of the workpiece coordinate system and the tool coordinate system, the execution template production process, the template production process, the feature point position approximation process, the feature image processing process, the attitude self-adaptation process and the height automatic adjustment process.

The process of establishing the workpiece coordinate system and the tool coordinate system includes the process of establishing the workpiece coordinate system and the tool coordinate system by the visual robot system, wherein the tool coordinate system includes the TCP position.

The N-point calibration process includes the visual robot system establishing a transformation matrix between the image coordinate system and the robot physical coordinate system, wherein N is a positive integer.

The template making process includes the visual robot system adjusting the shooting pose of the two-dimensional camera to the feature pattern, triggering the two-dimensional camera to take pictures and acquiring the image, processing the image to obtain the pixel outline of the feature pattern, and using the pixel outline of the feature pattern as a positioning template , calculating the positioning template to obtain the physical geometry parameters of the positioning template, wherein the feature pattern includes feature points and feature line segments;

In the template making process, the two-dimensional camera can acquire images in multiple shooting poses of the feature pattern and make multiple positioning templates. In the subsequent feature image processing process, the outline of the feature pattern can be matched with the positioning templates in multiple shooting poses, and the two-dimensional camera can also accurately match the corresponding positioning templates in different poses, thereby improving the matching. scope.

The process of approaching the position of the feature point includes setting the feature pattern on the workpiece by the visual robot system, triggering the camera to take pictures and acquiring the image, performing a feature image processing process on the image to obtain the physical geometric parameters of the feature contour, and controlling the camera to move between the X axis and the robot physical coordinate system. The translation on the Y axis makes the physical geometric parameters of the feature contour approach the physical geometric parameters of the positioning template;

The characteristic image processing flow includes the visual robot system processing the image to obtain the contour of the characteristic pattern, judging whether the contour of the characteristic pattern matches the positioning template, and if so, using the contour of the characteristic pattern as the characteristic contour to obtain the physical characteristic contour according to the characteristic contour calculation. Geometric parameters, if not, readjust the shooting pose of the two-dimensional camera to the characteristic pattern, trigger the two-dimensional camera to take pictures and obtain the image, wherein the characteristic contour is the pixel contour of the characteristic pattern.

The posture adaptation process includes that the visual robot system controls the two-dimensional camera to rotate around the X-axis, Y-axis and Z-axis of the robot's physical coordinate system, so that the physical geometric parameters of the feature contour approach the physical geometric parameters of the positioning template.

The height automatic adjustment process includes the visual robot system controlling the two-dimensional camera to translate on the Z-axis of the robot's physical coordinate system so that the physical geometric parameters of the feature contour approach the physical geometric parameters of the positioning template.

On the other hand, the present invention also provides a visual robot fusion method, which is executed by using the above-mentioned visual robot system. The steps of the visual robot fusion method include a preprocessing process and a real-time processing process. The specific steps of preprocessing include the process of establishing the workpiece coordinate system and the tool coordinate system, the process of N-point calibration and the process of template making. The real-time processing process includes the feature point position approximation process, the feature image processing process, the attitude adaptive process and the height automatic adjustment process.

As shown in Figure 3, in the actual positioning process, the visual robot system starts, and the visual robot system judges whether it is necessary to execute the preprocessing process. If not, the visual robot system enters to judge whether it needs to perform real-time processing. Display the interface of the preprocessing process. The visual robot system judges whether it is necessary to execute the establishment of the workpiece coordinate system and the tool coordinate system. If so, execute the establishment process of the workpiece coordinate system and the tool coordinate system. N-point calibration process, if not, judge whether to execute the template making process, if so, execute the template making process and return to judge whether to execute the preprocessing process, otherwise return to judge whether to execute the preprocessing process. The visual robot system judges whether it is necessary to execute the real-time processing process, and if so, executes the feature point position approximation process and the attitude adaptive process in sequence. After the attitude adaptive process is executed, the visual robot system judges the attitude adjustment angle (that is, the offset θ _Z1 , Whether θ _X1 and θ _Y1 ) exceed the threshold (that is, threshold θ _Z , threshold θ _X and threshold θ _Y ), if so, re-execute the feature point position approximation process and attitude adaptive process, otherwise execute the height automatic adjustment process, and then judge the height Whether the total adjusted value is less than the threshold ΔH3, if so, the visual robot system ends, otherwise, execute the feature point approximation process and then end. If it is not necessary to execute the real-time processing flow, it is judged whether it is necessary to exit the system, and if so, the visual robot system is terminated; otherwise, it returns to the judgment whether it is necessary to execute the preprocessing flow.

Workpiece coordinate system and tool coordinate system establishment process: The visual robot system establishes the workpiece coordinate system and tool coordinate system process, wherein the tool coordinate system includes the TCP position. In this embodiment, as shown in Figure 2, the X-axis and Y-axis in the figure respectively represent the X-axis and Y-axis of the workpiece coordinate system. By establishing the workpiece coordinate system, the X-axis and Y-axis of the workpiece coordinate system can be set at The processing surface of the workpiece, in order to accurately locate the feature pattern on the processing surface.

N-point calibration process: the visual robot system establishes a conversion matrix between the image coordinate system and the robot physical coordinate system, where N is a positive integer. In this embodiment, the value of N is 9, and the conversion between the image coordinate system and the robot physical coordinate system is realized through the N-point calibration process.

Template production process: the visual robot system adjusts the shooting pose of the two-dimensional camera to the feature pattern, triggers the two-dimensional camera to take pictures and acquires images, processes the image to obtain the pixel outline of the feature pattern, and uses the pixel outline of the feature pattern as a positioning template. The positioning template is calculated to obtain the physical geometry parameters of the positioning template, wherein the feature pattern includes feature points and feature line segments. In this embodiment, the positioning template is used to adjust the poses of the two-dimensional camera, the actuator, and the robot relative to the workpiece after the workpiece is subsequently loaded onto the workbench.

In the template making process, the two-dimensional camera can acquire images in multiple shooting poses of the feature pattern and make multiple positioning templates. In the subsequent feature image processing process, the outline of the feature pattern can be matched with the positioning templates in multiple shooting poses, and the two-dimensional camera can also accurately match the corresponding positioning templates in different poses, thus improving the matching process. scope.

Feature point position approximation process: the visual robot system sets the feature pattern on the workpiece, triggers the camera to take pictures and acquires the image, executes the feature image processing process on the image to obtain the physical geometry parameters of the feature outline, and controls the two-dimensional camera to move between the X axis and the robot physical coordinate system. The translation on the Y axis makes the physical geometric parameters of the feature contour approach the physical geometric parameters of the positioning template.

Feature image processing flow: the visual robot system processes the image to obtain the outline of the feature pattern, judges whether the outline of the feature pattern matches the positioning template, and if so, uses the outline of the feature pattern as the feature outline and calculates the physical geometry of the feature outline according to the feature outline parameter, if not, readjust the shooting pose of the two-dimensional camera to the characteristic pattern, trigger the two-dimensional camera to take pictures and obtain the image, wherein the characteristic contour is the pixel contour of the characteristic pattern.

Attitude adaptive process: the visual robot system controls the two-dimensional camera to rotate around the X-axis, Y-axis, and Z-axis of the robot's physical coordinate system, so that the physical geometric parameters of the feature contour approach the physical geometric parameters of the positioning template.

Height automatic adjustment process: the visual robot system controls the two-dimensional camera to translate on the Z-axis of the robot's physical coordinate system so that the physical geometric parameters of the feature contour approach the physical geometric parameters of the positioning template.

The procedure for establishing the workpiece coordinate system and the tool coordinate system specifically includes the following steps:

Step 111: the visual robot system establishes the tool coordinate system and determines the TCP position, and sets the TCP position on the visual center point, wherein the visual center refers to the center of the lower end surface of the two-dimensional camera lens;

Step 112: The lower end surface of the lens of the 2D camera is the exposed end surface of the 2D camera. The visual robot system determines the position of the processing surface of the workpiece through the workpiece coordinate system. The XY plane of the workpiece coordinate system is established on the processing surface. The processing surface is the cut surface on the feature pattern, and the common point of the processing surface and the feature pattern is the feature point.

As a preference, the N-point calibration process, where the value of N is 9, specifically includes the following process:

Step 121: the visual robot system sets the TCP position on the visual center, wherein the visual center refers to the center of the lower end surface of the two-dimensional camera lens;

Step 122: the visual robot system controls the two-dimensional camera to face the processing surface, and controls the object distance to reach the set value;

In the above steps, the position of the two-dimensional camera can be adjusted by the robot. If the two-dimensional camera is facing the processing surface, the lower end surface of the lens of the two-dimensional camera is parallel to the processing surface. The set value can be pre-set in the visual robot system. If the object distance reaches the set value, the focal length of the two-dimensional camera and the processing surface is appropriate, and the focus of the two-dimensional camera is clear.

Step 123: The visual robot system sets N calibration points on the processing surface, and the physical coordinates of the N calibration points are (X ₁ , Y ₁ , Z ₁ ), (X ₂ , Y ₂ , Z ₂ )...(X _N , Y _N , Z _N ), control TCP to move to N calibration points and obtain calibration images, and determine the pixel coordinates of TCP on N calibration points according to the calibration images, and the pixel coordinates of TCP on N calibration points are (x ₁ ,y ₁ ), (x ₂ ,y ₂ )…(x _N ,y _N );

(x₁,y₁)、(x₂,y₂)…(x_N,y_N)可表示为

和

In the above steps, as shown in Figure 4, it is a schematic diagram of the distribution of nine calibration points on the processing surface, and the points from left to right and from top to bottom are the first, second...Nth calibration points respectively. In this embodiment, the calibration point can be set on the feature point a of the feature pattern. (X ₁ ,Y ₁ ,Z ₁ ), (X ₂ ,Y ₂ ,Z ₂ )…(X _N ,Y _N ,Z _N ) can be expressed as

(x ₁ ,y ₁ ), (x ₂ ,y ₂ )…(x _N ,y _N ) can be expressed as

and

Step 124: The visual robot system substitutes the physical coordinates of the N calibration points and the pixel coordinates of the TCP on the N calibration points into the following formula and calculates the conversion matrix according to the least square method:

a、d、b和e分别为旋转分量，c和f分别为平移分量；In the formula, the transformation matrix is

a, d, b, and e are the rotation components, respectively, and c and f are the translation components;

In the above steps, the physical coordinates of the calibration point, the pixel coordinates of the TCP on the calibration point and the transformation matrix satisfy the following formula:

Expand the above formula to get the following formula:

ax+by+c=X(1),

dx+ey+f=Y(2),

In this embodiment, the value of N is 9, and the physical coordinates of the 9 calibration points and the pixel coordinates of the TCP on the 9 calibration points are substituted into the above formula (1) to obtain the following physical coordinate system X-axis and pixel coordinate system X-axis The conversion formula between:

ax ₁ +by ₁ +c＝X ₁

ax ₂ +by ₂ +c＝X ₂

ax ₃ +by ₃ +c＝X ₃

ax ₄ +by ₄ +c＝X ₄

ax ₅ +by ₅ +c＝X ₅

ax ₆ +by ₆ +c＝X ₆

ax ₇ +by ₇ +c＝X ₇

ax ₈ +by ₈ +c＝X ₈ ,

ax ₉ +by ₉ +c＝X ₉

Similarly, the physical coordinates of the 9 calibration points and the pixel coordinates of the TCP at the 9 calibration points are substituted into the above formula (2) to obtain the conversion formula between the Y axis of the physical coordinate system and the Y axis of the pixel coordinate system.

According to the least square method, the variance on both sides of the conversion formula between the X-axis of the physical coordinate system and the X-axis of the pixel coordinate system is minimized, and the following formula is obtained:

In the formula, S(a,b,c) is the difference on the X axis;

Solve the minimum value of S(a,b,c) by calculating the partial derivative of S(a,b,c) and make the value of the first derivative be 0, and then obtain the ternary linear equations of a, b and c, by By calculating the ternary linear equations, the values of a, b and c can be obtained.

Similarly, according to the least squares method, the variance on both sides of the conversion formula between the Y axis of the physical coordinate system and the Y axis of the pixel coordinate system is minimized, and the following formula is obtained:

In the formula, S(d,e,f) is the difference on the Y axis;

Similarly, the values of d, e, and f can be obtained by calculating S(d, e, f), which will not be repeated here, and the conversion matrix can be obtained according to the values of a, b, c, d, e, and f .

The template making process specifically includes the following steps:

Step 131: The visual robot system adjusts the shooting pose of the two-dimensional camera to the feature pattern, triggers the two-dimensional camera to take pictures and acquires images, wherein the feature pattern includes feature points and feature line segments;

In the above steps, the characteristic pattern is placed in the background without other scenes for shooting, and the acquired image only has the characteristic pattern as the scene, so as to facilitate subsequent acquisition of the pixel outline of the characteristic pattern. In this embodiment, the two-dimensional camera can take pictures of multiple poses of the feature pattern and obtain images, and subsequently process the images acquired by the two-dimensional cameras of multiple poses to obtain multiple positioning templates.

Step 132: the visual robot system obtains the pixel outline of the characteristic pattern, and uses the pixel outline of the characteristic pattern as a positioning template;

In the above steps, the visual robot system can obtain the pixel outline of the feature pattern in the image according to the Canny algorithm.

Step 133: The visual robot system calculates the pixel geometric parameters of the positioning template according to the positioning template, and converts the pixel geometric parameters of the positioning template into the physical geometric parameters of the positioning template through the transformation matrix, wherein the positioning template geometric parameters include the position parameters of the feature points and the position parameters of the feature line segments Size parameters.

As shown in FIG. 5 , it is a schematic diagram of the template making process. The visual robot system starts to enter the template making process. The executive mechanism of the visual robot system adjusts the shooting pose of the two-dimensional camera to the characteristic pattern, triggers the two-dimensional camera to take pictures and obtain the image, and then obtains the pixel contour of the characteristic pattern, and then judges whether the contour acquisition is successful , if otherwise, the touch screen of the visual robot system prompts: template matching failed, please adjust the position, if so, set it as a positioning template, and obtain the position of the feature point of the positioning template and the pixel size of the feature line segment and convert it into a physical size, then position the feature point of the template and the characteristic line segment correspond to the characteristic point and the characteristic line segment of the characteristic pattern, and finally judge whether each size value (i.e. the characteristic line segment in the positioning template) is within the threshold range of the actual size (i.e. the characteristic line segment of the characteristic pattern), and if so, record the positioning template The position of the feature point and the physical size of the feature line segment are used as the physical geometry parameters of the positioning template. Otherwise, if it prompts that the physical geometry parameters of the positioning template are wrong, please adjust the position (that is, adjust the shooting pose of the two-dimensional camera for the feature pattern).

The feature point position approximation process specifically includes the following steps:

Step 211: the visual robot system sets the feature pattern on the workpiece, and positions the workpiece on the workbench;

In the above steps, the visual robot system can monitor the position of the workpiece on the workbench through the photoelectric switch. If the workpiece reaches the predetermined position on the workbench, the photoelectric switch sends a trigger signal, and the visual robot system determines that the workpiece is positioned on the workbench.

Step 212: the visual robot system triggers the two-dimensional camera to take pictures and acquire images;

Step 213: The visual robot system executes a feature image processing process on the image to obtain the physical and geometric parameters of the feature profile;

In the above steps, after matching, the deflection angle of the pixel contour relative to the positioning template can also be recorded, and the rotation of the two-dimensional camera can be adjusted according to the deflection angle, so that the pixel contour can be directly facing the positioning template, so as to facilitate subsequent correction of the offset.

Step 214: The visual robot system calculates the offset according to the physical geometric parameters of the positioning template and the physical geometric parameters of the feature contour, where the offset includes ΔT _X and ΔT _Y , and ΔT _X is the relative positioning template in the feature contour in the robot physical coordinate system The offset on the X-axis, ΔT _Y is the offset of the feature contour relative to the positioning template on the Y-axis of the robot's physical coordinate system;

若ΔT_X超过阈值

则控制二维相机在机器人物理坐标系X轴上平移至ΔT_X不超过阈值

若ΔT_X不超过阈值

则控制二维相机在机器人物理坐标系X轴上迭代平移

的距离直至ΔT_X小于

若ΔT_X小于

时则控制二维相机在机器人物理坐标系X轴上平移ΔT_X的距离，其中，阈值

大于4倍的阈值ΔX_i；Step 215: The visual robot system judges whether ΔT _X is less than the threshold ΔX _i , and if ΔT _X is not less than the threshold ΔX _i , then judges whether ΔT _X exceeds the threshold

If ΔT _X exceeds the threshold

Then control the two-dimensional camera to translate on the X-axis of the robot's physical coordinate system until ΔT _X does not exceed the threshold

If ΔT _X does not exceed the threshold

Then control the two-dimensional camera to iteratively translate on the X-axis of the robot's physical coordinate system

The distance until ΔT _X is less than

If ΔT _X is less than

When , the two-dimensional camera is controlled to translate the distance of ΔT _X on the X-axis of the robot's physical coordinate system, where the threshold

Greater than 4 times the threshold value ΔX _i ;

若ΔT_Y超过阈值

则控制二维相机在机器人物理坐标系Y轴上平移至ΔT_Y不超过阈值

若ΔT_Y不超过阈值

则控制二维相机在机器人物理坐标系Y轴上迭代平移

的距离直至ΔT_Y小于

若ΔT_Y小于

时则控制二维相机在机器人物理坐标系Y轴上平移ΔY_i的距离，其中，阈值

大于4倍的阈值ΔY_i。Step 216: If ΔT _X is less than the threshold ΔX _i , the visual robot system judges whether ΔT _Y is less than the threshold ΔY _i , and if ΔT _Y is not less than the threshold ΔY _i , then judges whether ΔT _Y exceeds the threshold

If ΔT _Y exceeds the threshold

Then control the two-dimensional camera to translate on the Y axis of the robot's physical coordinate system until ΔT _Y does not exceed the threshold

If ΔT _Y does not exceed the threshold

Then control the two-dimensional camera to iteratively translate on the Y axis of the robot's physical coordinate system

The distance until ΔT _Y is less than

If ΔT _Y is less than

When , the two-dimensional camera is controlled to translate the distance of ΔY _i on the Y axis of the robot's physical coordinate system, where the threshold

Greater than 4 times the threshold value ΔY _i .

若是则控制机器人在X方向粗调，记录调节数值，若否则控制机器人在X方向上细调，记录调节数值，即步骤215。机器人在X方向粗调指的是控制二维相机在机器人物理坐标系X轴上平移至ΔT_X不超过阈值

控制机器人在X方向上细调指的是控制二维相机在机器人物理坐标系X轴上迭代平移

的距离直至ΔT_X小于

若图像X方向偏差小于ΔX_i则视觉机器人系统判断图像Y方向偏差(即ΔY_i)是否小于ΔY_i，若否则判断图像Y方向偏差是否超过

若是则控制机器人在Y方向粗调，记录调节数值，若否则控制机器人在Y方向细调，记录调节数值，即步骤216。控制机器人在Y方向粗调指的是控制二维相机在机器人物理坐标系Y轴上平移至ΔT_Y不超过阈值

控制机器人在Y方向细调指的是控制二维相机在机器人物理坐标系Y轴上迭代平移

的距离直至ΔT_Y小于

若图像Y方向偏差(即ΔY_i)小于ΔY_i或者执行完毕步骤216后，视觉机器人系统结束特征点位置逼近流程。采用在X和Y方向上粗调和细调相结合的调节流程，兼顾了调节效率和定位精度。As shown in FIG. 6 , it is a schematic diagram of the feature point position approximation process. The vision robot system starts to enter into the process of approximating the position of the feature points, and locates the workpiece in place, that is, step 211 . The visual robot system then triggers the two-dimensional camera to take pictures and acquire images, ie step 212 . The visual robot system then performs the feature image processing flow and obtains the offset, that is, steps 213 and 214 . The visual robot system then judges whether the deviation in the X direction of the image (ie ΔT _X ) is smaller than ΔX _i , if not, judges whether the deviation in the X direction of the image exceeds

If so, control the robot to make rough adjustments in the X direction, and record the adjustment values; otherwise, control the robot to perform fine adjustments in the X direction, and record the adjustment values, that is, step 215 . Coarse adjustment of the robot in the X direction refers to controlling the two-dimensional camera to translate on the X axis of the robot's physical coordinate system until ΔT _X does not exceed the threshold

Fine-tuning the robot in the X direction refers to controlling the iterative translation of the two-dimensional camera on the X-axis of the robot's physical coordinate system.

The distance until ΔT _X is less than

If the deviation in the X direction of the image is less than ΔX _i , then the visual robot system judges whether the deviation in the Y direction of the image (ie ΔY _i ) is less than ΔY _i , otherwise it judges whether the deviation in the Y direction of the image exceeds

If yes, control the robot to make rough adjustments in the Y direction, and record the adjustment values; otherwise, control the robot to perform fine adjustments in the Y direction, and record the adjustment values, that is, step 216 . Controlling the rough adjustment of the robot in the Y direction refers to controlling the two-dimensional camera to translate on the Y axis of the robot's physical coordinate system until ΔT _Y does not exceed the threshold

Controlling the fine-tuning of the robot in the Y direction refers to controlling the iterative translation of the two-dimensional camera on the Y-axis of the robot's physical coordinate system

The distance until ΔT _Y is less than

If the deviation in the Y direction of the image (ie ΔY _i ) is less than ΔY _i or after step 216 is executed, the visual robot system ends the process of approaching the feature point position. The adjustment process combining coarse adjustment and fine adjustment in the X and Y directions is adopted, taking into account both adjustment efficiency and positioning accuracy.

也为X方向位置调节的阈值，

还是粗调和细调过程的临界点。粗调和细调的详细流程如下：As shown in FIG. 7 , a schematic diagram of the position of feature points in the process of coarse adjustment and fine adjustment in the X direction. In the figure, Xi _is the real-time value of ΔT _x , T _x represents the target deviation value adjusted in the X direction, and ΔX _i is the threshold for position adjustment in the X direction,

Also adjust the threshold for X-direction position,

It is also the critical point of the coarse and fine tuning process. The detailed process of coarse adjustment and fine adjustment is as follows:

若是则进行粗调，控制机器人在X方向调节

确保调节后机器人与目标位置的距离在

范围内；The visual robot system judges whether X _i is greater than

If so, perform coarse adjustment and control the robot to adjust in the X direction

Make sure that the distance between the robot and the target position after adjustment is within

within the scope;

若是则进行细调，细调区域为

将该区域进行4等分，实时判断当前位置是在哪个区域，若在后三个区域位置区间内，则控制机器人在X方向运行

的距离，使其进入前一个区域直至进入

范围内；若在

范围内，则控制机器人运行当前的偏移量X_i。特征点位置逼近流程中，

阈值范围设置必须大于4倍的ΔX_i。The visual robot system judges whether _Xi is less than

If so, perform fine-tuning, and the fine-tuning area is

Divide the area into 4 equal parts, and judge in real time which area the current position is in. If it is within the position interval of the last three areas, control the robot to run in the X direction

distance, making it enter the previous area until entering

within the range; if

Within the range, the robot is controlled to run the current offset X _i . In the feature point position approximation process,

The threshold range setting must be greater than 4 times ΔX _i .

Similarly, according to the rough adjustment and fine adjustment processes in the X direction, the coarse adjustment and fine adjustment processes in the Y direction can be known.

The feature image processing flow,

Specifically include the following steps:

S11: The visual robot system extracts the edge points of the pattern in the image according to the Canny algorithm;

In the actual processing process, the image captured by the two-dimensional camera contains not only the characteristic pattern but also the pattern of other scenes. Canny algorithm not only extracts the edge points of feature patterns, but also extracts the edge points of other scene patterns.

S12: The visual robot system generates a pattern outline according to the extracted edge points of the pattern, wherein the pattern outline is composed of several continuous edge points;

In the above steps, the visual robot not only generates the outline of the feature pattern according to the edge points, but also generates the outline of other scene patterns. In this embodiment, the number of pattern outlines is at least two, one of which is the outline of the feature pattern, and the remaining is the outline of other scene patterns.

S13: The visual robot system calculates the number of edge points and the aspect ratio of the pattern outline according to the pattern outline, and judges whether the number of edge points of the pattern outline is within the safe range and whether the aspect ratio of the pattern outline is within the safe aspect ratio range, If so, keep the corresponding pattern outline, otherwise remove the corresponding pattern outline.

In the above steps, in order to be able to distinguish the outline of the characteristic pattern in the outline of the pattern, in this embodiment, a safe number range and a safe aspect ratio range are set in advance in the vision robot system. The safe number range indicates the possible value range of the number of edge points constituting the contour of the characteristic pattern. If the number of edge points of the pattern contour is outside the safe number range, the corresponding pattern contour cannot be the contour of the characteristic pattern, thus Eliminate the pattern outline; the safe aspect ratio range refers to the value range in which the aspect ratio of the characteristic pattern may appear. If the aspect ratio of the pattern outline is outside the safe aspect ratio range, the corresponding pattern outline cannot be If the number of edge points of the pattern contour is within the safe range and the aspect ratio of the pattern contour is within the safe aspect ratio range, the corresponding pattern contour is likely to be a characteristic pattern , thus preserving the pattern outline.

S14: When the visual robot judges whether the difference between an edge point of the retained pattern outline and an adjacent edge point is at the gradient change threshold, if so, it is determined that the edge point is the intersection point of the outline, and it is judged whether the intersection point of the outline of the pattern outline is at Within the range of the number of safe intersections, if so, reconstruct the outline of the feature pattern according to the outline intersection of the pattern outline;

In the above steps, since the direction of the contour line segment near the contour intersection point changes, it is judged whether there is an intersection point of the contour line segment by judging the vector change trend of the adjacent edge points. In this embodiment, a gradient change threshold is preset, and the gradient change threshold represents the value range of the vector difference between adjacent edge points when they are respectively on two adjacent contour line segments. When the difference between an edge point and an adjacent edge point of the pattern contour is located within the gradient change threshold, it is proved that the edge point is a contour intersection point.

In the manner described above, the vision robotic system can determine the number and location of contour intersections of pattern contours. In order to further ensure that the reserved pattern outline is the outline of the characteristic pattern, the visual robot system is also preset with a range of safe intersection points, which is the possible value range of the outline intersection points of the characteristic pattern. In this embodiment, the safety The number of intersections ranges from 9, and the contour intersections of the characteristic pattern include three vertices of a regular triangle, four vertices of a rectangle, and intersections of a regular triangle and a rectangle. If the outline intersection point of the pattern outline is outside the safe intersection point range, then reject the corresponding pattern outline; if the outline intersection point of the pattern outline is within the safe intersection point range, then keep the outline intersection point of the corresponding pattern outline and according to the outline of the pattern outline The intersection points are reconstructed to obtain the outline of the characteristic pattern;

S15: The visual robot system calculates the pixel geometric parameters of the feature pattern according to the outline of the feature pattern, converts the pixel geometric parameters of the feature pattern into the physical geometry parameters of the feature pattern according to the transformation matrix, and judges whether the pixel geometry parameters of the feature pattern and the physical geometry parameters of the positioning template are similar relationship, if it is determined that the matching between the feature pattern and the positioning template is successful and the contour of the feature pattern is used as the feature contour, otherwise it is judged that the matching between the feature pattern and the positioning template is unsuccessful and the two-dimensional camera is readjusted to the shooting pose of the feature pattern and the image is obtained.

In the above steps, the visual robot system also forms a similar relationship between the pixel geometric parameters of the feature pattern and the physical geometric parameters of the positioning template, which means that when the similarity between the feature pattern and the positioning template is less than the similarity threshold, the camera’s shooting pose for the feature pattern is adjusted and Take another photo. In this embodiment, the similarity between the feature pattern and the positioning template is determined according to the different feature line segments of the feature pattern and the different feature line segment ratios of the positioning template. For example, the similarity between the feature pattern and the positioning template is equal to A1 of the feature pattern and A1 of the positioning template. The ratio of A1 is subtracted from the ratio of B1 of the feature pattern and B1 of the positioning template.

In the feature image processing process, the visual robot system matches the outline of the feature pattern in the image with multiple positioning templates, so as to quickly and accurately match the two successfully, and then quickly and accurately calibrate the actuator.

As shown in Figure 8, it is a schematic diagram of the attitude adaptation process, which is used to realize the attitude adjustment of the actuator and the two-dimensional camera on the X-axis, Y-axis, and Z-axis of the physical coordinate system, and ensure that the position of the feature points remains unchanged. The attitude adaptation process specifically includes the following steps:

Step 221: the visual robot system triggers the two-dimensional camera to take pictures and acquire images;

Step 222: The visual robot system executes a feature image processing process on the image to obtain the physical geometry parameters of the feature profile;

In the above steps, the physical geometry parameters of the feature profile include the physical values of A1, A2, B1 and B2.

Step 223: The visual robot system calculates the offset according to the physical geometric parameters of the feature contour and the physical geometric parameters of the positioning template. The offset includes θ _Z1 , θ _X1 and θ _Y1 , and θ _Z1 is the relative positioning template of the feature contour in the physical coordinate system of the robot. The offset angle on the Z axis, θ _X1 is the offset angle of the feature profile relative to the positioning template on the X axis of the robot physical coordinate system, and θ _Y1 is the offset angle of the feature profile relative to the positioning template on the Y axis of the robot physical coordinate system;

In the above steps, the offsets θ _X1 and θ _Y1 are calculated by the proportional relationship formulas of A1, A2 and B1, B2. The above proportional relationship formula is as follows:

分别为第n次获取的A1和A2的比值…第i次获取的A1和A2的比值，a_n...a_i...a₀分别为第n次…第i次…第0次的权重系数，f(θ_X1)为偏移量θ_X1，

分别为第n次获取的A1和A2的比值…第i次获取的A1和A2的比值，b_n...b_i...b₀分别为第n次…第i次…第0次的权重系数。In the formula, f(θ _Y1 ) is the offset θ _Y1 ,

Respectively, the ratio of A1 and A2 acquired for the nth time ... the ratio of A1 and A2 acquired for the i-th time, a _n ... a _i ... a ₀ is the n-th time ... the i-th time ... the 0th time Weight coefficient, f(θ _X1 ) is the offset θ _X1 ,

Respectively, the ratio of A1 and A2 acquired for the nth time ... the ratio of A1 and A2 acquired for the i-th time, b _n ... b _i ... b ₀ are the n-th time ... the i-th time ... the 0th time weight factor.

Step 224: The visual robot system judges whether θ _Z1 is smaller than the threshold θ _Z , if not, controls the two-dimensional camera to rotate around the Z-axis of the robot's physical coordinate system until it is smaller than the threshold θ _Z ;

In the above steps, the visual robot system controls the two-dimensional camera to rotate around the Z-axis of the robot's physical coordinate system until θ _Z is less than the threshold θ _Z1 using a one-step adjustment method.

若θ_X1大于阈值

则控制二维相机绕机器人物理坐标系X轴旋转至θ_X1不大于阈值

若θ_X1不大于阈值

则控制二维相机绕机器人物理坐标系X轴迭代旋转

的角度直至θ_X1小于

若θ_Z1小于

则控制二维相机绕机器人物理坐标系X轴旋转θ_X1的角度，其中，阈值θ_X大于4倍的阈值

Step 225: If θ _Z1 is smaller than the threshold θ _Z , the visual robot system judges whether θ _X1 is smaller than the threshold θ _X , and if θ _X1 is not smaller than the threshold θ X, then judges whether _{θ X1 is} greater than the threshold

If θ _X1 is greater than the threshold

Then control the two-dimensional camera to rotate around the X axis of the robot's physical coordinate system until θ _X1 is not greater than the threshold

If θ _X1 is not greater than the threshold

Then control the iterative rotation of the two-dimensional camera around the X-axis of the robot's physical coordinate system

The angle until θ _X1 is less than

If θ _Z1 is less than

Then control the two-dimensional camera to rotate around the X-axis of the robot's physical coordinate system by an angle of θ _X1 , where the threshold θ _X is greater than 4 times the threshold

In the above steps, the visual robot system controls the movement of the two-dimensional camera on the X-axis of the robot's physical coordinate system, and also adopts rough adjustment and fine adjustment.

若θ_Y1大于阈值

则控制二维相机绕机器人物理坐标系Y轴旋转至θ_Y1不大于阈值

若θ_Y1不大于阈值

则控制二维相机绕机器人物理坐标系Y轴迭代旋转

的角度直至θ_Y1小于

若θ_Y1小于

则控制二维相机绕机器人物理坐标系Y轴旋转θ_Y1的角度，其中，阈值θ_Y大于4倍的阈值

Step 226: If θ _X1 is smaller than the threshold θ _X , the visual robot system executes the feature point approximation process, and then judges whether θ _Y1 is smaller than the threshold θ _Y , and if θ _Y1 is not smaller than the threshold θ _Y , then judges whether θ _Y1 is greater than the threshold

If θ _Y1 is greater than the threshold

Then control the two-dimensional camera to rotate around the Y axis of the robot's physical coordinate system until θ _Y1 is not greater than the threshold

If θ _Y1 is not greater than the threshold

Then control the iterative rotation of the two-dimensional camera around the Y axis of the robot's physical coordinate system

The angle until θ _Y1 is less than

If θ _Y1 is less than

Then control the two-dimensional camera to rotate around the Y axis of the robot's physical coordinate system by an angle of θ _Y1 , where the threshold θ _Y is greater than 4 times the threshold

In the above steps, the way the visual robot system controls the movement of the two-dimensional camera on the Y-axis of the robot's physical coordinate system is also adjusted by means of coarse adjustment and fine adjustment.

Step 227: If θ _Y1 is less than the threshold θ _Y , the visual robot system executes the feature point position approximation process, and records the current posture information of the robot;

In the above steps, after the adjustments of θ _Z1 , θ _X1 , θ _Y1 , θT _X and ΔT _Y are completed, the vision robot system can record the posture of the robot as the final posture when positioning and processing the workpiece.

As shown in Figure 9, it is a schematic diagram of the height automatic adjustment process. The height automatic adjustment process includes the following steps:

Step 241: the visual robot system triggers the two-dimensional camera to take pictures and acquire images;

Step 242: The visual robot system executes a feature image processing process on the image to obtain the physical and geometric parameters of the feature profile;

In the above steps, the physical geometry parameters of the feature profile also include the height of the feature profile, which is the vertical distance between the camera and the feature pattern.

In the above steps, the height in the physical geometry parameters of the feature profile satisfies the following formula:

f(A ₁ )=c _n (A ₁ ) ⁿ +...+c _i (A ₁ ) ⁱ +...+c ₀ ,

f(A ₂ )＝d _n (A ₂ ) ⁿ +...+d _i (A ₂ ) ⁱ +...+d ₀

f(B ₁ )＝e _n (B ₁ ) ⁿ +...+e _i (B ₁ ) ⁱ +...+e ₀

f(B ₂ )＝f _n (B ₂ ) ⁿ +...+f _i (B ₂ ) ⁱ +...+f ₀

In the formula, f(A ₁ ) is the feature contour height calculated according to A1, c _n ... c _i ... c ₀ are the weight coefficients of the nth time...i time...0th time, (A ₁ ) ⁿ ... (A ₁ ) ⁱ is the physical value of A1 obtained for the nth time ... the physical value of A1 obtained for the ith time, f(A ₂ ) is the characteristic contour height calculated according to A2, d _n . ..d _i ... d ₀ are the weight coefficients of the nth time...i time...0th time, (A ₂ ) ⁿ ... (A ₂ ) ⁱ are the physical values of A2 obtained in the nth time ...the physical value of A2 obtained at the i-th time, f(B ₁ ) is the characteristic contour height calculated according to B1, e _n ...e _i ...e ₀ are the n-th time...i-th time...0th time The weight coefficient of the times, (B ₁ ) ⁿ ... (B ₁ ) ⁱ is the physical value of B1 obtained in the nth time ... the physical value of B1 obtained in the i-th time, and f(B ₂ ) is calculated based on B2 The feature contour height of f _n ... f _i ... f ₀ are the weight coefficients of the nth ... ith ... 0th time respectively, (B ₂ ) ⁿ ... (B ₂ ) ⁱ are respectively The physical value of B2 obtained for n times...the physical value of B2 obtained for the ith time. The height of the feature profile can be obtained according to the above formula.

Step 243: The visual robot system calculates the offset according to the physical geometric parameters of the feature contour and the physical geometric parameters of the positioning template, the offset includes ΔH, and ΔH is the height offset of the feature contour relative to the positioning template;

In the above steps, the physical geometry parameters of the positioning template include the height of the positioning template. The height of the positioning template is the vertical distance between the pre-calculated feature pattern and the camera. The height of the positioning template can also be calculated using the formula in step 242.

的距离直至ΔH小于

若ΔH小于

则控制二维相机在机器人物理坐标系Z轴上平移ΔH的距离，其中，阈值ΔH2大于4倍的阈值ΔH1；Step 244: The visual robot system judges whether ΔH is smaller than the threshold ΔH1, if ΔH is smaller than the threshold ΔH1, then judges whether ΔH is larger than the threshold ΔH2, and if ΔH is larger than the threshold ΔH2, controls the two-dimensional camera to translate on the Z-axis of the robot's physical coordinate system until ΔH is not larger than the threshold ΔH2, if ΔH is not greater than the threshold ΔH2, control the iterative translation of the two-dimensional camera on the Z-axis of the robot's physical coordinate system

The distance until ΔH is less than

If ΔH is less than

Then control the two-dimensional camera to translate the distance of ΔH on the Z axis of the robot's physical coordinate system, where the threshold ΔH2 is greater than 4 times the threshold ΔH1;

Step 245: The visual robot system judges whether the total adjustment value of the height is less than ΔH3. If so, it determines that the height of the robot is adjusted too much and the position of the feature point is shifted, and executes the process of approaching the position of the feature point again to adjust the position of the feature point. The above-mentioned total height adjustment The value is equal to the difference between the final 2D camera height and the initial 2D camera height.

The foregoing embodiments are only preferred embodiments of the present invention, and are not intended to limit the scope of protection of the present invention. Therefore, all equivalent changes made according to the structures, shapes and principles of the present invention shall be covered by the protection of the present invention. within range.

Claims (8)

1. A visual robot fusion method is characterized by comprising the following steps:

establishing a workpiece coordinate system and a tool coordinate system: establishing a workpiece coordinate system and a tool coordinate system flow, wherein the tool coordinate system comprises a TCP position;

and (4) calibrating the flow at N points: establishing a conversion matrix between an image coordinate system and a robot physical coordinate system, wherein N is a positive integer;

the template manufacturing process comprises the following steps: adjusting the shooting pose of the two-dimensional camera on the feature pattern, triggering the two-dimensional camera to shoot and acquire an image, processing the image to obtain a pixel outline of the feature pattern, taking the pixel outline of the feature pattern as a positioning template, and calculating the positioning template to obtain physical and geometric parameters of the positioning template, wherein the feature pattern comprises feature points and feature line segments;

the characteristic point position approximation process: setting the characteristic pattern on a workpiece, triggering a camera to take a picture and obtain an image, executing a characteristic image processing flow on the image to obtain a characteristic profile physical geometric parameter, and controlling the camera to translate on an X axis and a Y axis of a robot physical coordinate system to enable the characteristic profile physical geometric parameter to approach a positioning template physical geometric parameter;

characteristic image processing flow: processing the image to obtain a feature profile, judging whether the feature profile is matched with the positioning template, if so, calculating to obtain physical geometric parameters of the feature profile according to the feature profile, otherwise, readjusting the shooting pose of the two-dimensional camera on the feature pattern, triggering the two-dimensional camera to shoot and acquiring the image, wherein the feature profile is a pixel profile of the feature pattern;

gesture self-adaptation process: respectively controlling the two-dimensional camera to rotate around the X axis, the Y axis and the Z axis of the robot physical coordinate system so as to enable the physical geometric parameters of the characteristic outline to approach the physical geometric parameters of the positioning template;

the height automatic adjustment process comprises the following steps: and controlling the two-dimensional camera to translate on the Z axis of the robot physical coordinate system so that the physical geometric parameters of the characteristic outline approach the physical geometric parameters of the positioning template.

2. The visual robot fusion method of claim 1, wherein the process of establishing the workpiece coordinate system and the tool coordinate system comprises the following steps:

step 111: establishing a tool coordinate system, determining the position of a TCP (transmission control protocol), and setting the position of the TCP on a visual center point, wherein the visual center refers to the center of the lower end face of a two-dimensional camera lens;

step 112: and determining the working surface of the robot through a workpiece coordinate system, wherein an XY plane of the workpiece coordinate system is established on the processing surface, the processing surface is a tangent plane on the characteristic pattern, and a common point of the processing surface and the characteristic pattern is a characteristic point.

3. The visual robot fusion method according to claim 1, wherein the N-point calibration process specifically comprises the following processes:

step 121: setting a TCP location on a visual center;

step 122: controlling the two-dimensional camera to face the processing surface, and controlling the object distance to reach a set value;

step 123: setting N calibration points on the processing surface, wherein the physical coordinates of the N calibration points are respectively (X) ₁ ,Y ₁ ,Z ₁ )、(X ₂ ,Y ₂ ,Z ₂ )…(X _N ,Y _N ,Z _N ) Controlling the TCP to move to the N calibration points and acquiring calibration images, and determining the pixel coordinates of the TCP on the N calibration points according to the calibration images, wherein the pixel coordinates of the TCP on the N calibration points are (x) respectively ₁ ,y ₁ )、(x ₂ ,y ₂ )…(x _N ,y _N )；

Step 124: substituting the physical coordinates of the N calibration points and the pixel coordinates of the TCP on the N calibration points into the following formula and calculating to obtain a conversion matrix according to a least square method:

in the formula, the conversion matrix is

a. d, b and e are rotational components, respectively, and c and f are translational components, respectively.

4. The visual robot fusion method according to claim 1, wherein the template making process specifically comprises the following steps:

step 131: adjusting the shooting pose of a two-dimensional camera to the feature pattern, triggering the two-dimensional camera to shoot and acquire an image, wherein the feature pattern comprises feature points and feature line segments;

step 132: acquiring a pixel outline of the characteristic pattern, and taking the pixel outline of the characteristic pattern as a positioning template;

step 133: and calculating to obtain a positioning template pixel geometric parameter according to the positioning template, and converting the positioning template pixel geometric parameter into a positioning template physical geometric parameter through a conversion matrix, wherein the positioning template geometric parameter comprises a position parameter of a characteristic point and a size parameter of a characteristic line segment.

5. The visual robot fusion method according to claim 1, wherein the feature point position approximation process specifically includes the following steps:

step 211: setting the characteristic pattern on the workpiece, and positioning the workpiece on a workbench;

step 212: triggering a two-dimensional camera to take a picture and acquiring an image;

step 213: executing a characteristic image processing flow to the image to obtain a characteristic contour physical geometric parameter;

step 214: calculating to obtain an offset according to the physical geometric parameters of the positioning template and the physical geometric parameters of the characteristic contour, wherein the offset comprises delta T _X And Δ T _Y ，ΔT _X For the offset, delta T, of the feature profile relative to the positioning template in the X-axis of the robot's physical coordinate system _Y Relatively positioning templates for feature contours on a machine figureThe offset on the Y axis of the physical coordinate system;

step 215: determination of Delta T _x Whether or not less than threshold value DeltaX _i If Δ T _x Not less than threshold value DeltaX _i Then determine Δ T _X Whether or not a threshold value is exceeded

If Δ T _X Exceeds a threshold value

Then the two-dimensional camera is controlled to translate to delta T on the X axis of the robot physical coordinate system _X Does not exceed a threshold

If Δ T _X Does not exceed a threshold

Then the two-dimensional camera is controlled to iteratively translate on the X axis of the robot physical coordinate system

Until Δ T _X Is less than

If Δ T _X Is less than

Then the two-dimensional camera is controlled to translate delta T on the X axis of the robot physical coordinate system _X Wherein the threshold value

Greater than 4 times the threshold Δ X _i ；

Step 216: if Δ T _X Less than a threshold value DeltaX _i Then determine Δ T _Y Whether or not less than threshold value DeltaY _i If Δ T _Y Not less than threshold value DeltaY _i Then determine Δ T _Y Whether or not to exceedOver-threshold

If Δ T _Y Exceeds a threshold value

Then the two-dimensional camera is controlled to translate to delta T on the Y axis of the robot physical coordinate system _Y Does not exceed a threshold

If Δ T _Y Does not exceed a threshold

Then the two-dimensional camera is controlled to iteratively translate on the Y axis of the robot physical coordinate system

Until Δ T _Y Is less than

If Δ T _Y Is less than

Then the two-dimensional camera is controlled to translate delta T on the Y axis of the robot physical coordinate system _Y Wherein the threshold value

Greater than 4 times the threshold value DeltaY _i 。

6. The visual robot fusion method according to claim 1, wherein the feature image processing procedure specifically comprises the following steps:

s11: the visual robot system extracts edge points of the patterns in the image according to a Canny algorithm;

s12: the visual robot system generates a pattern contour according to the extracted edge points of the pattern, wherein the pattern contour is composed of a plurality of edge points;

s13: the visual robot system calculates the number of edge points and the length-width ratio of the pattern contour according to the pattern contour, judges whether the number of the edge points of the pattern contour is within a safe number range and the length-width ratio of the pattern contour is within a safe length-width ratio range, if so, retains the corresponding pattern contour, and otherwise, rejects the corresponding pattern contour;

s14: the visual robot system judges whether the difference value between one edge point of the reserved pattern contour and the adjacent edge point is located at a gradient change threshold value, if so, the edge point is judged to be a contour intersection point, whether the contour intersection point of the pattern contour is located within the range of the number of safe intersection points is judged, and if so, the contour of the characteristic pattern is reconstructed according to the contour intersection point of the pattern contour;

s15: the vision robot system obtains the feature pattern pixel geometric parameters through calculation according to the outline of the feature pattern, converts the feature pattern pixel geometric parameters into feature pattern physical geometric parameters according to a conversion matrix, judges whether the feature pattern pixel geometric parameters and the positioning template physical geometric parameters form a similar relation, judges whether the feature pattern and the positioning template are successfully matched and takes the outline of the feature pattern as the feature outline if the feature pattern pixel geometric parameters and the positioning template physical geometric parameters form the similar relation, judges that the feature pattern and the positioning template are not successfully matched if the feature pattern pixel geometric parameters and the positioning template physical geometric parameters do not form the similar relation, and readjusts the shooting pose of the two-dimensional camera on the feature pattern and obtains an image if the feature pattern pixel geometric parameters and the positioning template physical geometric parameters do not form the similar relation.

7. The visual robot fusion method according to claim 1, wherein the pose adaptation process specifically comprises the steps of:

step 221: triggering a two-dimensional camera to take a picture and acquiring an image;

step 222: executing a characteristic image processing flow to the image to obtain a characteristic contour physical geometric parameter;

step 223: calculating to obtain an offset according to the physical geometric parameters of the characteristic contour and the physical geometric parameters of the positioning template, wherein the offset comprises theta _Z1 、θ _X1 And theta _Y1 ，θ _Z1 Relatively positioning die for feature profileOffset angle of plate on Z-axis of robot physical coordinate system, θ _X1 For the offset angle theta of the feature profile relative to the positioning template on the X axis of the robot physical coordinate system _Y1 The characteristic outline is subjected to deviation angle on the Y axis of the robot physical coordinate system relative to the positioning template;

step 224: determine theta _Z1 Whether or not less than threshold value theta _Z If not, controlling the two-dimensional camera to rotate around the Z axis of the robot physical coordinate system until the Z axis is smaller than the threshold theta _Z ；

Step 225: if theta _Z1 Less than threshold theta _Z Then judge theta _X1 Whether or not it is less than threshold value theta _X If theta _X1 Not less than threshold value theta _X Then judge theta _X1 Whether or not it is greater than a threshold value

If theta _X1 Greater than a threshold value

Then the two-dimensional camera is controlled to rotate to theta around the X axis of the robot physical coordinate system _X1 Is not greater than the threshold

If theta is _X1 Is not greater than the threshold

Then the two-dimensional camera is controlled to rotate around the X axis of the robot physical coordinate system in an iterative way

Up to theta _X1 Is less than

If theta is _Z1 Is less than

Then the two-dimensional camera is controlled to rotate theta around the X axis of the robot physical coordinate system _X1 Wherein the threshold value θ _X Greater than 4 times the threshold

Step 226: if theta _X1 Less than threshold theta _X Then executing the characteristic point approaching process and judging theta _Y1 Whether or not less than threshold value theta _Y If theta _Y1 Not less than threshold value theta _Y Then judge theta _Y1 Whether or not it is greater than a threshold value

If theta _Y1 Greater than a threshold value

The two-dimensional camera is controlled to rotate to theta around the Y axis of the robot physical coordinate system _Y1 Is not greater than the threshold

If theta _Y1 Is not greater than the threshold

Then the two-dimensional camera is controlled to rotate around the Y axis of the robot physical coordinate system in an iterative way

Up to theta _Y1 Is less than

If theta _Y1 Is less than

Then the two-dimensional camera is controlled to rotate theta around the Y axis of the robot physical coordinate system _Y1 Wherein the threshold value θ _Y Greater than 4 times the threshold

Step 227: if theta _Y1 Less than threshold theta _Y And then executing the feature point position approaching process, and recording the current posture information of the robot.

8. The visual robot fusion method according to claim 1, wherein the automatic height adjustment process comprises the following steps:

step 241: triggering a two-dimensional camera to take a picture and acquiring an image;

step 242: executing a characteristic image processing flow on the image to obtain a characteristic contour physical geometric parameter;

step 243: calculating to obtain an offset according to the physical geometric parameters of the characteristic contour and the physical geometric parameters of the positioning template, wherein the offset comprises delta H, and the delta H is the height offset of the characteristic contour relative to the positioning template;

step 244: judging whether delta H is smaller than a threshold delta H1 or not, if the delta H is smaller than the threshold delta H1, judging whether the delta H is larger than a threshold delta H2 or not, if the delta H is larger than the threshold delta H2, controlling the two-dimensional camera to translate on the Z axis of the robot physical coordinate system until the delta H is not larger than the threshold delta H2, and if the delta H is not larger than the threshold delta H2, controlling the two-dimensional camera to iteratively translate on the Z axis of the robot physical coordinate system

Until Δ H is less than

If Δ H is less than

The two-dimensional camera is controlled to translate a distance Δ H on the Z-axis of the robot physical coordinate system, wherein the threshold Δ H2 is greater than 4 times the threshold Δ H1.

Images