CN111775146A

CN111775146A - A visual alignment method under the multi-station operation of an industrial manipulator

Info

Publication number: CN111775146A
Application number: CN202010515905.8A
Authority: CN
Inventors: 叶南; 石世锋; 祝鸿宇; 张丽艳
Original assignee: Nanjing University of Aeronautics and Astronautics
Current assignee: Nanjing University of Aeronautics and Astronautics
Priority date: 2020-06-08
Filing date: 2020-06-08
Publication date: 2020-10-16
Anticipated expiration: 2040-06-08
Also published as: CN111775146B

Abstract

The invention discloses a visual alignment method under multi-station operation of an industrial mechanical arm, which comprises the following steps of firstly, utilizing a system to calibrate and establish the position relation among coordinate systems; secondly, designing a set of cooperative targets, arranging the cooperative targets near the alignment station, and solving the pose of the targets; acquiring a target expected pose, and then establishing an alignment task table for a plurality of stations; and performing alignment operation according to the task table, calculating the deviation between the actual pose and the expected pose of the target, resolving the deviation into motion data of the mechanical arm, driving the tail end to adjust the pose, and finally realizing the accurate alignment of the tail end tool and the station. The alignment method has the advantages of good robustness, reliability and high precision of the measurement result; the hardware system is simple in structure, low in cost and strong in flexibility; the alignment device has the advantages that vision shielding caused by alignment tools and targets can be avoided in the alignment process, the real-time requirement is met, the alignment device is used for solving the close-distance alignment problem, and the alignment device is suitable for application in industrial fields.

Description

Visual alignment method under industrial mechanical arm multi-station operation

Technical Field

The invention belongs to the technical field of intelligent manufacturing, and particularly relates to a visual alignment method for industrial mechanical arm in multi-station operation.

Background

With the continuous development of the aerospace field towards the intelligent manufacturing direction, the industrial mechanical arm is widely applied to assembly work such as alignment, screwing, grabbing and the like of airplane parts. The traditional mechanical arm assembly work realizes basic movement operation by means of teaching reproduction or fixed programming, has low absolute positioning precision and poor perception capability to the environment, and cannot meet the requirement of intelligent assembly. In order to improve the flexibility and flexibility of the assembly of the mechanical arm, the visual sensor is gradually fused with the industrial mechanical arm, and the construction of a proper mechanical arm visual system is an important premise for finishing the assembly operation.

The mechanical arm assembly system based on visual feedback can be divided into Eye-in-Hand and Eye-to-Hand according to the combination mode of the mechanical arm and the vision, and can also be divided into monocular, binocular and multiocular systems according to the number of visual sensors. According to the monocular Eye-to-Hand mechanical arm grabbing system, the size of a target workpiece is known, the pose of the workpiece is obtained through image preprocessing and monocular vision, the mechanical arm is guided to quickly and effectively grab the workpiece, the pose error is small, and the expected requirements of industrial production can be met. The binocular stereoscopic vision system completes three-dimensional reconstruction of key points of a target image according to a binocular three-dimensional reconstruction principle, and achieves pose measurement of a mechanical arm handheld workpiece shaft and a workpiece wafer with holes, wherein stereoscopic matching is a difficult problem. The monocular vision system takes the shaft sleeve of the spline shaft as a positioning target, and fixes an artificial marker on the shaft sleeve to assist in finishing the alignment of the spline shaft and the spline sleeve, but the process completely ignores the posture problem. At present, binocular and multi-view systems have the problems of complex structure, poor robustness, difficult stereo matching between images and the like; the monocular system is simple in structure, easy to calibrate the camera and high in precision, and most of researches in recent years are based on the monocular system. The Eye-to-Hand combination mode enlarges the view field of the mechanical arm, but the shielding problem is easy to generate; the Eye-in-Hand system has higher local precision and more flexible viewing range.

On the basis, in order to accurately reproduce the pose information of the alignment target, a cooperative scheme and a non-cooperative scheme are generally considered in engineering. The non-cooperative scheme is to assist in completing measurement by utilizing the characteristic attributes of the target, but the characteristics are not fixed, are difficult to stably extract, and do not necessarily meet the pose resolving requirement.

Disclosure of Invention

The technical problems solved by the invention are as follows: the existing positioning precision is low, and the perception capability to the environment is poor; the structure is complex, the robustness is poor, and the stereo matching between images is difficult.

The technical scheme is as follows: in order to solve the technical problems, the technical scheme adopted by the invention is as follows:

a visual alignment method under industrial mechanical arm multi-station operation comprises the following steps of firstly, utilizing a system to calibrate and establish a position relation among coordinate systems; secondly, designing a set of cooperative targets, arranging the cooperative targets near the alignment targets, and solving the target pose; acquiring an expected pose of a target, and then establishing an alignment task table for the target; and performing alignment operation according to the task table, calculating the deviation between the actual pose and the expected pose of the target, resolving the deviation into motion data of the mechanical arm, driving the tail end to adjust the pose, and finally realizing the accurate alignment of the tail end tool and the target.

Preferably, in the system calibration, TCP calibration is first performed to obtain a transformation relation between a flange coordinate system and a robot arm base coordinate system

Then camera calibration and hand-eye calibration are carried out, the camera calibration obtains the mapping relation between the camera coordinate system and the image coordinate system, and the hand-eye calibration obtains the transformation relation of the camera coordinate system relative to the flange coordinate system

Preferably, each target is provided with a plurality of feature points, wherein a plurality of feature points are connected to form a convex hull, the remaining feature points are located inside the convex hull, the feature points on the convex hull are numbered according to a clockwise sequence, the feature points on the convex hull are used for calculating a homography matrix, the feature points inside the convex hull are used for verifying the homography matrix, a three-dimensional coordinate point set of the target is obtained, then the camera collects a target image and processes the image to obtain an image feature point set, and the image feature points and the target are establishedMarking the corresponding relation between the three-dimensional points; calculating the pose transformation relation of the target under the camera coordinate system by utilizing a PNP algorithm (Pective-n-Point), namely

Preferably, the image processing procedure of the target is as follows: firstly, carrying out Gaussian filtering on an image to remove redundant noise; using a Canny operator to carry out edge detection, and storing the detected edge in a tree structure; obtaining possible feature point outlines according to area constraint and roundness criteria, then continuously judging whether the gray value of pixel points on the diameter of each outline is continuous or not, and further screening to obtain correct feature point outlines; and fitting the minimum circumscribed rectangle of the feature point profile by adopting least square, and determining the center of the rectangle as the center of the feature point to obtain an image feature point set.

Preferably, the method for establishing the task table comprises the following steps: controlling the mechanical arm to move to enable the tail end tool to be pre-aligned with the target on each station before assembly operation, acquiring an image of a cooperative target corresponding to the target by a camera, and obtaining an expected pose of the target corresponding to the target by using a pose solving algorithm and recording the expected pose as the expected pose

Then, an alignment task is established for the target, and the end tool information, the measured target feature point set and the expected pose are combined

And storing the information into the tasks as prior information, establishing the tasks for the targets of all the stations, and forming a task table.

Preferably, the camera acquires the current image of the target and calculates to obtain the current pose transformation relation of the target relative to the camera coordinate system

Then calculating the transformation relation of the current pose

Date of andview the position of the pose

Error of (2)

If the error is larger than the set threshold value, converting the error into a mechanical arm base coordinate system by combining a system calibration result, calculating the motion data of the mechanical arm and guiding the mechanical arm to move.

Preferably, the image is reacquired and the pose error is calculated after the robot arm moves to a new position until the pose error is calculated

If the value is less than the set threshold value, the alignment task is completed; and searching the task table, judging whether all tasks are finished, and if not, continuing to execute the next task until all tasks are finished.

Preferably, the pose transformation relation of the flange under the base coordinate system of the mechanical arm when the end tool and the target are in an aligned state is set as

When the position and posture of the mechanical arm are adjusted

And when the target and the camera reach the expected pose, the end tool and the target are in an aligned state.

Preferably, the amount of motion of the robot arm is interpolated to obtain a positioning error,

contains two components: rotation matrix

And translation vector

Converting the rotation matrix into quaternion to perform coordinate rotation interpolation, and performing position interpolation on the translation vector, wherein the calculation process is as follows:

wherein,

and

respectively is a pose transformation matrix of the current flange under a mechanical arm base coordinate system

A rotation matrix and a translation vector of (1);

and

respectively is a position and attitude transformation matrix of the flange under the base coordinate system of the mechanical arm in an alignment state

A rotation matrix and a translation vector of (1); k is an interpolation coefficient, and the value of the method is 0.8; slerp () is an interpolation function. Obtained finally

Constructing new transformation matrices

Namely the pose of the mechanical arm needs to be adjusted.

Has the advantages that: compared with the prior art, the invention has the following advantages:

the invention obtains the expected pose of the target and establishes a task table for the target on multiple stations, the system executes alignment operation by taking the task table as the basis, the deviation of the current pose and the expected pose of the target is taken as driving quantity, the motion of the mechanical arm is controlled to carry out iterative pose adjustment, and finally the deviation is converged in a certain range to successfully realize alignment, and the whole system forms a closed loop system based on position feedback. The experimental result shows that the measuring distance is about 260mm, the alignment precision reaches 0.1mm in the X, Y direction, the alignment precision reaches 0.2mm in the Z direction, and the angle error is better than 0.1 degrees.

On the basis of using a monocular Eye-in-Hand system and adopting a cooperative scheme, the visual alignment method under the multi-station operation of the mechanical arm disclosed by the invention is characterized in that a set of cooperative targets is designed to provide characteristic points, the three-dimensional coordinates of the characteristic points in a target coordinate system and the two-dimensional coordinates of the characteristic points on an image are utilized to solve the pose relationship between a camera and the targets, the robustness is good, the robustness and the reliability are high, and the precision of a measuring result is higher; the hardware system of the invention has simple structure, low cost and strong flexibility; and shielding can not be generated in the alignment process, and the real-time requirement is met. The method has the advantages of solving the close-range alignment problem and being suitable for application in industrial fields.

Drawings

FIG. 1 is a system diagram of a vision alignment method under multi-station operation of an industrial robot arm;

FIG. 2 is a flow chart of a vision alignment method under multi-station operation of an industrial robot arm;

FIG. 3 is a hand-eye calibration process diagram of a vision alignment method under multi-station operation of an industrial robot arm;

FIG. 4 is a cooperative target diagram of a visual alignment method under multi-station operation of an industrial robot arm;

FIG. 5 is a processed image of a visual alignment method target under multi-station operation of an industrial robot arm;

FIG. 6 is a diagram of convex hull point sequence adjustment of the visual alignment method under multi-station operation of an industrial robot arm;

fig. 7 is a data format of a robot arm in a visual alignment method under multi-station operation of an industrial robot arm.

Fig. 8 shows the number of times of posture adjustment of the vision alignment method in the multi-station operation of the industrial robot arm.

Detailed Description

The present invention will be further illustrated by the following specific examples, which are carried out on the premise of the technical scheme of the present invention, and it should be understood that these examples are only for illustrating the present invention and are not intended to limit the scope of the present invention.

According to the visual alignment method under the multi-station operation of the industrial mechanical arm, firstly, the position relation among coordinate systems is established by utilizing system calibration. Secondly, a set of cooperative targets is designed and arranged near the alignment target, a pose solving algorithm of the targets is given, and the expected pose of the targets can be obtained through the algorithm. An alignment task table is then built for the target. In the alignment link, the system executes alignment operation according to the task table, calculates the deviation between the actual pose and the expected pose of the target, calculates the deviation amount into motion data of the mechanical arm, drives the tail end to adjust the pose, and finally realizes the accurate alignment of the tail end tool and the target.

The application firstly discloses a multi-station alignment system based on visual guidance, and hardware equipment of the system comprises a mechanical arm, a camera, an end tool, a cooperation target and a plane calibration plate, wherein the end tool comprises a gripper, a screwing tool and the like, and the type of the end tool is shown in figure 1. The camera and the tail end tool are respectively fixedly connected to a flange at the tail end of the mechanical arm, the optical axis of the camera is approximately parallel to the central axis of the tail end tool, and the camera and the tail end tool form a Hand-Eye vision system in an Eye-in-Hand mode. The hand-eye relationship is obtained by system calibration and is kept unchanged in the whole operation process. Eight feature points are arranged on each cooperative target, and the spatial position constraint relationship between the feature points is accurately measured. And arranging the cooperative targets on the surface of the workpiece, wherein the target on each station corresponds to one target, and the position relation between each target and the corresponding target is not changed in the whole working process.

The coordinate system of the system is defined as follows: the coordinate system of the robot end flange comprises a world coordinate system { W }, a robot arm base coordinate system { B }, a robot arm end flange coordinate system { F }, a camera coordinate system { C }, and a cooperation target coordinate system { T }. The origin of the camera coordinate system is established at the center of the optical axis, Z_cThe axis is the optical axis of the camera, and the direction from the camera to the workpiece is the positive direction. The origin of the flange coordinate system is arranged on the flangeCenter, Z_fThe axis is the normal of the flange plane, and the direction from the flange to the workpiece is the positive direction. The system of the cooperative target needs to be established by means of measuring equipment and three-dimensional modeling software, the origin of a coordinate system is defined as the center of any one characteristic point, and the connecting line of the origin and the center of another characteristic point is defined as X_tAxis, the normal line of the plane fitted by the centers of the eight characteristic points is Z_tAxis, oriented outward of the vertical plane. All coordinate systems follow the right-hand rule.

The multi-station alignment system based on visual guidance completes the alignment work of the end tool and the target based on task driving, and the working flow is shown in fig. 2, and the specific steps are as follows:

step (1): system calibration

Firstly, calibrating TCP (tool center point) by utilizing a mechanical arm self program, setting the TCP as a flange center at the tail end of the mechanical arm, and obtaining a pose transformation relation of a mechanical arm base coordinate system relative to a flange coordinate system

Then camera calibration and hand-eye calibration are carried out, the camera calibration obtains the mapping relation between the camera coordinate system and the image coordinate system, and the hand-eye calibration obtains the pose transformation relation of the camera coordinate system relative to the flange coordinate system

The camera calibration is the key content of the system, and the camera calibration is used for associating a three-dimensional space with an image space and is the basis for subsequent hand-eye calibration and image processing. The most common camera model is the pinhole model, the general imaging mode is perspective projection, and second-order distortion compensation is considered. The present application uses an 11 x 9 array of large and small circular hole plane calibration plates.

The purpose of hand-eye calibration is to obtain the pose relationship between a camera coordinate system and a flange coordinate system so as to convert the pose of the cooperative target into a mechanical arm base coordinate system and calculate the motion data of the mechanical arm. For the Eye-in-Hand vision system of the application, the end tool fixedly connected with the flange is a 'Hand', the camera is an 'Eye', and the camera phase is solvedPose transformation relation to flange

The process of (2) is called mechanical arm hand-eye calibration. The basic idea of hand-eye calibration is to control the mechanical arm to move to different positions, and to use a known calibration reference object in the camera observation space to deduce

And multiple observations, fig. 3 shows the calibration process. The basic equation for hand-eye relationship is:

CX＝XD (1)

where X is the hand-eye relationship to be solved, i.e.

C is obtained by external parameters calibrated by a camera; d is the transformation matrix of the flange when position 1 moves to position 2, provided by the robot arm's own program. The present application uses a Tsai two-step method for calibration, the reference used being a planar calibration plate.

Step (2): designing a cooperative target, arranging the cooperative target near an alignment target, and solving the pose of the target; and acquiring the expected pose of the target, and establishing an alignment task table for the target.

1) Target design:

most of the surfaces of workpieces in the aerospace field do not have obvious textural features, and the difficulty in directly extracting the features of the target on the workpiece is high, so that the tasks such as positioning measurement and the like need to be completed by means of a cooperative target. In designing a cooperative target, circular features are typically used to implement the coding. The application designs a set of cooperative targets, and as shown in fig. 4, eight feature points are arranged on each target, wherein four feature points are connected to form a convex hull, and the remaining four feature points are located inside the convex hull. The feature points on the convex hull are numbered as 1, 2, 3 and 4 according to the clockwise sequence, the feature points in the convex hull are numbered as 5, 6, 7 and 8 respectively, the feature points No. 1-4 are used for calculating the homography matrix, and the feature points No. 5-8 are used for verifying the homography matrix.

2) Target feature extraction and pose solution:

before using the cooperative target, the position relation of the feature points needs to be measured by using related equipment, and the result is imported into three-dimensional modeling software for establishing a system, so as to obtain a target three-dimensional coordinate point set P ═

X

_i1, 2, 8, and this data is used as prior information to facilitate subsequent calculation of homography matrix and acquisition of expected pose.

And after the camera acquires the target image, image processing is required to obtain an accurate characteristic point center to form an image characteristic point set. Figure 5 shows the image processing of the target. Firstly, carrying out Gaussian filtering on an image to remove redundant noise; using a Canny operator to carry out edge detection, and storing the detected edge in a tree structure; obtaining possible feature point outlines according to area constraint and a roundness criterion, wherein the roundness criterion is as a formula (2), then continuously judging whether the gray value of pixel points on the diameter of each outline is continuous, and further screening to obtain correct feature point outlines; and fitting the minimum circumscribed rectangle of the feature point profile by adopting least square, identifying the center of the rectangle as the center of the feature point, and obtaining an image feature point set Q ═ x_j}，j＝1，2，...，8。

Where L represents the perimeter of the profile, a represents the area within the profile, and N represents the roundness.

After the image features are extracted, the corresponding relation between the image feature points and the target three-dimensional coordinate points needs to be established. The specific solving process is as follows:

1) finding the convex hull of the target three-dimensional coordinate point set P by using a convex hull algorithm, sequencing the points on the convex hull according to a clockwise order, and finally obtaining the point set P on the convex hull₁＝{

X

_i1, 2, 3, 4 and a set of points P within the convex hull₂＝{X_iI-5, 6, 7, 8, i denotes the number of the feature points. Repeating the operation on the image characteristic point set Q to obtain a point set Q on the convex hull₁＝{x_jJ ═ 1, 2, 3, 4, and set of points Q within the convex hull₂＝{x_j}，j＝5，6，7，8。

2) Using a set of points P₁And Q₁And calculating a homography matrix H by using four pairs of feature points with the numbers of 1-4, and performing projection transformation on the target three-dimensional coordinate point set by using the homography relation.

3) And traversing the image feature point set, and respectively calculating the Euclidean distance between each image feature point and the transformed target three-dimensional coordinate point. If the distance is smaller than the set value, the image characteristic point and the target three-dimensional coordinate point before transformation are a pair of corresponding points and are stored for numbers of the corresponding points. And if the distances are larger than the set value, giving up the target three-dimensional coordinate point, and continuously searching for a corresponding point for the next target three-dimensional coordinate point.

4) And if eight pairs of corresponding points are found finally, solving the homography matrix H correctly, and establishing a correct corresponding relation between the target three-dimensional coordinate point set and the image characteristic point set. When the finally found corresponding points are less than eight pairs, the homography matrix H is solved incorrectly, the number sequence of the feature points in the point set Q1 is adjusted (adjusted at most four times), and the steps (2) and (3) are repeatedly executed until eight corresponding points are found, as shown in FIG. 6.

Under the condition that the three-dimensional coordinate point set, the image characteristic point set and the camera model parameters of the target are known, calculating the pose of the cooperative target in the camera coordinate system can be summarized to solve a PNP (Positive-negative-point) problem, and the pose transformation relation of the target relative to the camera coordinate system, namely the pose transformation relation can be solved by utilizing a PNP algorithm

3) The method for establishing the task table comprises the following steps:

the robotic arm movements are controlled so that the end tool is pre-aligned with the alignment targets on the workpiece prior to the assembly operation. Each target corresponds to a cooperative target, and the expected pose transformation relation of the target corresponding to the target is obtained by using the pose solving algorithm and is recorded as

Then, an alignment task is established for the target, and the terminal tool information, the measured target three-dimensional coordinate point set and the expected pose are transformedMatrix array

Stored as a priori information in the task. And establishing tasks for the targets of all the stations to form a task list. In the subsequent operation process, the system drives the mechanical arm to adjust the pose according to the table to finish the alignment work.

In the alignment link, the system drives the end effector to complete the designated alignment work according to the task table. In the process of executing each alignment task, the control strategy adopted by the system is a hook-then-Move closed-loop control mode. Where the "Look" section includes camera acquisition of target images and pose estimation of the target and the "Move" section is the robotic arm movement to align the end tool with the target.

The camera acquires the image of the cooperative target and calculates the current pose of the target relative to the camera coordinate system

Calculate its expected pose

Error of (2)

And judging whether the error meets the technical requirements. If yes, the system judges that the current alignment task is finished, then the system searches the task table and continues to execute the next task. And if the error does not meet the technical requirements, calculating the pose of the mechanical arm to be reached next step, and guiding the mechanical arm to move.

The pose transformation relation of the current flange under the base coordinate system of the mechanical arm is set as

Pose transformation of the target with respect to the base coordinate system

Comprises the following steps:

pose transformation of flange under mechanical arm base coordinate system when end tool and target are aligned

The pose of the target is transformed relative to the base coordinate system at this time

Comprises the following steps:

because the position and attitude relationship between the target and the mechanical arm base coordinate system in the alignment process

Since no change occurs, the expression (4) and (5) can be used to calculate

Wherein,

reading the program of the mechanical arm;

is obtained by the calibration of the hands and the eyes,

and

performing reciprocal operation;

representing a current pose transformation relationship of the target relative to a camera coordinate system;

representing the expected pose transformation relationship of the target relative to the camera coordinate system, is directly read from the data of the task table,

and

are inverse operations of each other. All transformation matrices are known quantities and can be directly calculated

The mechanical arm has a positioning error, and the motion amount of the mechanical arm is interpolated according to the method and the device for improving the alignment precision of the system.

Contains two components: rotation matrix

And translation vector

And converting the rotation matrix into a quaternion to perform coordinate rotation interpolation, and performing position interpolation on the translation vector. The calculation process is as follows:

wherein,

and

A rotation matrix and a translation vector of (1);

and

Constructing new transformation matrices

Namely the pose of the mechanical arm needs to be adjusted.

In order to verify the effectiveness of the vision alignment system under the multi-station operation, a screw thread screwing and gripper clamping test platform is set up. The UR10 mechanical arm is used as a motion mechanism, the used visual devices are a German Allied Vision Prosilica GC1600 industrial camera and a large constant image HN-0914-2M-C2/3X fixed focus lens, and the focal length is 9 mm.

In the process of UR10 robot pose solving and controlling robot motion, the computer software system needs to read pose information from the robot itself and send related data to the robot, and the communication between the two is based on TCP/IP (transmission control/network protocol). The robot arm sends 1108 bytes of data from the communication port to the computer software system at 128Hz, the data format is shown in FIG. 7, in which 445 bytes areThe position and pose of the flange under a base coordinate system of the mechanical arm are 492 bytes, and the position and pose comprise position information x, y and z and pose information r_x，r_y，r_z. When the software system sends pose information to the mechanical arm to control the mechanical arm to move, the movement instruction format is as follows:

command([x，y，z，r_x，r_y，r_z]，v，a) (8)

the command is a mechanical arm motion mode, generally MoveJ, MoveL, MoveP and MoveC, and the MoveL linear motion mode is used in the application; [ x, y, z, r_x，r_y，r_z]The pose of the flange to be reached under a mechanical arm base coordinate system; v and a are the robot arm motion speed and acceleration, respectively, and v is 100mm/s and a is 30mm/s as used in this application²。

The experimental procedure was as follows:

(1) and setting TCP as the central point of the flange at the tail end of the mechanical arm, and establishing a conversion relation between the flange at the tail end and a base coordinate system of the mechanical arm.

(2) And sequentially carrying out camera calibration and hand-eye calibration. The planar calibration plate is fixedly placed on the platform, and the distance between the camera and the calibration plate is about 260 mm. And controlling the mechanical arm to move so that the mechanical arm makes 12 times of movement within the field of view of the camera, and acquiring the image of the calibration plate and the pose of the mechanical arm after each movement. The camera calibration results are shown in table 1, and the hand-eye calibration results are shown in formula (9).

TABLE 1 calibration results of the cameras

Internal reference of camera	Numerical value	Distortion ofParameter(s)	Numerical value
				α	2115.637	k₁	-0.289
β	2115.596	k₂	0.615
				u₀	828.534	p₁	-0.000730
v₀	632.431	p₂	0.000168

(3) And respectively establishing alignment tasks for the targets on each station, and finally forming a task table.

(4) In the alignment link, the tail end of the mechanical arm is in different initial positions, and alignment operation is executed according to a task table, wherein the alignment operation comprises thread screwing operation and automatic gripper clamping operation. Experimental results show that the multi-station alignment system can achieve accurate alignment of various targets and meet assembly requirements of industrial fields.

To verify the accuracy that can be achieved by the alignment system, the present application designed three sets of experiments as follows.

(1) Precision verification experiment 1:

the alignment accuracy is required to reach 0.5mm in the direction X, Y, 1mm in the Z direction and an angleAnd when the error is not more than 0.1 degree, selecting a group of target deviation between the actual pose and the expected pose, and showing in a table 2. Wherein, the delta X, the delta y and the delta z respectively represent the actual pose of the target to be compared with the expected pose along the X direction under the camera coordinate system_cAxis, Y_cAxis, Z_cThe displacement deviation amount of the axis, delta η represents the included angle between the normal direction of the target actual pose and the normal direction of the expected pose in the camera coordinate system, and delta α, delta β and delta gamma respectively represent the target actual pose and the expected pose around X in the camera coordinate system_cAxis, Y_cAxis, Z_cThe difference in the rotational angle of the shaft.

In the process of aligning the end tool with the target, firstly, adjusting the posture part, wherein the posture adjustment causes the target posture to change in the axial direction; then, three axial alignments are considered, and axial adjustments affect the attitude change, and the position and attitude adjustments are coupled. After the 6 th adjustment, the angle deviation delta eta reaches 0.026 degrees, the requirement of attitude precision is met, but the deviation of the X, Y direction is overlarge. The adjustment of the axial deviation increases Δ η to 0.091 °, but still meets the accuracy requirement. Thus, the robotic arm achieved alignment of the end tool to the target through 7 automatic adjustments.

(2) Precision verification experiment 2:

when the alignment accuracy requirement reaches 0.25mm in the X, Y direction, reaches 0.5mm in the Z direction and the attitude error does not exceed 0.1 degree, the deviation between the actual attitude and the expected attitude of a group of targets is selected, as shown in Table 3. As can be seen from the table, the first 6 times of adjustment of the system gradually approaches the expected pose, the 7 th time of adjustment reduces the angular deviation, the delta eta reaches 0.040 degrees, the 8 th time of adjustment of the axial deviation is carried out, the angular deviation delta eta is influenced and increased to 0.098 degrees but still better than 0.1 degrees, and finally the alignment is realized through 8 times of automatic adjustment.

(3) Precision verification experiment 3:

when the position accuracy is required to reach 0.1mm in the X, Y direction, 0.2mm in the Z direction and the angle error is not more than 0.1 degrees, a group of data is selected for analysis, and the table 4 shows. The previous 8 times of adjustment of the system approaches to the vicinity of the expected pose, and only the precision in the X direction does not meet the requirement after the 10 th adjustment. In the subsequent fine adjustment process, under the influence of the self-positioning precision and the position and posture coupling effect of the robot arm, the system deviates from the expected pose, and finally the alignment is realized through automatic adjustment for 12 times.

In addition to alignment accuracy, the present application also makes statistics of positioning efficiency. The accuracy requirements were divided into five levels according to table 5, 5 sets of alignment tests were performed under each accuracy level condition, and the average number of pose adjustments was counted, with the results shown in fig. 8. Compared with the first four precision levels, under the precision condition of level five, the average number of times of automatic posture adjustment of the mechanical arm is increased to 14, namely, the mechanical arm needs more times of fine adjustment to achieve high precision.

TABLE 2 deviation of target actual pose from expected pose (experiment 1)

	Δx/mm	Δy/mm	Δz/mm	Δη/(°)	Δα/(°)	Δβ/(°)	Δγ/(°)
								0	-53.566	51.819	54.748	19.698	9.492	17.252	0.518
1	-45.656	47.975	15.863	3.973	1.821	3.530	0.092
								2	-12.129	10.276	7.859	3.830	1.583	3.487	0.071
3	-9.445	8.758	3.601	0.797	0.278	0.745	0.041
								4	-2.525	1.846	1.572	0.771	0.214	0.740	0.030
5	-2.075	1.918	1.156	0.223	0.101	0.198	0.020
								6	-1.714	1.817	0.924	0.026	0.020	0.017	0.001
7	-0.414	-0.093	0.274	0.091	-0.087	0.029	0.004

TABLE 3 deviation of target actual pose from expected pose (experiment 2)

	Δx/mm	Δy/mm	Δz/mm	Δη/(°)	Δα/(°)	Δβ/(°)	Δγ/(°)
								0	-45.196	67.901	44.479	12.420	5.382	11.119	-1.290
1	-44.669	64.972	21.092	2.611	1.161	2.324	-0.262
								2	-10.235	12.653	8.271	2.428	0.784	2.280	-0.285
3	-9.326	12.083	6.195	0.539	0.188	0.503	-0.050
								4	-1.993	2.349	2.275	0.460	0.107	0.446	-0.040
5	-1.607	2.279	2.020	0.090	0.025	0.085	-0.011
								6	-0.948	0.364	0.571	0.219	-0.010	0.219	0.001
7	-0.400	0.298	0.551	0.040	-0.026	-0.030	-0.005
								8	0.026	-0.227	0.047	0.098	-0.090	-0.038	0.009

TABLE 4 deviation of target actual pose from expected pose (experiment 3)

	Δx/mm	Δy/mm	Δz/mm	Δη/(°)	Δα/(°)	Δβ/(°)	Δγ/(°)
								0	49.954	39.415	117.568	21.226	11.843	-17.523	1.798
1	63.047	42.904	72.464	4.274	2.211	-3.637	0.388
								2	14.527	9.162	26.568	4.110	1.922	-3.612	0.391
3	14.027	8.454	21.938	0.965	0.470	-0.841	0.054
								4	3.123	1.635	7.687	0.832	0.346	-0.755	0.053
5	2.833	1.393	6.948	0.193	0.072	-0.179	0.006
								6	0.565	0.357	2.398	0.149	0.073	-0.130	0.005
7	0.197	0.256	0.948	0.162	0.101	-0.126	0.011
								8	0.116	-0.076	0.329	0.115	0.033	-0.109	0.016
9	0.430	0.294	0.259	0.205	0.088	-0.182	0.028
								10	-0.237	0.060	-0.003	0.083	-0.006	0.083	0.005
11	0.289	-0.056	-0.023	0.023	-0.021	-0.010	0.003
								12	0.072	0.052	-0.128	0.033	-0.004	-0.033	0.001

TABLE 5 precision rankings

Grade	Δx/mm	Δy/mm	Δz/mm	Δη/(°)
					1	0.5	0.5	1.0	0.1
2	0.25	0.25	0.5	0.1
					3	0.2	0.2	0.4	0.1
4	0.15	0.15	0.3	0.1
					5	0.1	0.1	0.2	0.1

The foregoing is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, various modifications and decorations can be made without departing from the principle of the present invention, and these modifications and decorations should also be regarded as the protection scope of the present invention.

Claims

1. a visual alignment method under the multi-station operation of an industrial manipulator, is characterized in that: at first, the positional relationship between each coordinate system is established by means of system calibration; secondly, a set of cooperative targets is designed and arranged in Align the vicinity of the station and solve the target pose; obtain the desired pose of the target, and then target multiple stations around the target. Multiple stations can establish multiple alignment tasks to form an alignment task table; The task table is based on the alignment operation, calculates the deviation between the actual pose and the desired pose of the target, calculates the deviation into the motion data of the robotic arm, drives the end to adjust the pose, and finally realizes the precise alignment between the end tool and the target. allow.

2. The visual alignment method under the multi-station operation of an industrial manipulator according to claim 1, characterized in that: in the system calibration, TCP calibration is first performed to obtain the transformation of the flange coordinate system relative to the base coordinate system of the manipulator. relation

Then perform camera calibration and hand-eye calibration. Camera calibration obtains the mapping relationship between the camera coordinate system and the image coordinate system, and hand-eye calibration obtains the transformation relationship between the camera coordinate system and the flange coordinate system.

3. The visual alignment method under the multi-station operation of an industrial manipulator according to claim 1, wherein a plurality of feature points are arranged on each target, wherein several feature points are connected to form a convex hull, and the remaining feature points Inside the convex hull, the feature points on the convex hull are numbered in clockwise order. The feature points on the convex hull are used to calculate the homography matrix, and the feature points inside the convex hull are used to verify the homography matrix. After processing, a set of image feature points is obtained, the unique identity number of each point on the target is established based on the convex hull rule and the principle of homography transformation, and the corresponding relationship between the image feature points and the three-dimensional feature points of the target is established through the number; the PNP algorithm is used to calculate the The transformation relationship between the target coordinate system and the camera coordinate system is

4. the visual alignment method under the multi-station operation of the industrial manipulator according to claim 3, is characterized in that: the image processing process of the target is: first, the image is subjected to Gaussian filtering to remove excess noise; Edge detection, store the detected edges in a tree structure; obtain possible feature point contours according to area constraints and roundness criteria, and then judge whether the gray values of pixels on the diameter of each contour are continuous, and further filter to obtain the correct The feature point contour is obtained; the minimum circumscribed rectangle of the feature point contour is fitted by least squares, and the center of the rectangle is identified as the feature point center, and the image feature point set is obtained.

5. The visual alignment method under the multi-station operation of an industrial manipulator according to claim 1, wherein the method for establishing a task table is: controlling the motion of the manipulator so that the end tool is aligned with each station before the assembly operation. The target is pre-aligned, the camera collects the image of the cooperative target corresponding to the target, and the desired pose transformation relationship of the target relative to the camera coordinate system is obtained by calculation, denoted as

Then establish an alignment task for the target, and convert the end tool information, the measured three-dimensional coordinate point set of the target and the desired pose transformation relationship

As a priori information, it is stored in the task, and tasks are established for the goals of all workstations to form a task table.

6. The visual alignment method under the multi-station operation of an industrial manipulator according to claim 1, wherein the camera collects the current image of the target and calculates the current pose transformation relationship of the target relative to the camera coordinate system.

Then calculate the current pose transformation relationship

Transformation relationship with the desired pose

error

If the error is greater than the set threshold, the error is converted into the base coordinate system of the manipulator based on the system calibration result, the motion data of the manipulator is calculated, and the manipulator is guided to move.

7. The visual alignment method under the multi-station operation of an industrial manipulator according to claim 6, characterized in that: when the manipulator moves to a new position, the image is collected again and the pose error is calculated until

If it is less than the set threshold, the alignment task is completed; look up the task table to determine whether all tasks are completed, if not, continue to execute the next task until all tasks are completed.

8. The visual alignment method under the multi-station operation of an industrial manipulator according to claim 6, characterized in that: when the end tool is in an aligned state with the target, the position and orientation of the flange under the base coordinate system of the manipulator is transformed. relationship is

When the robot arm adjusts the pose to reach

When the pose relationship between the target and the camera reaches the desired pose, the end tool is aligned with the target.

9. The visual alignment method under the multi-station operation of an industrial manipulator according to claim 6, wherein the interpolation of the positioning error is carried out to the motion of the manipulator,

contains two components: the rotation matrix

and translation vector

Convert the rotation matrix to a quaternion for coordinate rotation interpolation, and perform position interpolation on the translation vector. The calculation process is as follows: