CN113487510A - Method, system and equipment for detecting needle point position for automatic liquid preparation of robot - Google Patents

Abstract

The invention belongs to the field of image detection, and in particular relates to a needle tip position detection method, system and equipment for automatic liquid dispensing by a robot. The correct guidance of the robotic arm causes the needle to stick to the mouth of the vial, which leads to the problem of low dispensing efficiency or even failure of dispensing. The method includes collecting the image of the needle to be detected; preprocessing the image; extracting the ROI area of the image; obtaining the edge contour of the needle in the ROI area by an edge detection method, and determining the position of the needle tip through corner point detection to obtain the pixel coordinates of the needle tip ; Combining the pixel coordinates of the needle tips in two different orientations, the world three-dimensional coordinates corresponding to the needle tip are obtained by the preset first pose solution method; the liquid is dispensed according to the obtained world three-dimensional coordinates. The invention reduces the needle sticking error and improves the liquid dispensing efficiency and accuracy of the liquid dispensing robot.

Description

Needle tip position detection method, system and equipment for automatic liquid dispensing by robot

technical field

The invention belongs to the field of image detection, and in particular relates to a needle tip position detection method, system and equipment for automatic liquid dispensing by a robot.

Background technique

In the dispensing robot, it is a necessary process for the syringe to pierce the medicines such as vials, ampoules, and infusion bags. However, there are many types of needles on the market, and the types of needles are also different. Moreover, due to the soft materials of ordinary medical syringes and needle tips, the needles may be twisted and deformed under the grip of the robotic arm or even in a natural state, which is very easy. Errors occurred when taking vials. The diameter of the bottle mouth that can be pierced from the vial is between 1-2mm, which leads to the fact that when the robotic arm carries the needle tube, it is very easy to pierce the vial, and the liquid extraction is unsuccessful. At present, liquid dispensing robots at home and abroad, such as WEINAS of Weibang Company in Shenzhen, mostly use a special needle tube to extract liquid medicine. As a medical consumable, this kind of needle tube is bound to increase unnecessary medical costs for hospitals and patients. Based on this, the present invention proposes a needle tip position detection method for automatic liquid dispensing by a robot.

SUMMARY OF THE INVENTION

In order to solve the above problems in the prior art, that is, in order to solve the problem that the existing liquid dispensing robot can not determine the position of the needle during the dispensing process, the robot arm can be correctly guided, causing the needle to stick to the mouth of the vial, and then In the first aspect of the present invention, a method for detecting the position of a needle tip for automatic liquid dispensing by a robot is proposed, which includes:

S10, collecting the image of the needle to be detected under the setting of the color type light source and the setting of the lighting direction of the light source through two cameras in different orientations, as an input image;

S20, preprocessing the input image to obtain a preprocessing image; the preprocessing includes denoising, binarization, and histogram equalization;

S30, using a preset corner-based ROI extraction method to extract the ROI region in the preprocessed image;

S40, obtain the edge contour of the needle tip in the ROI area through an edge detection method, and determine the position of the needle tip through corner point detection, and then obtain the pixel coordinates of the needle tip;

S50, combining the pixel coordinates of the needle tips in two different orientations, obtain the three-dimensional world coordinates corresponding to the needle tip through a preset first pose solution method; the first pose solution method is to solve the pixel points according to the pose results of the two images method of world coordinates;

S60, calculating the error between the world three-dimensional coordinates obtained in step S50 and the standard world three-dimensional coordinates calculated by the robot before starting the liquid dispensing, and correcting the world three-dimensional coordinates obtained in step S50 according to the error, and then the robot performs the correction based on the corrected world three-dimensional coordinates. Dosing.

In some preferred embodiments, before step S10, it also includes the steps of selecting the specification of the calibration board and calibrating the internal and external parameters of the camera:

Select a checkerboard with an even-numbered column as an odd number or an odd-numbered column with an even number as the calibration board, and when the camera collects a clear calibration board image, the area of the calibration board is at least half of the available pixel area; the available pixel area is the collected pixel area. The pixel area of the entire calibration plate image;

After selecting the specifications of the calibration board, based on the conversion relationship between the world coordinate system, the camera coordinate system, and the pixel coordinate system, use Opencv to calibrate the camera's internal and external parameters;

Among them, the conversion relationship between the world coordinate system, the camera coordinate system, and the pixel coordinate system is:

、

代表像素坐标系坐标，

、

、

代表世界坐标系坐标，

、

、

代 表相机坐标系坐标，

、

分别代表相机坐标系相对于世界坐标系的旋转矩阵、平移矩阵，

、

为相机的焦距，

、

代表图像的中心像素坐标和图像原点像素坐标之间相差的横向 和纵向像素数，

、

、

为相机光心的世界坐标系坐标，

表示转置，

表示相机内参。 in,

,

represents the coordinates of the pixel coordinate system,

,

,

represents the coordinates of the world coordinate system,

,

,

Represents the camera coordinate system coordinates,

,

respectively represent the rotation matrix and translation matrix of the camera coordinate system relative to the world coordinate system,

,

is the focal length of the camera,

,

The number of horizontal and vertical pixels representing the difference between the center pixel coordinates of the image and the pixel coordinates of the image origin,

,

,

is the world coordinate system coordinate of the camera optical center,

means transpose,

Indicates camera internal parameters.

In some preferred embodiments, the setting of the color type light source and the setting of the lighting direction of the light source are:

The set color type light source is a red light source;

The set light source lighting direction is back lighting.

In some preferred embodiments, "the ROI region in the preprocessed image is extracted by a preset corner-based ROI extraction method", and the method is:

Corner detection is performed on the preprocessed image through the corner detection function, and the minimum circumscribed rectangle corresponding to all corners is used as the ROI area.

In some preferred embodiments, the edge detection method is a Canny operator edge detection method.

In some preferred embodiments, "acquiring the edge contour of the needle in the ROI area by an edge detection method, and determining the position of the needle tip by detecting a corner point", the method is:

Obtain the edge contour of the needle in the ROI area by an edge detection method;

Extract the vertical lines in the edge contour, and calculate the maximum distance between the vertical lines in the horizontal direction;

Determine whether the maximum distance is greater than the set distance threshold, and if it is greater, delete the ROI area above the horizontal direction;

Extract the vertical line in the undeleted ROI area and perform corner detection, and use the bottom corner of the longest vertical line in the undeleted ROI area as the position of the needle tip.

In some preferred embodiments, the world three-dimensional coordinate of the pixel point in the first pose solution method is the midpoint of the coordinates of the closest points of the two straight lines.

In the second aspect of the present invention, a needle tip position detection system for automatic liquid dispensing by a robot is proposed. The system includes: an image acquisition module, a preprocessing module, an ROI region extraction module, a detection module, a world three-dimensional coordinate solution module, and a configuration module. liquid module;

The image acquisition module is configured to collect the image of the needle to be detected as the input image through two cameras in different orientations under the set color type light source and the set light source illumination direction;

The preprocessing module is configured to preprocess the input image to obtain a preprocessed image; the preprocessing includes denoising, binarization and histogram equalization;

The ROI region extraction module is configured to use a preset corner-based ROI extraction method to extract the ROI region in the preprocessed image;

The detection module is configured to obtain the edge contour of the needle tip in the ROI area through an edge detection method, and determine the position of the needle tip through corner detection, thereby obtaining the pixel coordinates of the needle tip;

The world three-dimensional coordinate solving module is configured to combine the pixel coordinates of the needle tips in two different orientations, and obtain the world three-dimensional coordinates corresponding to the needle tip through a preset first pose solution method; the first pose solution method is based on two images. The method of solving the world coordinate of the pixel point from the pose result of the image;

The liquid dispensing module is configured to calculate the error between the world three-dimensional coordinates obtained by the world three-dimensional coordinate solving module and the standard world three-dimensional coordinates calculated by the robot before starting liquid dispensing, and correct the world three-dimensional coordinates obtained by the world three-dimensional coordinate solving module according to the error. coordinates, and then the robot dispenses liquid based on the corrected three-dimensional coordinates of the world.

A third aspect of the present invention provides an electronic device, at least one processor; and a memory communicatively connected to at least one of the processors; wherein, the memory stores instructions executable by the processor, The instructions are used to be executed by the processor to implement the needle tip position detection method for automatic liquid dispensing by a robot as claimed in the preceding claims.

In a fourth aspect of the present invention, a computer-readable storage medium is provided, where computer instructions are stored in the computer-readable storage medium, and the computer instructions are used to be executed by the computer to realize the above-mentioned automatic application for robots in the claims. A method for detecting the position of the needle tip for liquid dispensing.

Beneficial effects of the present invention:

The invention reduces the needle sticking error and improves the liquid dispensing efficiency and accuracy of the liquid dispensing robot.

After the detection of the position of the needle tip is completed, the present invention calculates the position deviation of the curved needle tip compared with the needle tip that is normally plucked, so as to provide guiding data for the robot manipulator to pluck the medicine bottle, extract the liquid correctly and accurately, and provide the whole robot system with guiding data. Accurate and orderly operation provides a strong guarantee. So as to save manpower, save medical consumables, and achieve a fully automatic liquid dispensing process with accurate dosage.

Description of drawings

Other features, objects and advantages of the present application will become more apparent upon reading the detailed description of non-limiting embodiments taken with reference to the following drawings.

1 is a schematic flowchart of a needle tip position detection method for automatic liquid dispensing by a robot according to an embodiment of the present invention;

2 is a schematic diagram of the frame of a needle tip position detection system for automatic liquid dispensing by a robot according to an embodiment of the present invention;

3 is a schematic flow chart of a needle tip position correction according to an embodiment of the present invention;

FIG. 4 is a schematic flow chart of needle tip position determination according to an embodiment of the present invention;

FIG. 5( a ) is a schematic flowchart of the calibration of internal and external parameters of a camera according to an embodiment of the present invention;

Fig. 5(b) is a schematic diagram of a calibration plate shot by two cameras respectively according to an embodiment of the present invention;

6 is a schematic diagram of forward lighting and back lighting according to an embodiment of the present invention;

Figure 7(a) is a schematic diagram of a needle image after denoising and histogram equalization according to an embodiment of the present invention;

Figure 7(b) is a schematic diagram of a binarized needle image according to an embodiment of the present invention;

Fig. 7(c) is a schematic diagram of an extracted ROI region according to an embodiment of the present invention;

FIG. 8( a ) is a schematic diagram of a region of interest after edge detection according to an embodiment of the present invention;

FIG. 8( b ) is a schematic diagram of extracting vertical lines in the region of interest after edge detection according to an embodiment of the present invention;

FIG. 8( c ) is a schematic diagram of a reduced-range edge image according to an embodiment of the present invention;

Figure 8(d) is a schematic diagram of the detected needle tip position according to an embodiment of the present invention;

Fig. 9(a) is a schematic diagram of the coordinate principle of finding a common point by two cameras according to an embodiment of the present invention;

在相机坐标系下的示意图； Figure 9(b) is the point of an embodiment of the present invention

Schematic diagram in the camera coordinate system;

Fig. 9(c) is a schematic diagram of two straight line corner points according to an embodiment of the present invention;

FIG. 10 is a schematic structural diagram of a computer system suitable for implementing the electronic device of the embodiment of the present application according to an embodiment of the present invention.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not All examples. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

The present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the related invention, but not to limit the invention. In addition, it should be noted that, for the convenience of description, only the parts related to the related invention are shown in the drawings.

It should be noted that the embodiments in the present application and the features of the embodiments may be combined with each other in the case of no conflict.

A method for detecting the position of a needle tip for automatic liquid dispensing by a robot of the present invention, as shown in FIG. 1 , includes:

S10, collecting the image of the needle to be detected under the setting of the color type light source and the setting of the lighting direction of the light source through two cameras in different orientations, as an input image;

S20, preprocessing the input image to obtain a preprocessing image; the preprocessing includes denoising, binarization, and histogram equalization;

S30, using a preset corner-based ROI extraction method to extract the ROI region in the preprocessed image;

S40, obtain the edge contour of the needle tip in the ROI area through an edge detection method, and determine the position of the needle tip through corner point detection, and then obtain the pixel coordinates of the needle tip;

S50, combining the pixel coordinates of the needle tips in two different orientations, obtain the three-dimensional world coordinates corresponding to the needle tip through a preset first pose solution method; the first pose solution method is to solve the pixel points according to the pose results of the two images method of world coordinates;

S60, calculating the error between the world three-dimensional coordinates obtained in step S50 and the standard world three-dimensional coordinates calculated by the robot before starting the liquid dispensing, and correcting the world three-dimensional coordinates obtained in step S50 according to the error, and then the robot performs the correction based on the corrected world three-dimensional coordinates. Dosing.

In order to more clearly describe the needle tip position detection method for robot automatic liquid dispensing of the present invention, each step in an embodiment of the method of the present invention will be described in detail below with reference to the accompanying drawings.

S10, collecting the image of the needle to be detected under the setting of the color type light source and the setting of the lighting direction of the light source through two cameras in different orientations, as an input image;

In this embodiment, before the needle image is acquired by the camera, the specifications of the checkerboard (that is, the calibration board) are selected first, and then the camera is calibrated with the internal and external parameters of the camera, that is, based on the world coordinate system, the camera coordinate system, and the pixel coordinate system. The conversion relationship of the camera is calibrated by OpenCV.

The calibration board is used in machine vision, image measurement, photogrammetry, three-dimensional reconstruction and other applications to correct lens distortion, determine the conversion relationship between physical size and pixels, and determine the three-dimensional geometric position of a point on the surface of a space object and its corresponding point in the image. The relationship between them provides important reference information for establishing the geometric model of camera imaging.

Checkerboard is the most popular and common pattern design, such as the black and white pattern in Figure 5(b). Generally, by binarizing the camera (or camera) image and finding candidate points for the corners of the checkerboard, filtering and retaining those quadrilaterals that meet certain size criteria, and organizing them in a regular grid structure, the size of the grid structure is the same as User-specified size matches. After a preliminary inspection of the calibration plate, the position of the corner points can be determined with very high accuracy. This is because angles (mathematically: saddle points) are essentially infinitesimally small and therefore unbiased under perspective transformation or lens distortion.

Because the camera needs to be focused at a specific distance for calibration, the physical size of the calibration plate is an important consideration. For accurate calibration, the camera model is preferably constrained when the camera sees the calibration target filling most of the image.

According to experience, in order to reduce the error of camera calibration as much as possible, the selection of checkerboard specifications should refer to the following points:

(1) When a clear calibration plate image is observed from the front, the area of the calibration plate should be at least half of the available pixel area (usually the pixel area of the entire image collected);

(2) To keep the rotation constant, the number of rows must be even, and the number of columns must be odd, or vice versa. For example, if both are even, there is an ambiguity of a 180 degree rotation. For single-camera calibration, this is not a problem, but if the same point needs to be recognized by two or more cameras (for stereo calibration), this ambiguity must not exist;

To sum up, and according to the real situation, the distance between the camera and the needle tip is about 5-20cm. In order to accurately calibrate the internal and external parameters of the camera, we select a 7*5 checkerboard calibration board, and the side length of each grid is 12mm.

After selecting the specifications of the calibration board, use OpenCV to calibrate the camera's internal and external parameters, as shown in Figures 5(a) and 5(b). The specific calibration process is as follows:

a) Internal parameter calibration. According to the actual distance from the camera to the subject, after adjusting the focal lengths of the two cameras (or cameras), use the camera to collect pictures of the calibration board in various orientations and shapes of the camera, and use OpenCV to obtain the internal parameters of the camera;

b) Fix the camera. The needle is kept vertical, the ideal vertical direction of the needle is taken as the Z-axis direction, the central axes of the two cameras are perpendicular to the Z-axis, and the angle between the central axes of the two cameras is 90 degrees, and the shooting plane is in the Z-axis direction. Parallel, that is, adjust the camera position;

c) External parameter calibration. Place the calibration plate perpendicular to the Z-axis direction, so that both cameras can capture the calibration plate and the needle at the same time, and then use OpenCV to calibrate the external parameters of the cameras in both directions, obtain the external parameters of the two cameras, and combine them with The robotic arm of the robot has a unified coordinate system.

Among them, the conversion relationship between the world coordinate system, the camera coordinate system, and the pixel coordinate system is:

（1）

(1)

（2）

(2)

（3）

(3)

（4）

(4)

、

代表像素坐标系坐标，

、

、

代表世界坐标系坐标，

、

、

代 表相机坐标系坐标，

、

分别代表相机坐标系相对于世界坐标系的旋转矩阵、平移矩阵，

、

为相机的焦距，

、

代表主点坐标，即成像平面，

、

、

为相机

的世界坐标 系坐标，

表示转置，

表示相机的内参。 in,

,

represents the coordinates of the pixel coordinate system,

,

,

represents the coordinates of the world coordinate system,

,

,

Represents the camera coordinate system coordinates,

,

respectively represent the rotation matrix and translation matrix of the camera coordinate system relative to the world coordinate system,

,

is the focal length of the camera,

,

represents the coordinates of the principal point, that is, the imaging plane,

,

,

for the camera

the world coordinate system coordinates,

means transpose,

Indicates the internal parameters of the camera.

In the pinhole model, the direction of a straight line on the imaging plane should still be a straight line, but in the actual shooting process, since the shooting equipment uses lenses, the straight line is often projected as a curve, and the closer it is to the edge of the image, the more obvious it is. Lenses tend to be centrosymmetric, so that this irregular distortion is usually radially symmetric, collectively referred to as radial distortion.

为畸变后的像素坐标，

表示理想的无畸变的像素坐标，

为畸变后归一化图像坐标，

表示 理想的无畸变的归一化图像坐标，则径向畸变图像坐标为： When calibrating the internal reference, the present application also includes the step of eliminating distortion.

is the distorted pixel coordinates,

represents ideal undistorted pixel coordinates,

is the normalized image coordinates after distortion,

Representing the ideal undistorted normalized image coordinates, the radially distorted image coordinates are:

（5）

(5)

和

表示径向畸变的系数；in,

and

is a coefficient representing radial distortion;

The center position of the radial distortion is the same as the camera's main center position, as follows:

（6）

(6)

、

表示图像在

和

方向上单位距离上像素的个数，

是扭曲参数，表示 像素坐标系两个坐标轴的扭曲； in,

,

Indicates that the image is in

and

The number of pixels per unit distance in the direction,

is the distortion parameter, which represents the distortion of the two coordinate axes of the pixel coordinate system;

，则径向畸变像素坐标为： Assume

, the radial distortion pixel coordinates are:

（7）

(7)

which is:

（8）

(8)

There are n images, each image has m points, then there are 2mn radial distortion pixel coordinates, and the distortion pixel coordinate matrix is:

（9）

(9)

表示约束方程系数矩阵，

表示畸变系数，

表示等式又短非齐次项； in,

represents the constraint equation coefficient matrix,

represents the distortion coefficient,

Represents an equation and a short inhomogeneous term;

Based on the distorted pixel coordinate matrix, the distortion coefficients are obtained by the least squares method:

（10）

(10)

The objective function is to minimize the reprojection error, and the spatial coordinates are projected onto the image according to the estimated projection equation, so that the error between the pixel estimated value and the actual observed value is minimized, and the objective function is optimized by the maximum likelihood estimation method:

（11）

(11)

表示

的像素点，

表示相机内参，

和

表示 第

幅图像上对应相机的旋转矩阵和平移矩阵，

表示第

幅图像上第

个像点对应的标 定板上的三维点； in,

express

pixels,

Indicates the camera internal parameters,

and

means the first

The rotation matrix and translation matrix of the corresponding camera on the image,

means the first

the top of the image

3D points on the calibration board corresponding to each image point;

Based on the objective function, through the LM algorithm, the optimal solution of the objective function is obtained iteratively with the solution of the distortion coefficient as the initial value.

The objective function is iteratively optimized by maximum likelihood estimation and LM algorithm, and the optimal solution is obtained, which can reduce the calibration error caused by noise, obtain more accurate internal and external parameters of the camera, and improve the accuracy of the final calculation of the world coordinates.

In the process of obtaining the internal and external parameters of the camera, it is necessary to minimize the calibration error and improve the accuracy of the final calculation of the three-dimensional coordinates. In addition to the above introduction, according to the focal length of the camera and the distance from the object to the camera, selecting the appropriate calibration plate specifications can reduce the calibration error accordingly, and there are other ways to reduce the calibration error from other aspects.

The ways to reduce the calibration error in the present invention mainly include:

(1) The plane of the calibration plate should be kept flat. The calibration board is a two-dimensional image, whether it is manually printed or an existing calibration board, all checkerboards need to be kept on the same plane;

(2) When taking pictures with internal parameter calibration, it is better for the camera to take pictures from different perspectives of the calibration board, and the photos should cover every position of the calibration board, and the number of pictures should be more than 20;

(3) When calibrating the internal parameters, the re-projection (that is, converting the points of the world coordinate system into pixel coordinates through the acquired internal and external parameters of the camera) is excluded from the calibration images with large errors (that is, greater than the set error threshold). The re-projection error of the calibration image is used to determine the calibration effect. The calibration image with a large re-projection error is often inaccurate in corner detection, which affects the calibration.

In addition, the process of forming images of objects in the three-dimensional objective world by using the principle of optical imaging and processing them into a format that can be recognized by computers is called image acquisition. In the process of image acquisition, illumination plays an important role, which can directly affect the quality of the formed image and the application efficiency of the image. Light sources can be classified into ordinary natural light and artificial light sources, which are described by light intensity, color temperature and the geometry of the light source. In order to make the collected images meet the high-quality requirements, it is necessary to consider the intensity, light path and spectrum of the required light source according to the color, material and shape of the target to be detected.

The visual light source has white, blue, red, green, infrared, ultraviolet and other colors. The wavelength of each color is different, and it is also suitable for use in different scenarios. White light sources are usually light sources with wide applicability and high brightness, and are more often used in the scene of shooting color images. Blue light sources are widely used in metal products. Red light sources can pass through some darker objects and are often used to improve contrast. The green light source mainly needles red and silver background products. Infrared light belongs to invisible light, but it has strong transmittance, and it is generally used in LCD screen detection and video surveillance industries. Ultraviolet light has strong penetrating power and is mainly used in document detection, touch screen ITO detection, cloth surface damage, glue spill detection, etc., metal surface scratch detection and other fields.

The most commonly used lighting methods for machine vision lighting systems are forward lighting and back lighting, as shown in Figure 6. The forward illumination light source is placed in front of the object and is mainly used to illuminate the surface defects and detailed features of the object. The back-illuminated light source is placed behind the object to improve the contrast between the target and the surrounding environment, and can highlight the contour of the object and the position of the liquid level. It is mainly used in precision measurement systems.

The needle image collected in the dosing room is often disturbed by many background elements, and the needle tip itself is a relatively small target. Therefore, it is preferable to use a red backlight to illuminate the needle of the syringe to eliminate the interference of background cluttered objects, and at the same time Highlight the outline of the needle. That is, after the internal and external parameters of the camera are calibrated, two cameras with different orientations are used to collect the image of the needle to be detected under the set color type light source and the set light source illumination direction as the input image.

S20, preprocessing the input image to obtain a preprocessing image; the preprocessing includes denoising, binarization, and histogram equalization;

The image sensor in the camera, the transmission process of the image signal, and the complexity of the light source and background in the collection site will all lead to a lot of noise in the collected image. Therefore, during image preprocessing, we need to denoise the image first. In this visual detection scheme, edge detection is required for the detection of the needle tip, which is a very important link. Therefore, in order to preserve the edge information as much as possible during the denoising process. Therefore, in this embodiment, the bilateral filtering method is used to denoise the needle image, which can effectively remove noise while keeping the boundary clear.

After denoising, the needle image is equalized. During the equalization process, the preservation of boundary information is also considered. Therefore, in this embodiment, histogram equalization is performed on the needle image. The equalized image is shown in Fig. 7(a).

In addition, in the processing of liquid dispensing images, we pay more attention to the edge information of objects. Binarized images can already meet the requirements, and various processing speeds based on binary images have also been greatly improved. In the present invention, we use the dynamic threshold method in OpenCV, which automatically selects different thresholds for different pixels under the premise of considering the coordinate position relationship of the pixels to realize the dynamic threshold method. The dynamic threshold binarization results are shown in Fig. 7(b).

S30, using a preset corner-based ROI extraction method to extract the ROI region in the preprocessed image;

In this embodiment, only the needle site is the area we need to pay attention to. The extraction of ROI not only reduces the interference of the background area features to a certain extent, but also improves the efficiency of the detection algorithm. From the grayscale image 7(b), we can see that the area of the backlight source is the image we want to extract, that is, the area composed of the four corners of the light source.

In the present invention, a preset corner-based ROI extraction method is used to extract the ROI region in the preprocessed image. The principle is that the grayscale of the image changes in two directions perpendicular to the two edges of the corners of the backlight source. If such corner points are found, then the ROI area of the image can be drawn, that is, the corner points are obtained by using the corner detection function in OpenCV, and then the minimum circumscribed rectangle corresponding to all the corner points can be obtained. The ROI area of the point, the extraction result is shown in Figure 7(c). This method can adaptively extract the ROI area according to the actual application scenario.

S40, obtain the edge contour of the needle tip in the ROI area through an edge detection method, and determine the position of the needle tip through corner point detection, and then obtain the pixel coordinates of the needle tip;

Edge detection is to simplify image information and use edge lines to represent the information carried by the image. The edge detection of the needle position is the key to determine the needle position.

Edge detection is essentially a filtering algorithm. Different operators have different extraction effects, and the difference lies in the selection of filters. Traditional edge detection operators include Sobel operator, Robert operator, Prewitt operator, LOG operator, Canny operator, etc. The Canny edge operator is one of the most commonly used edge detection operators, and is generally considered to be the best edge detection operator. It is a multi-level edge detection algorithm, which proposes three criteria for the evaluation of edge detection: high detection rate, precise positioning, and clear response. Therefore, we use the Canny operator edge detection in OpenCV to find the edge of the needle, as shown in Figure 8(a).

For the needle part, only need to pay attention to the slender needle part, and the outline is basically a vertical line, so according to the detection of the vertical line, referring to Figure 8(b), roughly determine the range of the slender needle, and the result is shown in the figure 8(c), while removing the distraction of too many background elements. Finally, using corner detection, it is determined that the bottom corner is the needle tip, and the result is shown in Figure 8(d). The specific process is as follows:

Obtain the edge contour of the needle in the ROI area by an edge detection method;

Extract the vertical lines in the edge contour, and calculate the maximum distance between the vertical lines in the horizontal direction;

Determine whether the maximum distance is greater than the set distance threshold, and if it is greater, delete the ROI area above the horizontal direction;

Extract the vertical line in the undeleted ROI area and perform corner detection, and use the bottom corner of the longest vertical line in the undeleted ROI area as the position of the needle tip.

S50, combining the pixel coordinates of the needle tips in two different orientations, obtain the three-dimensional world coordinates corresponding to the needle tip through a preset first pose solution method; the first pose solution method is to solve the pixel points according to the pose results of the two images method of world coordinates;

The method of solving the world coordinates of the pixel points according to the pose results of the two images is as follows:

、

、

、

是世界坐标系下的几个点，用来确定世界坐标系的原点与相机外参。根据相机的内 外参数，可以估计出两次拍照的相机位姿

与

，也就知道了相机的坐标

与

。 那么将

与

，

与

连成直线，则可以获得两条直线方程，组成方程组求解得到它们 的交点，即为待求点

的坐标。 According to the principle of determining a point by two straight lines, in a two-dimensional plane, as long as the equations of the two intersecting lines are known, the coordinates of their intersection can be solved. Now suppose we take a picture in a two-dimensional plane, as shown in Figure 9(a), where

,

,

,

are several points in the world coordinate system, which are used to determine the origin of the world coordinate system and the external parameters of the camera. According to the internal and external parameters of the camera, the camera pose of the two photos can be estimated

and

, also know the coordinates of the camera

and

. then will

and

,

and

If connected into a straight line, two straight line equations can be obtained, and the equation system can be solved to obtain their intersection, which is the point to be found.

coordinate of.

与

，在根据点

在图像中的坐标， 可以确定一条从相机指向点

的射线。用两幅图像确定了关于

的两条射线，那么解方程求 出他们的交点坐标，就能得到点

的空间坐标。然而在现实中，由于误差的存在，空间中两 条相交的可能性几乎不存在，因此在计算时，应该求他们之间的最近点坐标，即为

点的三 维坐标。 In three-dimensional space, the principle is the same as two-dimensional space, except that two straight lines rise from two-dimensional space to three-dimensional space. Also, by solving PNP, the spatial position of the two shots of the camera is obtained

and

, at the base point

The coordinates in the image that can determine a line pointing from the camera to the point

ray. Determined with two images about

The two rays of , then solve the equation to find the coordinates of their intersection, you can get the point

space coordinates. However, in reality, due to the existence of errors, the possibility of two intersections in space is almost non-existent, so when calculating, the coordinates of the closest point between them should be calculated, which is

3D coordinates of the point.

在图像中的像素坐标假设为

,需要求出

在相机坐标系中的坐标

，参考公式（2）可推出公式（12），计算出点

在相机坐标系下的坐标。这是一 个从二维映射到三维的过程，参考图9（b）。 waiting point

The pixel coordinates in the image are assumed to be

, need to find

Coordinates in the camera coordinate system

, referring to formula (2), formula (12) can be derived to calculate the point

The coordinates in the camera coordinate system. This is a process of mapping from 2D to 3D, refer to Fig. 9(b).

（12）

(12)

，但这个点坐标是在相机坐标系中的，而我们需要 知道的其实是

点在世界坐标系中对应的坐标

。为了将

转为

，需要使 用到解PNP求位姿时得到的三个欧拉角

。在这三次旋转中

显然是跟着坐标系 旋转的，其在世界坐标系中的位置会随着改变。 At this point we get

, but the coordinates of this point are in the camera coordinate system, and what we need to know is actually

The coordinates of the point in the world coordinate system

. in order to

Convert to

, you need to use the three Euler angles obtained when solving the PNP to find the pose

. in these three spins

Obviously, it rotates with the coordinate system, and its position in the world coordinate system changes accordingly.

点的影响，保证相机坐标系旋转后

点依然保持在世界坐标系 原本的位置，需要对

进行三次反向旋转，旋转后得到点

在相机坐标系中新的坐标值记 为

，

的值等于世界坐标系中向量

的值。那么：

点的世界坐标

=

的值+

的世 界坐标值。 To counteract the rotating pair

The influence of the point, to ensure that the camera coordinate system is rotated

The point remains in the original position of the world coordinate system, and needs to be adjusted

Perform three reverse rotations, and get the point after rotation

The new coordinate value in the camera coordinate system is recorded as

,

is equal to the vector in the world coordinate system

value of . So:

the world coordinates of the point

=

value+

The world coordinate value of .

，即为点

，展示图9（c）。 The cameras in both directions perform the conversion between the camera coordinate system and the world coordinate system, and obtain two straight lines A and B respectively, and obtain the coordinates of the intersection of the two straight lines, which is point P. Usually due to the existence of errors, the possibility of A and B intersecting is almost non-existent, so when calculating, find the midpoint of the coordinates of the closest point between the two

, which is the point

, showing Figure 9(c).

In this embodiment, combining the pixel coordinates of the needle tips in two different orientations, the world three-dimensional coordinates corresponding to the needle tip are obtained by solving the world coordinates of the pixel points according to the pose results of the two images.

S60, calculating the error between the world three-dimensional coordinates obtained in step S50 and the standard world three-dimensional coordinates calculated by the robot before starting the liquid dispensing, and correcting the world three-dimensional coordinates obtained in step S50 according to the error, and then the robot performs the correction based on the corrected world three-dimensional coordinates. Dosing.

，即步骤S50获取的世界三维坐标与获取标准的标准世界三维坐标

的误 差，根据误差，将当前次的针头的世界三维坐标调整至标准世界三维坐标时的位置（即机器 人开始配液前计算校正的标准位置）。 In this embodiment, before the robot starts dispensing liquid, based on steps S10 to S50, the world three-dimensional coordinates of the needle tip are calculated as the standard world three-dimensional coordinates. After the robot starts to dispense liquid, it calculates the current three-dimensional coordinates of the world (

, that is, the world three-dimensional coordinates obtained in step S50 and the standard standard world three-dimensional coordinates obtained in step S50

According to the error, adjust the current world 3D coordinates of the needle to the position of the standard world 3D coordinates (that is, the standard position calculated and corrected before the robot starts dispensing).

As shown in FIG. 2 , a needle tip position detection system for automatic liquid dispensing by a robot according to the second embodiment of the present invention specifically includes: an image acquisition module 100 , a preprocessing module 200 , a ROI region extraction module 300 , and a detection module 400 , the world three-dimensional coordinate solution module 500, the liquid dispensing module 600;

The image acquisition module 100 is configured to collect the image of the needle to be detected as the input image through two cameras in different orientations under the set color type light source and the set light source illumination direction;

The preprocessing module 200 is configured to preprocess the input image to obtain a preprocessed image; the preprocessing includes denoising, binarization and histogram equalization;

The ROI region extraction module 300 is configured to use a preset corner-based ROI extraction method to extract the ROI region in the preprocessed image;

The detection module 400 is configured to obtain the edge contour of the needle tip in the ROI region through an edge detection method, and determine the position of the needle tip through corner detection, thereby obtaining the pixel coordinates of the needle tip;

The world three-dimensional coordinate solving module 500 is configured to combine the pixel coordinates of the needle tips in two different orientations, and obtain the world three-dimensional coordinates corresponding to the needle tip through a preset first pose solution method; the first pose solution method is based on two The method of solving the world coordinate of the pixel point from the pose result of the image;

The liquid dispensing module 600 is configured to calculate the error between the world three-dimensional coordinates obtained by the world three-dimensional coordinate solving module and the standard world three-dimensional coordinates calculated by the robot before starting liquid dispensing, and correct the world three-dimensional coordinates obtained by the world three-dimensional coordinate solving module according to the error. Three-dimensional coordinates, and then the robot dispenses liquid based on the corrected three-dimensional coordinates of the world.

Those skilled in the technical field can clearly understand that, for the convenience and brevity of description, for the specific working process and related description of the system described above, reference may be made to the corresponding process in the foregoing method embodiments, which will not be repeated here.

It should be noted that the needle tip position detection system for automatic liquid dispensing by the robot provided in the above embodiment is only illustrated by the division of the above functional modules. In practical applications, the above functions can be allocated by different The modules or steps in the embodiments of the present invention are further decomposed or combined. For example, the modules in the above-mentioned embodiments can be combined into one module, or can be further split into multiple sub-modules to complete all the above-described or some functions. The names of the modules and steps involved in the embodiments of the present invention are only for distinguishing each module or step, and should not be regarded as an improper limitation of the present invention.

An electronic device according to a third embodiment of the present invention, at least one processor; and a memory communicatively connected to at least one of the processors; wherein the memory stores instructions executable by the processor, the instructions The method is used for being executed by the processor to realize the method for detecting the position of the needle tip for automatic liquid dispensing by the robot as claimed above.

A computer-readable storage medium according to a fourth embodiment of the present invention stores computer instructions, and the computer instructions are used to be executed by the computer to realize the automatic liquid dispensing for robots described in the claims. The needle tip position detection method.

Those skilled in the technical field can clearly understand that the undescribed convenience and brevity, the specific working process and related description of the above-described device and computer-readable storage medium, can refer to the corresponding process in the foregoing method example, here. No longer.

Referring to FIG. 10 below, it shows a schematic structural diagram of a computer system suitable for implementing the server of the system, method, and device embodiments of the present application. The server shown in FIG. 10 is only an example, and should not impose any limitations on the functions and scope of use of the embodiments of the present application.

As shown in FIG. 10 , the computer system includes a central processing unit (CPU, Central Processing Unit) 1001, which can be loaded into a random access memory according to a program stored in a read only memory (ROM, Read Only Memory) 1002 or from a storage part 1008 (RAM, Random Access Memory) 1003 executes various appropriate operations and processes. In the RAM 1003, various programs and data necessary for system operation are also stored. The CPU 1001 , the ROM 1002 , and the RAM 1003 are connected to each other through a bus 1004 . An input/output (I/O, Input/Output) interface 1005 is also connected to the bus 1004 .

The following components are connected to the I/O interface 1005: an input section 1006 including a keyboard, a mouse, etc.; an output section 1007 including a cathode ray tube, a liquid crystal display, etc., a speaker, etc.; a storage section 1008 including a hard disk, etc.; The communication part 1009 of the network interface card of , modem etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I/O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is mounted on the drive 1010 as needed so that a computer program read therefrom is installed into the storage section 1008 as needed.

In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the method illustrated in the flowchart. In such an embodiment, the computer program may be downloaded and installed from the network via the communication portion 1009, and/or installed from the removable medium 1011. When the computer program is executed by the CPU 1001, the above-described functions defined in the method of the present application are executed. It should be noted that the computer-readable medium mentioned above in the present application may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination of the above. More specific examples of computer readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer disks, hard disks, RAM, ROM, erasable programmable read only memory (EPROM or flash memory), Optical fiber, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. In this application, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, however, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code therein. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device . Program code embodied on a computer readable medium may be transmitted using any suitable medium including, but not limited to, wireless, wire, optical fiber cable, etc., or any suitable combination of the foregoing.

Computer program code for performing the operations of the present application may be written in one or more programming languages, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional A procedural programming language, such as C or a similar programming language. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. Where a remote computer is involved, the remote computer may be connected to the user computer through any kind of network, including a local or wide area network, or may be connected to an external computer (eg, using an Internet service provider to connect through the Internet).

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code that contains one or more logical functions for implementing the specified functions executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented in dedicated hardware-based systems that perform the specified functions or operations , or can be implemented in a combination of dedicated hardware and computer instructions.

The terms "first," "second," etc. are used to distinguish between similar objects, and are not used to describe or indicate a particular order or sequence.

The term "comprising" or any other similar term is intended to encompass a non-exclusive inclusion such that a process, method, article or device/means comprising a list of elements includes not only those elements but also other elements not expressly listed, or Also included are elements inherent to these processes, methods, articles or devices/devices.

So far, the technical solutions of the present invention have been described with reference to the preferred embodiments shown in the accompanying drawings, however, those skilled in the art can easily understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

Claims (10)

1. A method for detecting the position of a needle tip for automatic liquid distribution by a robot is characterized by comprising the following steps:

s10, acquiring a needle head image to be detected by two cameras in different directions under the set color type light source and the set light source illumination direction as an input image;

s20, preprocessing the input image to obtain a preprocessed image; the preprocessing comprises denoising, binarization and histogram equalization processing;

s30, extracting an ROI (region of interest) region in the preprocessed image by adopting a preset corner-based ROI extraction method;

s40, obtaining the edge contour of the needle head in the ROI through an edge detection method, and determining the position of the needle point through corner detection so as to obtain the pixel coordinate of the needle point;

s50, combining the pixel coordinates of the needle points in two different directions, and obtaining world three-dimensional coordinates corresponding to the needle points by presetting a first position solving method; the first pose solving method is a method for solving world coordinates of pixel points according to pose results of two images;

and S60, calculating the error between the world three-dimensional coordinates acquired in the step S50 and the calculated standard world three-dimensional coordinates of the robot before the liquid dispensing is started, correcting the world three-dimensional coordinates acquired in the step S50 according to the error, and further dispensing the liquid by the robot based on the corrected world three-dimensional coordinates.

2. The method for detecting the position of the needle tip for the automatic liquid dispensing of the robot as claimed in claim 1, further comprising the steps of selecting the specification of a calibration plate, calibrating the internal and external parameters of the camera before the step S10:

selecting a checkerboard with rows even and odd or rows odd and even as a calibration board, wherein the area of the calibration board is at least half of the available pixel area when a camera acquires a clear calibration board image; the available pixel area is the pixel area of the collected whole calibration plate image;

after the specification of the calibration board is selected, camera internal reference and external reference calibration is carried out through OpenCV based on the conversion relation among a world coordinate system, a camera coordinate system and a pixel coordinate system;

the conversion relation among the world coordinate system, the camera coordinate system and the pixel coordinate system is as follows:

wherein,

、

representing the coordinates of the coordinate system of the pixel,

、

、

representing the coordinates of the world coordinate system,

、

、

representing the coordinates of the coordinate system of the camera,

、

respectively representing a rotation matrix and a translation matrix of the camera coordinate system relative to the world coordinate system,

、

is the focal length of the camera and,

、

the number of horizontal and vertical pixels representing the phase difference between the center pixel coordinate of the image and the origin pixel coordinate of the image,

、

、

is the world coordinate system coordinate of the optical center of the camera,

the transpose is represented by,

representing the internal parameters of the camera.

3. The method of claim 1, wherein the set color type light source and the set light source illumination direction are:

the light source with the set color type is a red light source;

and the set light source lighting direction is back lighting.

4. The method for detecting the position of a needle tip for robotic automatic dispensing according to claim 3, wherein the "ROI area in the pre-processed image is extracted by a preset corner-based ROI extraction method" by:

and performing corner detection on the preprocessed image through a corner detection function, and taking the minimum circumscribed rectangle corresponding to all corners as an ROI (region of interest).

5. The method according to claim 1, wherein the edge detection method is Canny operator edge detection.

6. The needle tip position detection method for robotic automatic dispensing according to claim 1, wherein "edge contour of the needle in the ROI area is obtained by an edge detection method and the position of the needle tip is determined by corner detection" by the method of:

acquiring the edge contour of the needle head in the ROI area by an edge detection method;

extracting vertical lines in the edge profile, and calculating the maximum distance between the vertical lines in the horizontal direction;

judging whether the maximum distance is greater than a set distance threshold value, and if so, deleting the ROI above the horizontal direction;

and extracting vertical lines in the undeleted ROI area, detecting corner points, and taking the corner point at the lowest part of the longest vertical line in the undeleted ROI area as the position of the needle point.

7. The method according to claim 1, wherein the world three-dimensional coordinates of the pixel points in the first pose solution method are the midpoints of the coordinates of the closest points of the two straight lines.

8. A needle tip position detection system for robotic automatic dispensing, the system comprising: the system comprises an image acquisition module, a preprocessing module, an ROI (region of interest) region extraction module, a detection module, a world three-dimensional coordinate solving module and a liquid preparation module;

the image acquisition module is configured to acquire a needle head image to be detected under the conditions of setting color type light sources and setting the lighting direction of the light sources through two cameras in different directions to serve as an input image;

the preprocessing module is configured to preprocess the input image to obtain a preprocessed image; the preprocessing comprises denoising, binarization and histogram equalization processing;

the ROI region extraction module is configured to extract an ROI region in the preprocessed image by adopting a preset corner-based ROI extraction method;

the detection module is configured to acquire an edge contour of the needle head in the ROI through an edge detection method, determine the position of the needle point through angular point detection and further obtain a pixel coordinate of the needle point;

the world three-dimensional coordinate solving module is configured to combine the pixel coordinates of the needle points in two different directions and obtain a world three-dimensional coordinate corresponding to the needle point by presetting a first position solving method; the first pose solving method is a method for solving world coordinates of pixel points according to pose results of two images;

the liquid preparation module is configured to calculate an error between the world three-dimensional coordinate acquired by the world three-dimensional coordinate solving module and a standard world three-dimensional coordinate calculated by the robot before liquid preparation is started, correct the world three-dimensional coordinate acquired by the world three-dimensional coordinate solving module according to the error, and then prepare the liquid by the robot based on the corrected world three-dimensional coordinate.

9. An electronic device, comprising:

at least one processor; and

a memory communicatively coupled to at least one of the processors; wherein,

the memory stores instructions executable by the processor for performing the method of any one of claims 1-7 for robotic automatic fluid dispensing.

10. A computer-readable storage medium storing computer instructions for execution by the computer to perform the method for robotic automatic fluid dispensing tip position detection of any of claims 1-7.

Images