CN118052883A - Binocular vision-based multiple circular target space positioning method - Google Patents

Abstract

The present invention provides a method for spatial positioning of multiple circular targets based on binocular vision, which relates to the technical field of visual sensor positioning. A binocular camera is stereoscopically calibrated to obtain the internal and external parameters of the camera and then stereo correction is performed; circular markers are identified and extracted by using image processing related technologies. On this basis, the Canny operator is used for coarse positioning, and then the sub-pixel extraction of the coarse positioning edge is performed by combining the Zernike moment method with the Otsu method, the grayscale gradient optimization is used to refine the edge, and then the least squares ellipse fitting method is used to locate the center coordinate of the refined edge. Secondly, the method based on spatiotemporal characteristic tracking is combined to track the image sequence at continuous moments, and finally stereo matching based on the same numbered target pair is realized, and the spatial coordinates are located based on the triangulation principle of binocular vision. Finally, it is proved through experiments that the present invention can achieve fast, high-precision and stable spatial positioning of multiple circular targets.

Description

A spatial positioning method for multiple circular targets based on binocular vision

Technical Field

The present invention relates to the field of vision sensor positioning technology, and in particular to a binocular vision-based spatial positioning method for multiple circular targets.

Background technique

Circular target spatial positioning is target recognition and spatial positioning technology. As the basis of computer vision, this technology has a wide range of applications in the field of robotics, image retrieval, and UAV flight environment perception. Circular target recognition is to detect the target object of interest in a still image or dynamic video. In the field of visual sensor positioning technology, especially in machine vision and target positioning, people are paying more and more attention to the development of more accurate and efficient positioning methods. Vision-based measurement technology is a method of close-range photogrammetry using digital cameras and computer technology. It is mainly used in areas such as obtaining detailed information on the surface of an object, performing three-dimensional modeling, and measuring shapes and sizes. According to the principle of visual measurement, the principle is based on the target image pairs obtained by photography. At present, the methods for obtaining image pairs mainly include monocular vision and multi-ocular vision. Monocular vision needs to be solved by camera movement and external fixed markers, while in multi-ocular vision, binocular vision is widely used because of its simple operation and high precision. Binocular vision obtains the relative solution parameters of the target by directly solving the external parameter matrix between the two cameras. In deformation monitoring, since it is usually difficult to find fixed external markers, binocular vision measurement technology has been widely used.

The basic principle of binocular vision measurement technology is to simulate the way human eyes obtain stereoscopic visual information, use two cameras to shoot the same scene separately, and analyze the image differences between the cameras to obtain the three-dimensional spatial information of the target. At present, many scholars have studied the project vision measurement technology and applied it to many fields. Binocular vision systems have been widely used in the field of machine vision because they can simulate the stereoscopic perception of the human eye. However, there are still some challenges in the current binocular vision method, especially in dealing with the positioning problem of multiple circular targets. Existing technologies may be limited by the shape, size and stereo matching of the target in multi-target positioning, resulting in positioning accuracy that is difficult to meet the high requirements of practical applications.

Summary of the invention

In view of the shortcomings of the prior art, the present invention provides a method for spatial positioning of multiple circular targets based on binocular vision, aiming to overcome the limitations of the prior art, improve positioning accuracy and robustness, and better adapt to application requirements in various complex environments. The present invention combines binocular vision measurement technology with image processing related technologies to complete the recognition and spatial positioning of circular targets of specific colors, which can improve the accuracy of positioning monitoring based on markers.

The technical solution adopted by the present invention is:

A method for spatially locating multiple circular targets based on binocular vision comprises the following steps:

Step 1: Use the left and right cameras of the binocular camera to take pictures of the calibration plate at different angles, and complete the stereo calibration and correction of the binocular camera according to Zhang Zhengyou's calibration method and the Matlab calibration toolbox;

Step 2: Start the calibrated binocular camera to collect circular target images, filter and denoise the collected circular target images, and then perform binary segmentation on the denoised images to obtain segmented target images. Then, use morphological filtering, and finally use the circularity criterion to identify the circular target area.

Step 2-1: Filter and denoise the collected circular target image;

The original target image is filtered and denoised using guided filtering, and the target image is guided based on the guidance signal to adjust the weight in the filtering process;

Step 2-2: Binarization segmentation;

The HSI color space is used to identify the markers. The denoised target image is converted from the RGB space to the HSI space. The red markers are extracted according to the range thresholds of each HSI component, as shown in formula (1):

In the formula, H is the frequency that defines the color, that is, the hue; S represents the depth of the color, that is, the saturation; I represents the brightness; when each HSI component of the image pixel value satisfies the red threshold condition corresponding to formula (1) at the same time, the pixel point at the corresponding position is set to the foreground color 1, otherwise it is set to the black background 0, and then the morphological filtering method combining the opening operation and the closing operation is used to further remove the noise;

Step 2-3: using a circularity criterion to identify a circular target area; specifically, identifying the circular target area by calculating the circularity of the target area;

The circularity indicates the similarity between the target circle and the ideal circle, which is determined by calculating the perimeter and area of the target area. The calculation formula of the ratio of the perimeter and area of the target area is shown in formula (2):

In the formula, C is the circularity of the target area, _Sc represents the area of the target area, _Lc is the contour perimeter of the target area, and β is the proportional coefficient. The expression is:

S _c =πr ² , L _c =2πr (3)

Where r is the radius of the target area. According to the circularity calculation formula, when C=1, the connected area is considered a circle, and the circularity C of the target area satisfies the following conditions:

C _min ≤C ≤C _max (4)

In the formula, C _min and C _max represent the circularity of the landmark point when the angle between the line connecting the circular target and the binocular camera and the optical axis of the binocular camera is the maximum and minimum respectively, and its value is expressed as:

Where θ is the deflection angle of the binocular camera;

Step 3: Use the Canny edge detection algorithm to roughly determine the edge of the identified circular target area, and then use the Zernike moment to perform sub-pixel edge detection on the roughly determined edge. At the same time, the edge threshold judgment criterion based on the Otsu method is introduced to determine the sub-pixel edge, and then the sub-pixel edge is refined by using the grayscale gradient optimization method. Finally, the sub-pixel edge is centrally located by applying the least squares ellipse fitting method to obtain the central pixel coordinates of the circular target;

Step 3-1: Use the Canny edge detection algorithm to roughly determine the edge of the identified circular target area;

First, the binarized circular target image is smoothed by using a Gaussian filter. Secondly, the gradient operator is used to calculate the gradient of the image on the smoothed image to obtain the gradient strength and direction of each pixel in the image and obtain the gradient map. The gradient map is non-maximum suppressed by comparing the gradient strength of a pixel with the gradient strength of its two adjacent pixels in the gradient direction, and only the pixel with the local maximum gradient value is retained, thereby suppressing the response of the non-edge area. The gradient map is then divided into two thresholds, a high threshold and a low threshold. The pixels are divided into three categories: strong edge, weak edge, and non-edge according to the gradient strength of the pixel. A strong edge is a true edge, a weak edge is part of a true edge, and a non-edge is the remaining area. For a weak edge, a continuous coarse edge is formed by connecting the strong edge pixels around it.

Step 3-2: Use Zernike moments to perform sub-pixel edge detection on the roughly determined edge;

Let A _nm represent the Zernike moment of the image f(x, y). Through the conversion formula, A _nm is expressed as:

Where (n+1)/π is the normalization factor; is the Zernike moment polynomial, ρ is the vector distance from the origin to the pixel point (x, y); θ is the angle between ρ and the x-axis. If the image is rotated by angle θ, the Zernike moment A′ _nm of the rotated image f(x, y) satisfies the following formula:

According to the above formula, the Zernike moments A ₀₀ , A ₁₁ and A ₂₀ of the roughly defined edge target image are calculated, and the Zernike moments A′ ₀₀ , A′ ₁₁ , A′ ₂₀ of the rotated image satisfy the following relationship:

The four parameters h, k, l and The calculation formula is as follows:

In formula (9), h is the background gray value, k is the gray step value, l is the vertical distance from the center to the edge of the template, is the angle between the edge and the x-axis. Using the N×N Zernike moment template and the parameters calculated by the above formula, the ith sub-pixel edge point P _i (x', y') is obtained, and the calculation formula is:

Step 3-3: Introduce the edge threshold judgment criterion based on Otsu's method to determine the sub-pixel edge;

The threshold judgment condition of the sub-pixel edge point based on the Zernike moment algorithm is as follows:

l≤l _t ∩k≥k _t (11)

In formula (11), _kt is the grayscale step threshold, _lt is the distance threshold, and its value is:

The judgment criteria of the Otsu method are introduced. Based on the inter-class variance of the edge area and the background area, the basis for selecting the optimal threshold is to maximize the inter-class variance. Specifically, the Otsu method traverses all potential thresholds, calculates the inter-class variance under each threshold, and finally selects the threshold that can maximize the inter-class variance as the best choice. In a sign image containing n pixels, let the number of pixels with gradient amplitude i be n _i , therefore, the probability of the occurrence of gradient amplitude i is p _i = n _i /n. By calculating the threshold t, the image is divided into two categories as the foreground edge area and the background, and the corresponding probabilities are:

Their means are:

The average gradient amplitude is Then the between-class variance of the two classes is

When the variance σ ² is the largest, the difference between the foreground edge and the background reaches the maximum, so the segmentation accuracy is the highest, and the grayscale step threshold k _t at this time is the optimal value; by using the optimal grayscale step threshold k _t and the distance threshold l _t to calculate the sub-pixel edge based on the Zernike moment, the sub-pixel edge point p _i ′(x', y') is finally obtained and stored in the edge point set {P _i }.

Step 3-4: Sub-pixel edge refinement is performed using a grayscale gradient optimization-based method;

The center value of the distribution in the grayscale gradient optimization method is the position where the gradient direction changes the most, that is, the position of the edge. First, the sub-pixel edge information is extracted by the improved sub-pixel Zernike moment algorithm in step 3-3, and then the least squares ellipse fitting method is used to perform center fitting on all sub-pixel edge points to obtain the rough center of the circular marker. Based on the rough center and the corresponding rough edge, the radial line equation corresponding to each edge pixel is obtained:

y＝kx+b (16)

in:

In the formula, (x ₀ , y ₀ ) represents the roughly determined center coordinates; ( _xi , _yi ) represents the coordinates of each edge point. For a certain edge point, the first-order derivative of the grayscale along the gradient direction is calculated along its radial direction, and the central grayscale value is obtained using the Gaussian function. The expression of the Gaussian function is:

In formula (18), a is the Gaussian proportional coefficient, b is the center value of the first-order derivative Gaussian distribution of the gradient grayscale, c ² is the variance of the first-order derivative Gaussian distribution of the gradient grayscale, and p _i ′(x _b , y _b ) is the location of the edge point after refinement. All edge points are refined in turn;

Step 3-5: Use the least squares ellipse fitting method to locate the center of the sub-pixel edge and obtain the center pixel coordinates of the circular target;

The sub-pixel edge point set {P _i } is processed using the least squares ellipse fitting method to locate the center of the circular target. The general equation of the ellipse is as follows:

F(x)＝ax ² +bxy+cy ² +dx+ey+f＝0 (19)

The edge points are fitted by the least squares ellipse fitting method to obtain the parameters A, B, C, D, E, and F. The coordinate calculation formula of the ellipse center point is:

The final coordinates of the center point of the circular target are (x _r , y _r ).

Step 4: Initially number the multiple circular targets photographed by the binocular camera, and then use the target tracking method based on spatiotemporal features to continuously identify the numbered circular targets, locate the pixel coordinates of the center of the circular targets, and then stereo match the two circular target images taken by the binocular camera at the same time according to the same numbered pixel coordinates, and perform spatial positioning according to the binocular vision measurement principle.

Step 4-1: Initially locate all circular targets, obtain their central pixel coordinate values, sort their pixel center coordinate values in ascending order of pixel x values, and store them in set C; divide set C into m subsets P ₁ , P ₂ , ... P _k , P _m , each subset contains n elements; further, sort each element in each subset P _k in ascending order of y values, k = 1, 2, ... k, n, and obtain Q _ki , k = 1, 2, ...., m, i = 1, 2, ..., n, Q _ki is the pixel coordinate of the circular target in the mth row and nth column, the elements in set C represent the pixel points of the target image sorted from top to bottom and from left to right, and the number of each central pixel point is the number of the corresponding marker;

Step 4-2: Use the marker recognition method to process the image sequence and obtain the multi-marker extraction result image to distinguish the marker foreground and background space; calculate the spatial features based on the minimum enclosing rectangle and the temporal features based on the image time sequence of the target for each circular target, and use the circular marker of the monitoring point identified at the current moment and its minimum enclosing rectangle as the spatial feature operator of the target, and record the current moment as the time description operator. Finally, integrate the spatial feature operator and the time description operator to jointly construct the target feature vector A, as shown in the following formula:

A＝{O _cen ，M _matrix ，T _fps } (21)

Wherein, the structural attribute of A is a cell array, O _cen is the coordinate number of the center of the circular target pixel, M _matrix is the minimum circumscribed matrix of the circular mark, and T _fps is the time sequence of the binary circular mark at the current moment.

Create a time series data set B, which consists of n feature vectors A, and its expression is as follows:

B＝{A ₁ , A ₂ , … , _An } (22)

A _n represents the feature vector of the circular target with sequence number n in the image at the current moment, and n represents the number of foreground targets.

Step 4-3: By tracking the position of the circular target in the image, the same target in consecutive image frames is associated. The target feature vector is used in combination with the target circumscribed rectangle overlap OLD to achieve matching association of the circular target at different times. The calculation model of OLD is as follows:

OLD＝K/( _PT +K+QT ₊₁ ) (23)

In the formula, OLD is used to evaluate the similarity of two target regions. The higher its value, the greater the possibility that the two targets are the same object; K represents the similar area of the two target regions, which refers to the area of the two targets to be detected in consecutive moments. The matching conditions for determining whether two targets are successfully associated are as follows:

OLD≥α (24)

Among them, α represents the discrimination threshold. When the target to be detected meets the above similarity judgment conditions, the two targets are considered to be the same symbol. If not, they are considered to be different targets. If two targets at different time points are judged to be the same target, the target feature information in the latest image frame will replace the target feature associated with the previous moment in the time series cell group B;

Step 4-4: Track the circular target with the initial number at continuous moments, obtain the circular target with the corresponding number in the left and right camera images in the sequence at the same time, realize stereo matching through the corresponding number, and finally solve its spatial coordinates by using the binocular vision measurement principle;

The binocular camera uses two images taken at different angles to calculate the disparity map, and then obtains the three-dimensional information of the pixel through the disparity map. b is the baseline distance between the two cameras, f is the focal length of the camera lens, L is the imaging width, _XL and _XR respectively represent the pixel distances of the two imaging points _PL and _PR in the X-axis direction in their respective coordinate systems, and the corresponding difference in pixel coordinates is called disparity d, and Z is the depth distance from the object point in space to the baseline of the two cameras. The disparity d is expressed as:

d＝| _XL - _XR | (25)

Then the distance between the two imaging points _PL and _PR is:

According to the theory of similar triangles:

Then the distance Z from point P to the projection center plane is obtained, which is expressed as follows:

When a point P in three-dimensional space moves, its imaging points _PL and _PR on the left and right cameras will also change accordingly, and the parallax of the left and right cameras will change accordingly; the depth information of the point can be obtained based on the parallax principle, and the three-dimensional coordinates of the point in space can be obtained based on the principle of triangle similarity:

The beneficial effects of adopting the above technical solution are:

The present invention provides a method for spatial positioning of multiple circular targets based on binocular vision. The present invention combines binocular vision measurement technology with image processing related technology to complete the recognition and spatial positioning of circular targets of specific colors. Firstly, a center positioning method based on improved sub-pixel edges of circular markers is proposed, and circular markers are identified and extracted by using image processing related technology. On this basis, the Canny operator is used for coarse positioning, and then the sub-pixel extraction of the coarse positioning edge is performed by combining the Zernike moment method with the Otsu method, and the grayscale gradient optimization is used to refine the edge, and then the least squares ellipse fitting method is used to locate the center coordinate of the refined edge. Secondly, a stereo matching method based on numbering is proposed, and the target tracking of the image sequence at continuous moments is performed in combination with a method based on spatiotemporal characteristic tracking, and finally stereo matching based on the same numbered target pair is realized, so as to locate the spatial coordinates. Finally, it is proved through experiments that the method proposed in this paper can achieve fast, high-precision and stable spatial positioning.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a flow chart of a method for spatially locating multiple circular targets based on binocular vision provided by a specific embodiment of the present invention;

FIG2 is a physical picture of a HBVCAM-4M2142HD-2 model binocular camera provided in a specific embodiment of the present invention;

FIG3 is a diagram of a checkerboard grid calibration plate provided in a specific embodiment of the present invention;

FIG4 is a real-life picture of a scene captured by a binocular camera provided in a specific embodiment of the present invention;

FIG5 is a set of binocular camera calibration images provided in a specific embodiment of the present invention;

Among them, 5(a) is the calibration image taken by the left camera, and 5(b) is the calibration image taken by the right camera;

FIG6 is a schematic diagram of the measurement principle of a binocular vision system provided in a specific embodiment of the present invention;

FIG7 is a schematic diagram of camera imaging of a binocular vision system provided in a specific embodiment of the present invention;

FIG8 provides a plurality of circular logo recognition images according to a specific embodiment of the present invention;

8(a) is a picture of the original objects of multiple circular signs taken by the left camera, 8(b) is a picture of the original objects of multiple circular signs taken by the right camera, 8(c) is a picture of the recognized images of multiple circular signs taken by the left camera, and 8(d) is a picture of the recognized images of multiple circular signs taken by the right camera;

Detailed ways

The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

A method for spatial positioning of multiple circular targets based on binocular vision, as shown in FIG1 , comprises the following steps:

Step 1: Use the left and right cameras of the binocular camera to take pictures of the calibration plate at different angles, and complete the stereo calibration and correction of the binocular camera according to Zhang Zhengyou's calibration method and the Matlab calibration toolbox;

The ultimate goal of the binocular camera is to obtain the pixel coordinates of the center of the circular target, and then obtain the spatial coordinates of the center of the circular target through the triangular similarity principle after obtaining the disparity map through stereo matching. However, this is obtained when the binocular camera is in an ideal condition. Usually, the binocular camera will have distortion. Therefore, the binocular camera needs to be stereo calibrated and corrected before use so that it can work in an ideal condition.

According to Zhang Zhengyou's calibration method, MATLAB is used to realize offline calibration of the binocular camera, HBVCAM-4M2142HD-2-V22, whose photosensitive element is CMOS and fixed focal length is 3mm, as shown in Figure 2; the calibration board uses a self-made 7×7 chessboard grid, and the distance between the corner points is 40mm, as shown in Figure 3. Use Python+opencv to write the image acquisition program;

The binocular camera calibration is carried out in an open outdoor flat area with good natural lighting conditions. The calibration plate is placed at the center of the left and right camera images, and the binocular acquisition device is used to collect calibration plates at different positions. Since the relative direction between the binocular camera itself and the calibration plate is not perpendicular, the movement angle of the calibration plate should not be too large during the calibration image acquisition process, as shown in Figure 4. Therefore, images are randomly collected within the following position parameter range: the object-machine distance is 800mm-1000mm, the acquisition window resolution is 640×480pixel, the angle between the target plane and the image plane is 0-45 degrees, and 25 groups of calibration images are collected. As shown in Figure 5, this is a group of calibration images collected using the binocular acquisition device.

Open the "Stereo Camera Calibrator" toolbox in the MATLAB simulation software platform. After setting "Coefficients", "Skew" and "Tangential Distortion", import the 25 sets of left and right view images of the calibration plate collected earlier. After the import is complete, click the Calibrator button on the toolbox. The toolbox will automatically find and import each corner point in the image. During calibration, some images with large deviations will be removed to improve the calibration effect. After the calibration is completed, the internal and external parameters of the stereo camera will be obtained. After calibration through the toolbox in the MATLAB software, the left and right camera internal parameters of the stereo camera, as well as the rotation matrix and translation matrix of the camera will be obtained. The parameters of the stereo camera are shown in Table 1 below.

Table 1. Stereo camera calibration results

After obtaining the intrinsic rotation and translation matrix of the binocular camera through calibration, the left and right images of the calibration plate collected are corrected using the correction principle. After the correction, each pixel corresponding to the left and right images of the calibration plate collected by the binocular camera is on the same horizontal line. The stereo calibration of the binocular camera can provide more accurate data for subsequent stereo matching and binocular ranging.

Step 2: Start the calibrated binocular camera to collect circular target images, filter and denoise the collected circular target images, and then perform binary segmentation on the denoised images to obtain segmented target images. Then, use morphological filtering, and finally use the circularity criterion to identify the circular target area.

Step 2-1: Filter and denoise the collected circular target image;

Guided filtering is used to filter and denoise the original target image. The target image is guided based on the guidance signal to adjust the weights in the filtering process to better preserve the edges and details of the image.

Step 2-2: Binarization segmentation;

The HSI color space is used to identify the markers. The denoised target image is converted from the RGB space to the HSI space. The red markers are extracted according to the range thresholds of each HSI component, as shown in formula (1):

In the formula, H is the frequency that defines the color, that is, the hue; S represents the depth of the color, that is, the saturation; I represents the brightness; when each HSI component of the image pixel value satisfies the red threshold condition corresponding to formula (1) at the same time, the pixel point at the corresponding position is set to the foreground color 1, otherwise it is set to the black background 0, and then the morphological filtering method combining the opening operation and the closing operation is used to further remove the noise;

Step 2-3: Use the circularity criterion to identify the circular target area; specifically, the circular target area is identified by calculating the circularity of the target area, thereby extracting the mark area and excluding possible interfering targets.

The circularity indicates the similarity between the target circle and the ideal circle, which is determined by calculating the perimeter and area of the target area. The calculation formula of the ratio of the perimeter and area of the target area is shown in formula (2):

In the formula, C is the circularity of the target area, _Sc represents the area of the target area, _Lc is the contour perimeter of the target area, and β is the proportional coefficient. The expression is:

S _c =πr ² , L _c =2πr (3)

Where r is the radius of the target area. According to the circularity calculation formula, when C=1, the connected area is considered a circle. Considering the differences in imaging distance and angle, the target in the binary image may be stretched or distorted. Therefore, the circularity C of the target area satisfies the following conditions:

C _min ≤C ≤C _max (4)

In the formula, C _min and C _max represent the circularity of the landmark point when the angle between the line connecting the circular target and the binocular camera and the optical axis of the binocular camera is the maximum and minimum respectively, and its value is expressed as:

Where θ is the deflection angle of the binocular camera, and in an ideal state, β = 1. Through the above operations, the circular target is identified from the background.

Step 3: Use the Canny edge detection algorithm to roughly determine the edge of the identified circular target area, and then use the Zernike moment to perform sub-pixel edge detection on the roughly determined edge. At the same time, the edge threshold judgment criterion based on the Otsu method is introduced to determine the sub-pixel edge, and then the sub-pixel edge is refined by using the grayscale gradient optimization method. Finally, the sub-pixel edge is centrally located by applying the least squares ellipse fitting method to obtain the central pixel coordinates of the circular target;

Step 3-1: Use the Canny edge detection algorithm to roughly determine the edge of the identified circular target area;

First, the binarized circular target image is smoothed by using a Gaussian filter to reduce the noise effect in the image while retaining more details in the image; secondly, the gradient operator such as the Sobel operator is used to calculate the gradient of the image on the smoothed image, and the gradient intensity and direction of each pixel in the image are obtained to obtain a gradient map; the gradient intensity represents the grayscale change rate of the image at that point, and the gradient direction represents the direction of the fastest change; the gradient map is non-maximum suppressed, by comparing the gradient intensity of a pixel with the gradient intensity of its two adjacent pixels in the gradient direction, only the pixel with the local maximum gradient value is retained, thereby suppressing the response of the non-edge area, and then the gradient map is divided into two thresholds, a high threshold and a low threshold; according to the gradient intensity of the pixel, the pixel is divided into three categories: strong edge, weak edge and non-edge. Strong edge is the real edge, weak edge is part of the real edge, and non-edge is the remaining area; for weak edge, by connecting the strong edge pixels around it, a continuous rough edge is formed; the edge tracking algorithm based on gradient direction is usually used, such as the connection based on 4 neighborhoods or 8 neighborhoods.

Step 3-2: Use Zernike moments to perform sub-pixel edge detection on the roughly determined edge;

Let A _nm represent the Zernike moment of the image f(x, y). Through the conversion formula, A _nm is expressed as:

Where (n+1)/π is the normalization factor; is the Zernike moment polynomial, ρ is the vector distance from the origin to the pixel point (x, y); θ is the angle between ρ and the x-axis. If the image is rotated by θ, the Zernike moment A′ _nm of the rotated image f(x, y) satisfies the following formula:

According to the above formula, the Zernike moments A ₀₀ , A ₁₁ and A ₂₀ of the roughly defined edge target image are calculated, and the Zernike moments A′ ₀₀ , A′ ₁₁ , A′ ₂₀ of the rotated image satisfy the following relationship:

The four parameters h, k, l and The calculation formula is as follows:

In formula (9), h is the background gray value, k is the gray step value, l is the vertical distance from the center to the edge of the template, is the angle between the edge and the x-axis. Using an N×N Zernike moment template, where N is 7, and the parameters calculated using the above formula, the ith sub-pixel edge point P _i (x', y') is obtained. The calculation formula is:

Step 3-3: Introduce the edge threshold judgment criterion based on Otsu's method to determine the sub-pixel edge;

The threshold judgment condition of the sub-pixel edge point based on the Zernike moment algorithm is as follows:

l≤l _t ∩k≥k _t (11)

In formula (11), _kt is the grayscale step threshold, _lt is the distance threshold, and its value is:

In formula (12), N represents the size of the template, which is usually taken as 7. As the grayscale step threshold _kt , it is crucial to the extraction of edge information. The judgment criterion of the Otsu method is introduced. Based on the inter-class variance of the edge area and the background area, the basis for selecting the optimal threshold is to maximize the inter-class variance; specifically, the Otsu method traverses all potential thresholds, calculates the inter-class variance under each threshold, and finally selects the threshold that can maximize the inter-class variance as the best choice; in the sign image containing n pixels, let the number of pixels with gradient amplitude i be n _i , therefore, the probability of the occurrence of gradient amplitude i is p _i = n _i /n, by calculating the threshold t, the image is divided into two categories as the foreground edge area and the background, and the corresponding probabilities are:

Their means are:

The average gradient amplitude is Then the between-class variance of the two classes is

σ ² ＝ω ₀ (-μ _t ) ² +ω ₁ (μ ₀ -μ _t ) ² (15)

When the variance σ ² is the largest, the difference between the foreground edge and the background reaches the maximum, so the segmentation accuracy is the highest, and the grayscale step threshold k _t at this time is the optimal value; by using the optimal grayscale step threshold k _t and the distance threshold l _t to calculate the sub-pixel edge based on the Zernike moment, the sub-pixel edge point p _i ′(x', y') is finally obtained and stored in the edge point set {P _i }.

Step 3-4: Sub-pixel edge refinement is performed using a grayscale gradient optimization-based method;

The center value of the distribution in the grayscale gradient optimization method is the position where the gradient direction changes the most, that is, the position of the edge. First, the sub-pixel edge information is extracted by the improved sub-pixel Zernike moment algorithm in step 3-3, and then the least squares ellipse fitting method is used to perform center fitting on all sub-pixel edge points to obtain the rough center of the circular marker. Based on the rough center and the corresponding rough edge, the radial line equation corresponding to each edge pixel is obtained:

y＝kx+b (16)

in:

In the formula, (x ₀ , y ₀ ) represents the roughly determined center coordinates; ( _xi , _yi ) represents the coordinates of each edge point. For a certain edge point, the first-order derivative of the grayscale along the gradient direction is calculated along its radial direction, and the central grayscale value is obtained using the Gaussian function. The expression of the Gaussian function is:

In formula (18), a is the Gaussian proportional coefficient, b is the center value of the first-order derivative Gaussian distribution of the gradient grayscale, c ² is the variance of the first-order derivative Gaussian distribution of the gradient grayscale, and p _i ′(x _b , y _b ) is the location of the edge point after refinement. All edge points are refined in turn;

Step 3-5: Use the least squares ellipse fitting method to locate the center of the sub-pixel edge and obtain the center pixel coordinates of the circular target;

The sub-pixel edge point set {P _i } is processed using the least squares ellipse fitting method to locate the center of the circular target. The general equation of the ellipse is as follows:

F(x)＝ax ² +bxy+cy ² +dx+ey+f＝0 (19)

The edge points are fitted by the least squares ellipse fitting method to obtain the parameters A, B, C, D, E, and F. The coordinate calculation formula of the ellipse center point is:

The final coordinates of the center point of the circular target are (x _r , y _r ).

Step 4: Initially number the multiple circular targets photographed by the binocular camera, and then use the target tracking method based on spatiotemporal features to continuously identify the numbered circular targets, locate the pixel coordinates of the center of the circular targets, and then stereo match the two circular target images taken by the binocular camera at the same time according to the same numbered pixel coordinates, and perform spatial positioning according to the binocular vision measurement principle.

Step 4-1: Initially locate all circular targets, obtain their central pixel coordinate values, sort their pixel center coordinate values in ascending order of pixel x values, and store them in set C; divide set C into m subsets P ₁ , P ₂ , ... P _k , P _m , each subset contains n elements; further, sort each element in each subset P _k in ascending order of y values, k = 1, 2, ... k, n, and obtain Q _ki , k = 1, 2, ...., m, i = 1, 2, ..., n, Q _ki is the pixel coordinate of the circular target in the mth row and nth column, the elements in set C represent the pixel points of the target image sorted from top to bottom and from left to right, and the number of each central pixel point is the number of the corresponding marker;

Step 4-2: Use the marker recognition method to process the image sequence and obtain the multi-marker extraction result image to distinguish the marker foreground and background space; calculate the spatial features based on the minimum enclosing rectangle and the temporal features based on the image time sequence of the target for each circular target, and use the circular marker of the monitoring point identified at the current moment and its minimum enclosing rectangle as the spatial feature operator of the target, and record the current moment as the time description operator. Finally, integrate the spatial feature operator and the time description operator to jointly construct the target feature vector A, as shown in the following formula:

A＝{O _cen ，M _matrix ，T _fps } (21)

Wherein, the structural attribute of A is a cell array, O _cen is the coordinate number of the center of the circular target pixel, M _matrix is the minimum circumscribed matrix of the circular mark, which is used to provide the judgment information required for the next step of target association; at the same time, T _fps is the time sequence of the binary circular mark at the current moment.

A time series data group B is established. This group summarizes the state changes of all marker targets in the entire tracking process as a unit of time. It consists of n feature vectors A, and its expression is as follows:

B＝{A ₁ , A ₂ , … , _An } (22)

A _n represents the feature vector of the circular target with sequence number n in the image at the current moment, and n represents the number of foreground targets.

Step 4-3: By tracking the position of the circular target in the image, the same target in consecutive image frames is associated. The target feature vector is used in combination with the target circumscribed rectangle overlap OLD to achieve matching association of the circular target at different times. The calculation model of OLD is as follows:

OLD＝K/( _PT +K+QT ₊₁ ) (23)

In the formula, OLD is used to evaluate the similarity of two target regions. The higher its value, the greater the possibility that the two targets are the same object; K represents the similar area of the two target regions, which refers to the area of the two targets to be detected in consecutive moments. The matching conditions for determining whether two targets are successfully associated are as follows:

OLD≥α (24)

Among them, α represents the discrimination threshold, which is used to verify the reliability of target tracking. When the target to be detected meets the above similarity judgment conditions, the two targets are considered to be the same mark. If not, they are considered to be different targets. If two targets at different time points are judged to be the same target, the target feature information in the latest image frame will replace the target feature associated with the previous moment in the time series cell group B;

Step 4-4: Track the circular target with the initial number at continuous moments, obtain the circular target with the corresponding number in the left and right camera images in the moment sequence, realize stereo matching through the corresponding number, and finally solve its spatial coordinates by using the binocular vision measurement principle;

The binocular camera uses two images taken at different angles to calculate the disparity map, and then obtains the three-dimensional information of the pixel through the disparity map. The principle of binocular ranging is shown in Figure 6. b is the baseline distance between the two cameras, f is the focal length of the camera lens, L is the imaging width, _XL and _XR respectively represent the pixel distances of the two imaging points _PL and _PR in the X-axis direction in their respective coordinate systems, and the corresponding difference in pixel coordinates is called disparity d, and Z is the depth distance from the object point in space to the baseline of the two cameras. The disparity d is expressed as:

d＝| _XL - _XR | (25)

Then the distance between the two imaging points _PL and _PR is:

According to the theory of similar triangles:

Then the distance Z from point P to the projection center plane is obtained, which is expressed as follows:

When point P in the three-dimensional space moves, its imaging points _PL and _PR on the left and right cameras will also change accordingly, and the parallax of the left and right cameras will change accordingly; the depth information of the point is obtained according to the parallax principle, and the imaging of point P on the camera is shown in Figure 7. As can be seen from Figure 8, according to the principle of triangle similarity, the three-dimensional coordinates of the point in space can be obtained as:

In order to verify the effectiveness of the binocular vision-based spatial positioning method for multiple circular targets proposed in the present invention, a spatial positioning experiment for circular targets on a slope is conducted, as shown in FIG8 . In order to ensure the stability of the coordinates of the initial point, the initial position of the circular target is measured ten times in this embodiment, and the average of the ten measurements is used as the initial coordinate of the marker point. The pixel coordinates and three-dimensional coordinates (with the left camera coordinate system as the world coordinate system) of the multiple circular targets are obtained as shown in Table 1.

Table 1 Pixel coordinates and three-dimensional coordinates of monitoring point markers

Since it is impossible to directly verify the error of the three-dimensional coordinates, the present invention adopts an alternative method: the accuracy of the binocular vision measurement system is evaluated by measuring and analyzing the relative distance between two markers. In order to obtain accurate distance data, this embodiment uses a laser rangefinder for distance measurement, and the measurement accuracy of the instrument reaches 0.1mm. As shown in Table 2, the measurement and calculation data of the relative distance between different monitoring point markers are shown.

Table 2 Relative distance measurement of monitoring point marks Unit: mm

As shown in Table 2, the absolute error range of the relative distance measured by the circular target spatial positioning method based on binocular vision proposed in the present invention is between 1.8 mm and 3.4 mm. The average absolute error of the relative distance measurement is 2.6 mm. It can be seen that the circular target spatial positioning accuracy based on binocular vision proposed in the present invention can reach the millimeter level, which is sufficient to meet the high requirements of slope displacement monitoring and provides a reliable basis for high-precision monitoring of slope displacement.

The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) to form a technical solution.

Claims (4)

1. A method for spatial positioning of multiple circular targets based on binocular vision, characterized in that it comprises the following steps: Step 1: Use the left and right cameras of the binocular camera to take pictures of the calibration plate at different angles, and complete the stereo calibration and correction of the binocular camera according to Zhang Zhengyou's calibration method and the Matlab calibration toolbox; Step 2: Start the calibrated binocular camera to collect circular target images, filter and denoise the collected circular target images, and then perform binary segmentation on the denoised images to obtain segmented target images. Then, use morphological filtering, and finally use the circularity criterion to identify the circular target area. Step 3: Use the Canny edge detection algorithm to roughly determine the edge of the identified circular target area, and then use the Zernike moment to perform sub-pixel edge detection on the roughly determined edge. At the same time, the edge threshold judgment criterion based on the Otsu method is introduced to determine the sub-pixel edge, and then the sub-pixel edge is refined by using the grayscale gradient optimization method. Finally, the sub-pixel edge is centrally located by applying the least squares ellipse fitting method to obtain the central pixel coordinates of the circular target; Step 4: Initially number the multiple circular targets photographed by the binocular camera, and then use the target tracking method based on spatiotemporal features to continuously identify the numbered circular targets, locate the pixel coordinates of the center of the circular targets, and then stereo match the two circular target images taken by the binocular camera at the same time according to the same numbered pixel coordinates, and perform spatial positioning according to the binocular vision measurement principle. 2. The method for spatially locating multiple circular targets based on binocular vision according to claim 1, wherein step 2 specifically comprises the following steps: Step 2-1: Filter and denoise the collected circular target image; The original target image is filtered and denoised using guided filtering, and the target image is guided based on the guidance signal to adjust the weight in the filtering process; Step 2-2: Binarization segmentation; The HSI color space is used to identify the markers. The denoised target image is converted from the RGB space to the HSI space. The red markers are extracted according to the range thresholds of each HSI component, as shown in formula (1): In the formula, H is the frequency that defines the color, that is, the hue; S represents the depth of the color, that is, the saturation; I represents the brightness; when each HSI component of the image pixel value satisfies the red threshold condition corresponding to formula (1) at the same time, the pixel point at the corresponding position is set to the foreground color 1, otherwise it is set to the black background 0, and then the morphological filtering method combining the opening operation and the closing operation is used to further remove the noise; Step 2-3: using a circularity criterion to identify a circular target area; specifically, identifying the circular target area by calculating the circularity of the target area; The circularity indicates the similarity between the target circle and the ideal circle, which is determined by calculating the perimeter and area of the target area. The calculation formula of the ratio of the perimeter and area of the target area is shown in formula (2): In the formula, C is the circularity of the target area, _Sc represents the area of the target area, _Lc is the contour perimeter of the target area, and β is the proportional coefficient. The expression is: S _c =πr ² , L _c =2πr (3) Where r is the radius of the target area. According to the circularity calculation formula, when C=1, the connected area is considered a circle, and the circularity C of the target area satisfies the following conditions: C _min ≤C ≤C _max (4) In the formula, C _min and C _max represent the circularity of the landmark point when the angle between the line connecting the circular target and the binocular camera and the optical axis of the binocular camera is the maximum and minimum respectively, and its value is expressed as: Where θ is the deflection angle of the binocular camera. 3. The method for spatially locating multiple circular targets based on binocular vision according to claim 1, wherein step 3 specifically comprises the following steps: Step 3-1: Use the Canny edge detection algorithm to roughly determine the edge of the identified circular target area; First, the binarized circular target image is smoothed by using a Gaussian filter. Secondly, the gradient operator is used to calculate the gradient of the image on the smoothed image to obtain the gradient strength and direction of each pixel in the image and obtain the gradient map. The gradient map is non-maximum suppressed by comparing the gradient strength of a pixel with the gradient strength of its two adjacent pixels in the gradient direction, and only the pixel with the local maximum gradient value is retained, thereby suppressing the response of the non-edge area. Then the gradient map is divided into two thresholds, a high threshold and a low threshold. According to the gradient strength of the pixel, the pixel is divided into three categories: strong edge, weak edge and non-edge. The strong edge is the real edge, the weak edge is part of the real edge, and the non-edge is the remaining area. For the weak edge, a continuous coarse edge is formed by connecting the strong edge pixels around it. Step 3-2: Use Zernike moments to perform sub-pixel edge detection on the roughly defined edge; Let A _nm represent the Zernike moment of the image f(x, y). Through the conversion formula, A _nm is expressed as: Where (n+1)/π is the normalization factor; is the Zernike moment polynomial, ρ is the vector distance from the origin to the pixel point (x, y); θ is the angle between ρ and the x-axis. If the image is rotated by angle θ, the Zernike moment A′ _nm of the rotated image f(x, y) satisfies the following formula: According to the above formula, the Zernike moments A ₀₀ , A ₁₁ and A ₂₀ of the roughly defined edge target image are calculated, and the Zernike moments A′ ₀₀ , A′ ₁₁ , A′ ₂₀ of the rotated image satisfy the following relationship: The four parameters h, k, l and The calculation formula is as follows: In formula (9), h is the background gray value, k is the gray step value, l is the vertical distance from the template center to the edge, and φ is the angle between the edge and the x-axis. Using an N×N Zernike moment template and the parameters calculated by the above formula, the i-th sub-pixel edge point P _i (x', y') is obtained, and its calculation formula is: Step 3-3: Introduce the edge threshold judgment criterion based on Otsu's method to determine the sub-pixel edge; The threshold judgment condition of the sub-pixel edge point based on the Zernike moment algorithm is as follows: l≤l _t ∩k≥k _t (11) In formula (11), _kt is the grayscale step threshold, _lt is the distance threshold, and its value is: The judgment criteria of the Otsu method are introduced. Based on the inter-class variance of the edge area and the background area, the basis for selecting the optimal threshold is to maximize the inter-class variance. Specifically, the Otsu method traverses all potential thresholds, calculates the inter-class variance under each threshold, and finally selects the threshold that can maximize the inter-class variance as the best choice. In a sign image containing n pixels, let the number of pixels with gradient amplitude i be n _i , therefore, the probability of the occurrence of gradient amplitude i is p _i = n _i /n. By calculating the threshold t, the image is divided into two categories as the foreground edge area and the background, and the corresponding probabilities are: Their means are: The average gradient amplitude is Then the between-class variance of the two classes is σ ² ＝ω ₀ (-μ _t ) ² +ω ₁ (μ ₀ -μ _t ) ² (15) When the variance σ ² is the largest, the difference between the foreground edge and the background is the largest, so the segmentation accuracy is the highest, and the grayscale step threshold k _t is the optimal value at this time; by using the optimal grayscale step threshold k _t and the distance threshold l _t to calculate the sub-pixel edge based on the Zernike moment, the sub-pixel edge point p′ _i (x', y') is finally obtained and stored in the edge point set {P _i }; Step 3-4: Sub-pixel edge refinement is performed using a grayscale gradient optimization-based method; The center value of the distribution in the grayscale gradient optimization method is the position where the gradient direction changes the most, that is, the position of the edge. First, the sub-pixel edge information is extracted by the improved sub-pixel Zernike moment algorithm in step 3-3, and then the least squares ellipse fitting method is used to perform center fitting on all sub-pixel edge points to obtain the rough center of the circular marker. Based on the rough center and the corresponding rough edge, the radial line equation corresponding to each edge pixel is obtained: y＝kx+b (16) in: Where (x ₀ , y ₀ ) represents the roughly determined center coordinates; ( _xi , _yi ) represents the coordinates of each edge point. For a certain edge point, the first-order derivative of the grayscale along the gradient direction is calculated along its radial direction, and the central grayscale value is obtained using the Gaussian function; where the expression of the Gaussian function is: In formula (18), a is the Gaussian proportional coefficient, b is the center value of the first-order derivative Gaussian distribution of the gradient grayscale, c ² is the variance of the first-order derivative Gaussian distribution of the gradient grayscale, and p′ _i (x _b , y _b ) is the position of the edge point after refinement. All edge points are refined in turn; Step 3-5: Use the least squares ellipse fitting method to locate the center of the sub-pixel edge and obtain the center pixel coordinates of the circular target; The sub-pixel edge point set {P _i } is processed using the least squares ellipse fitting method to locate the center of the circular target; the general equation of the ellipse is as follows: F(x)＝ax ² +bxy+cy ² +dx+ey+f＝0 (19) The edge points are fitted by the least squares ellipse fitting method to obtain the parameters A, B, C, D, E, and F. The coordinate calculation formula of the ellipse center point is: The final coordinates of the center point of the circular target are (x _r , y _r ). 4. The method for spatially locating multiple circular targets based on binocular vision according to claim 1, wherein step 4 specifically comprises the following steps: Step 4-1: Initially locate all circular targets, obtain their central pixel coordinate values, sort their pixel center coordinate values in ascending order of pixel x values, and store them in set C; divide set C into m subsets P ₁ , P ₂ , ... P _k , P _m , each subset contains n elements; further, sort each element in each subset P _k in ascending order of y values, k = 1, 2, ... k, n, and obtain Q _ki , k = 1, 2, ...., m, i = 1, 2, ..., n, Q _ki is the pixel coordinate of the circular target in the mth row and nth column, the elements in set C represent the pixel points of the target image sorted from top to bottom and from left to right, and the number of each central pixel point is the number of the corresponding marker; Step 4-2: Use the marker recognition method to process the image sequence and obtain the multi-marker extraction result image to distinguish the marker foreground and background space; calculate the spatial features based on the minimum enclosing rectangle and the temporal features based on the image time sequence of the target for each circular target, and use the circular marker of the monitoring point identified at the current moment and its minimum enclosing rectangle as the spatial feature operator of the target, and record the current moment as the time description operator. Finally, integrate the spatial feature operator and the time description operator to jointly construct the target feature vector A, as shown in the following formula: A＝{O _cen ，M _matrix ，T _fps } (21) Wherein, the structural attribute of A is a cell array, O _cen is the coordinate number of the center of the circular target pixel, M _matrix is the minimum circumscribed matrix of the circular mark, and T _fps is the time sequence of the binary circular mark at the current moment; Create a time series data set B, which consists of n feature vectors A, and its expression is as follows: B＝{A ₁ , A ₂ , … , _An } (22) A _n represents the feature vector of the circular target with sequence number n in the current image, and n represents the number of foreground targets; Step 4-3: By tracking the position of the circular target in the image, the same target in consecutive image frames is associated. The target feature vector is used in combination with the target circumscribed rectangle overlap OLD to achieve matching association of the circular target at different times. The calculation model of OLD is as follows: OLD＝K/( _PT +K+QT ₊₁ ) (23) In the formula, OLD is used to evaluate the similarity of two target regions. The higher its value, the greater the possibility that the two targets are the same object; K represents the similar area of the two target regions, which refers to the area of the two targets to be detected in consecutive moments respectively; the matching conditions for judging whether the two targets are successfully associated are as follows: OLD≥α (24) Among them, α represents the discrimination threshold. When the target to be detected meets the above similarity judgment conditions, the two targets are considered to be the same symbol. If not, they are considered to be different targets. If two targets at different time points are judged to be the same target, the target feature information in the latest image frame will replace the target feature associated with the previous moment in the time series cell group B. Step 4-4: Track the circular target with the initial number at continuous moments, obtain the circular target with the corresponding number in the left and right camera images in the sequence at the same time, realize stereo matching through the corresponding number, and finally solve its spatial coordinates by using the binocular vision measurement principle; The binocular camera uses two images taken at different angles to calculate the disparity map, and then obtains the three-dimensional information of the pixel through the disparity map. b is the baseline distance between the two cameras, f is the focal length of the camera lens, L is the imaging width, _XL and _XR respectively represent the pixel distances of the two imaging points _PL and _PR in the X-axis direction in their respective coordinate systems, and the corresponding difference in pixel coordinates is called disparity d. Z is the depth distance from the object point in space to the baseline of the two cameras; the disparity d is expressed as: d＝| _XL - _XR | (25) Then the distance between the two imaging points _PL and _PR is: According to the theory of similar triangles: Then the distance Z from point P to the projection center plane is obtained, which is expressed as follows: When a point P in three-dimensional space moves, its imaging points _PL and _PR on the left and right cameras will also change accordingly, and the parallax of the left and right cameras will change accordingly; the depth information of the point can be obtained based on the parallax principle, and the three-dimensional coordinates of the point in space can be obtained based on the principle of triangle similarity: