WO2023240805A1 - Connected vehicle overspeed early warning method and system based on filtering correction - Google Patents

Abstract

A connected vehicle overspeed early warning method and system based on filtering correction. A generation time deviation of a same target in sensing data of a laser radar and a camera is estimated, and the estimated time deviation distribution is used to perform filtering correction on the position of a point cloud target, thereby implementing mapping alignment of a multi-source target, and solving the problem of low accuracy of fusion between a point cloud and an image caused by the time deviation. In a continuous driving process, central point coordinates of a reference connected vehicle in point cloud and image data are marked; a central point of the reference connected vehicle in the point cloud is mapped to the image by using an affine transformation matrix; the generation time deviation of the target is derived by using a distance difference between a mapping point and the central point in the image; and an optimal position of the point cloud target of a connected vehicle is re-estimated by designing a confidence filtering method, thereby implementing vehicle overspeed recognition and early warning based on high-precision fusion between the point cloud and the image, and providing a technical support for safe driving of an intelligent connected vehicle.

Description

A speed warning method and system for connected vehicles based on filter correction Technical field

The present invention relates to the field of intelligent transportation technology, and in particular to a network-connected vehicle speed warning method and system based on filter correction.

Background technique

With the rapid development of smart transportation construction, smart connected vehicle-related technologies are gradually developing and emerging. Connected vehicles are an important part of the construction of smart parks and are also the main application of C-V2X (vehicle-to-road collaboration) technology. Safe driving of intelligent connected vehicles is an important topic, involving many aspects such as perception, coordination, decision-making, control, etc. Accurately sensing the surrounding environment and controlling the vehicle speed are the basic safe driving principles. Vehicle-road collaboration technology senses vehicle speed through roadside sensing equipment to further control the safe driving of connected vehicles. In the past, the method of monitoring vehicle speed through millimeter-wave radar was gradually abandoned due to the inability to accurately distinguish vehicles. It was replaced by a method of monitoring vehicle speed through fusion perception of lidar and cameras.

Currently, hardware line control is used to trigger the hardware time synchronization of lidar and camera. Due to uncertain factors such as lidar, camera sensor exposure mechanism, target movement, Ethernet transmission delay, data encoding and decoding, etc., the contents of the data frames collected by the two sensor devices are affected. They are not completely synchronized, and there is a certain range of deviation in the generation time of moving targets. As a result, the same target cannot be completely aligned when the two data are fused. Since the same target cannot be accurately associated, the detection accuracy of the vehicle speeding detection method based on the fusion of these two sensory data is reduced. lower. Therefore, the present invention proposes a method to improve the point cloud and image fusion alignment accuracy by estimating the time deviation of the same target generation in the lidar and camera sensing data, and using the estimated time deviation distribution to filter and correct the point cloud target position. , realize vehicle speeding recognition and early warning based on high-precision fusion of point clouds and images, and provide reliable technical support for connected vehicle safety monitoring based on multi-sensor fusion.

Contents of the invention

The purpose of this invention is to address the shortcomings of the existing technology and provide a network-connected vehicle overspeed warning method and system based on filter correction to solve the problem that the existing lidar and camera cannot match and align to the same target due to time deviation, resulting in problems based on lidar. The problem of low detection accuracy of the Internet-connected vehicle speed warning system that integrates multi-sensors with cameras. This invention marks the center point coordinates of the reference connected vehicle in the point cloud and image data during continuous driving, and uses an affine transformation matrix to map the center point of the reference connected vehicle in the point cloud to the image. Use the distance difference between the mapping point and the center point in the image to derive the generation time deviation of the target, and re-filter according to the time deviation distribution to estimate the optimal position of the center point of the point cloud network vehicle target, achieving high accuracy based on point cloud and image Fusion.

The object of the present invention is achieved through the following technical solution: a network-connected vehicle speed warning method based on filter correction, which method includes the following steps:

Step 1: Select a reference connected car, collect several frames of point cloud and image data of the reference connected car during continuous driving through the lidar and camera with data frame time synchronization, and mark the point where the reference connected car is For the center point coordinates in the cloud and image data, use the affine transformation matrix to map the center point of the reference connected vehicle in the point cloud to the image, and measure the mapping point on the image and the coordinates of the center point of the reference connected vehicle on the image. Position deviation, estimate the generation time deviation of the reference connected vehicle target in the point cloud and image, and calculate the time deviation distribution parameters;

Step 2: Acquire point cloud and image data in real time during the continuous driving of the road connected vehicle, and use the confidence filtering method to filter and correct the position of the point cloud center point for the point cloud connected vehicle target detected in any point cloud frame; specifically Calculate the confidence gain using the confidence score and time deviation distribution parameters of the point cloud network target detected by the detection algorithm, and re-filter to estimate the optimal position of the point cloud network target based on the confidence gain;

Step 3: Map the filtered and corrected point cloud connected vehicle targets to the corresponding image frames one by one, and calculate the mapping point coordinates of the center point of each point cloud connected vehicle target in the image and any connected vehicle in the image. The distance difference between the target center point coordinates. The one with the smallest distance difference and less than the threshold is the corresponding matching target in the image. In this way, the mapping, matching and alignment of all point clouds and connected vehicle targets in the image is completed;

Step 4: Fusion of the sensed information of the matched and aligned connected vehicle targets in point clouds and images to obtain the license plate number and instantaneous speed of the connected vehicle, and report the license plate number of the connected vehicle whose instantaneous speed exceeds the maximum speed limit to The connected vehicle cloud control platform also issues speeding warnings and remotely controls the connected vehicle to slow down to a safe speed.

Furthermore, in step one, hardware wired control is used to control the lidar and camera data frame time synchronization.

Further, in step one, it is assumed that the generation time deviation of the reference connected vehicle target in the point cloud to be estimated and the image is t, and the coordinates of the marked center point of the reference connected vehicle in the point cloud are (x, y, z) , the coordinates of the center point in the image are (u′, v′), the orientation angle is γ, the calculated instantaneous speed is v_t, the coordinates of the mapping point mapped from the center point of the point cloud to the image are (u, v), and the measurement The position deviation between the reference connected vehicle mapped to the mapping point coordinates (u, v) on the image and the center point coordinates (u′, v′) of the reference connected vehicle in the image is d;

Then the center point coordinates (x, y, z) of the reference connected car in the point cloud and the mapped point coordinates (u, v) mapped to the image satisfy the following relationship:

Among them, H is the affine transformation matrix from point cloud to image. The dimension of H matrix is 3*4, which is obtained by the joint external parameter calibration of lidar and camera and the internal parameter calibration of camera. It is expressed as follows: where h _rs is the element in the H matrix, All are real numbers, 1≤r≤4, 1≤s≤4;

According to the instantaneous driving speed of the reference connected vehicle, the coordinates x′, y′, and z′ of the reference connected vehicle’s position in the point cloud after moving within the time deviation t are obtained:

x′＝x+v_t*t*cosγ

y′＝y+v_t*t*cosγ

z′=z

where v_t is the instantaneous speed of the reference connected vehicle, (x′, y′, z′) is the position coordinate of the reference connected vehicle after movement;

Then the position coordinates (x′, y′, z′) of the reference connected vehicle after movement and the center point coordinates (u′, v′) in the image satisfy the following relationship:

Therefore, according to the measured position deviation d of the reference connected car mapped to the mapping point coordinates (u, v) on the image and the reference connected car’s center point coordinates (u′, v′) in the image, it is listed as follows Equation:

It can be deduced that the time deviation t is a value related to the known affine transformation matrix, point cloud center point coordinates, heading angle, instantaneous speed, and position deviation, expressed as follows:

Among them, the capital letters A and B are:

Further, the instantaneous driving speed of the reference connected vehicle is calculated by the moving distance of the center point of the reference connected vehicle target in the two frames before and after the nearest neighbor and the frame interval time ratio.

Further, the specific process of calculating the time deviation distribution parameters is as follows:

(1) Use the Kolmogorov-Smirnov test method to detect whether the time deviation conforms to the normal distribution; assuming that the estimated time deviation data is N groups, calculate the mean of the data as μ, the variance as σ ² , and set the detection significance level as α; Use the Kolmogorov-Smirnov test method to detect the probability that the data does not obey the normal distribution, that is, the P value. If the P value is less than or equal to the significance level, the time deviation does not conform to the normal distribution. If the P value is greater than the significance level, the time deviation conforms to the normal distribution. normal distribution;

( ² ) If the time deviation conforms to the normal distribution, its normal distribution expression is recorded as

(3) If the time deviation does not conform to the normal distribution, sort the time deviation data according to the numerical size from small to large, and for all data whose numerical size is between the second quartile and the third quartile, Calculate the median and variance.

Further, step two includes the following steps:

(1) For the k-th point cloud network connected vehicle target, the center point position coordinates detected by the deep learning detection algorithm are (x _k , y _k , z _k ), the orientation angle is γ _k , and the confidence score is c, The calculated instantaneous speed is v _k ;

(2) Calculate the confidence gain based on the time deviation distribution parameters, specifically:

(2.1) If the time deviation conforms to the normal distribution, assuming that the parameter mean in the normal distribution expression is u _t and the variance is σ _t ² ; then the confidence gain ε is:

Then based on the confidence gain re-filtering, the horizontal and vertical coordinates x _k ′ and y _k ′ of the point cloud center point of the connected vehicle target are estimated as:

x _k ′＝x _k *ε+(x _k +v _k *u _t *cosγ _k )(1-ε)

y _k ′＝y _k *ε+(y _k +v _k *u _t *sinγ _k )(1-ε)

方差为τ _t ²，则置信增益ε′为： (2.2) If the time deviation does not conform to the normal distribution, assuming that the time deviation is sorted from small to large, the median of the data between the second quartile and the third quartile is

The variance is τ _t ² , then the confidence gain ε′ is:

Then based on the confidence gain filter, the horizontal and vertical coordinates x _k ′ and y _k ′ of the point cloud center point of the connected vehicle target are re-estimated as:

(3) Since the connected car is a rigid object, its position movement will not change its value on the vertical coordinate, that is, the z-axis. That is, the vertical coordinate z _k ′ = z _k re-estimated based on the confidence gain filter, then re-filtering The estimated optimal position center point coordinates of the connected vehicle target are (x _k ′, y _k ′, z _k ′).

Further, step three includes the following steps:

(1) For the filtered and corrected point cloud connected vehicle target at any position, use the affine transformation matrix to map the coordinates of the point cloud center point to the image;

(2) Calculate the distance difference between the mapping point coordinates of the center point of each point cloud connected vehicle target in the image and the coordinates of the center point of any connected vehicle target in the image;

Assume that the coordinates of the corrected point cloud center point mapped to the mapped point on the image are (p, q), and the coordinates of the center point of the i-th connected vehicle target in the image are (mi _, n _i ), and 1≤i≤N _{img_obj} , N _{img_obj} is the total number of connected vehicle targets in the image;

Then the distance difference d _i between the coordinates of the mapping point and the coordinates of the center point of the i-th connected vehicle target in the image is:

(3) Calculate the minimum distance difference and determine whether the minimum distance difference is less than the set threshold Δ; where the minimum distance difference is:

If the minimum distance difference d _min is less than the threshold Δ, then the connected vehicle target in the corresponding image is the matching target;

(4) According to steps (1)-(3), complete the mapping, matching and alignment of all point cloud connected vehicle targets and image network connected vehicle targets.

Further, in step four, the OCR license plate number recognition method is used based on the image data to identify the license plate number of the connected vehicle target in the image.

On the other hand, the present invention also provides a network-connected vehicle speed warning system based on filter correction. The system includes a time deviation distribution parameter determination module, a filter correction module, a matching alignment module and a perception information fusion module;

The time deviation distribution parameter determination module is used to select a reference connected vehicle, collect several frames of point cloud and image data of the reference connected vehicle during continuous driving through the laser radar and camera with data frame time synchronization, and mark them Determine the coordinates of the center point of the reference connected vehicle in the point cloud and image data, use the affine transformation matrix to map the center point of the reference connected vehicle in the point cloud to the image, and measure the mapping points on the image and the reference connected vehicle. Based on the position deviation of the center point coordinates on the image, estimate the generation time deviation of the reference connected vehicle target in the point cloud and image, and calculate the time deviation distribution parameters;

The filter correction module is used to obtain point cloud and image data in real time during the continuous driving of road Internet-connected vehicles, and use the confidence filtering method to determine the position of the point cloud center point for the point cloud Internet-connected vehicle target detected in any point cloud frame. Filter correction; specifically: use the confidence score of the point cloud network vehicle target detected by the detection algorithm and the time deviation distribution parameter obtained by the time deviation distribution parameter determination module to calculate the confidence gain, and re-filter the estimated point cloud network connection based on the confidence gain The optimal position of the car target;

The matching and alignment module is used to map the point cloud network vehicle targets corrected by the filter correction module to the corresponding image frames one by one, and calculate the mapping point coordinates and image coordinates of the center point of each point cloud network vehicle target in the image. The distance difference between the coordinates of the center point of any connected vehicle target. The one with the smallest distance difference and less than the threshold is its corresponding matching target in the image. In this way, the mapping, matching and alignment of all point clouds and connected vehicle targets in the image is completed. ;

The perceptual information fusion module is used to fuse the perceptual information of the connected vehicle target in the point cloud and image after matching and alignment by the matching alignment module, so as to obtain the license plate number and instantaneous speed of the connected vehicle. If the instantaneous speed exceeds the maximum speed limit, The license plate number information of the connected vehicle is reported to the connected vehicle cloud control platform, and an overspeed warning is issued, and the connected vehicle is remotely controlled to slow down to a safe speed.

The beneficial effect of the present invention is that the present invention proposes a network-connected vehicle overspeed warning method and system based on filter correction, and uses a deep learning target detection method to detect connected vehicle targets in moving point clouds and image data. The position of the moving network vehicle in the continuous video frame is filtered and corrected to achieve fusion matching and alignment of the same moving target, which significantly reduces the problem of low accuracy of speeding detection methods based on lidar and camera fusion due to time deviation. Its implementation method is simple and efficient, and can be effectively applied to the safety monitoring of speeding driving of connected vehicles based on multi-sensor fusion, providing reliable technical support for accurate decision-making in safe driving management of intelligent connected vehicles.

Description of the drawings

Figure 1 is a flow chart of a speed warning method for connected vehicles based on filter correction according to the present invention.

Figure 2 is a schematic diagram of the position deviation of the reference connected vehicle point cloud detection frame mapped to the mapping frame on the image and the image detection frame under different time deviations.

Figure 3 is a schematic diagram of the pseudo 3D frame mapping the position-corrected point cloud reference connected vehicle to the image.

Figure 4 is a schematic structural diagram of a network-connected vehicle speed warning system based on filter correction according to the present invention.

Figure 5 is a schematic structural diagram of a network-connected vehicle speed warning device based on filter correction according to the present invention.

Detailed ways

The present invention will be described in detail below based on the accompanying drawings, and the purpose and effects of the present invention will become clearer. It should be understood that the specific embodiments described here are only used to explain the present invention and are not intended to limit the present invention.

As shown in Figure 1, the present invention proposes a network-connected vehicle speed warning method based on filter correction, which is used to solve the network-connected vehicle speed warning process based on the fusion of lidar and camera. Since the moving target is in the point cloud and image data, The problem of low detection accuracy caused by incomplete fusion alignment. The fusion method is a decision-level fusion method, that is, by separately detecting the position and speed information of the connected vehicle target in the point cloud and the license plate number information in the image, and then matching and aligning the connected vehicle targets in the two data sources. Fusion target information in two data sources to achieve the purpose of fusion using the sensing characteristics of multi-source heterogeneous data. The method includes the following steps:

Step 1: Install solid-state lidar and cameras at the park's road network vehicle speed monitoring points, and use hardware line control to control lidar and camera data frame time synchronization.

The product model of the above-mentioned solid-state lidar is DJI AVIA, which uses a non-repetitive scanning method, with a horizontal FOV of 70.4° and a vertical FOV of 77.2°. One frame of point cloud data contains 48,000 reflection points. The camera is a webcam. The two are installed in the same direction on the vertical pole of the park's networked vehicle speed monitoring point. The camera and lidar are controlled to be exposed simultaneously through hardware wire control. The data collection frequency is 10HZ. However, due to the exposure mechanism of the two sensors and the target movement , Ethernet transmission delay, data encoding and decoding and other uncertain factors cause the content of the data frames collected by the two sensor devices to be not completely synchronized, resulting in a certain range of deviation in the generation time of the moving target, resulting in the same target not being completely fused when the two data are fused. Alignment, so the time offset of the data frame needs to be estimated first.

Select a reference connected vehicle to drive multiple times into the monitoring points of solid-state lidar and cameras, and mark the center point coordinates of the reference connected vehicle in the point cloud and image data during the continuous driving process, using affine The transformation matrix maps the center point of the reference connected vehicle in the point cloud data to the image, measures the position deviation between the mapping point and the coordinates of the center point of the reference connected vehicle on the image, and estimates the target generation time deviation. The specific process is as follows:

(1) Select a reference connected vehicle to drive into the monitoring point multiple times, collect several frames of point cloud and image data of the reference connected vehicle during continuous driving, and mark the point cloud of the reference connected vehicle in each frame. and the coordinates of the center point in the image data.

(2) For any pair of synchronized data frames, measure the position deviation between the center point of the reference networked vehicle point cloud mapped to the mapped point coordinates in the image and the center point coordinates in the image data, and estimate the point cloud based on the position deviation and target generation time bias in image data.

Assume that the target generation time deviation to be estimated is t, the center point coordinates of the marked reference connected car in the point cloud data are (x, y, z), and the center point coordinates in the image data are (u′, v ′), the heading angle is γ, the calculated instantaneous speed is v_t, the point cloud center point is mapped to the mapping point coordinates on the image is (u, v), the measured reference connected vehicle is mapped to the mapping point coordinates on the image The position deviation between (u, v) and the center point coordinates (u′, v′) of the reference connected vehicle in the image is d.

Then the center point coordinates (x, y, z) of the reference connected car in the point cloud data and the mapped point coordinates (u, v) mapped to the image satisfy the following relationship:

Among them, H is the affine transformation matrix from point cloud to image. The dimension of H matrix is 3*4. It can be obtained by joint external parameter calibration of lidar and camera and internal parameter calibration of camera. It can be expressed as follows, where h _rs are the elements in the H matrix. , all are real numbers, 1≤r≤4, 1≤s≤4.

According to the instantaneous driving speed of the connected vehicle, it can be obtained that the coordinates x′, y′, and z′ of the center point of the point cloud after the reference connected vehicle moves within the time deviation t are:

x′＝x+v-t*t*cosγ

y′＝y+v-t*t*cosγ

z′=z

where v_t is the instantaneous speed of the reference connected vehicle, which can be calculated by the moving distance of the center point of the connected vehicle target in the two frames before and after the nearest neighbor and the frame interval time ratio, (x′, y′, z′) is the reference The position coordinates of the connected vehicle after it moves. Since the connected vehicle is a rigid object, its vertical coordinate, that is, the value of the z-axis, does not change with the movement of the vehicle's position.

Then the center point coordinates (x′, y′, z′) of the position point cloud of the reference connected vehicle after movement and the center point coordinates (u′, v′) in the image data satisfy the following relationship:

Therefore, according to the measured position deviation d of the reference connected vehicle mapped to the mapping point coordinates (u, v) on the image and the reference connected vehicle's center point coordinates (u′, v′) in the image data, it can be listed Get the following equation:

According to formulas (1), (2), and (3), it can be deduced that the time deviation t is a value related to the known affine transformation parameters, point cloud center point coordinates, heading angle, instantaneous speed, and position deviation. Expressed as follows:

Among them, the capital letters A and B are:

As shown in Figure 2, the reference connected vehicle target in the point cloud has not been position corrected. Under different time deviations, the reference connected vehicle point cloud detection frame is mapped to the mapping frame on the image and the reference connected vehicle detection frame in the image. When the time deviation is small, the overlap between the mapping frame and the detection frame in the image is high. When the time deviation is large, the position deviation between the mapping frame and the detection frame in the image is large, and there is almost no overlap.

Assuming that the estimated time deviations are N groups, use the Kolmogorov-Smirnov test method to detect whether the time deviations conform to the normal distribution. If the time deviation conforms to the normal distribution, find its normal distribution expression. If it does not conform to the normal distribution, calculate the second quartile and third quartile of the time deviation sorted by numerical value. The median and variance of the data.

The above Kolmogorov-Smirnov test method is often used to detect whether a certain data distribution conforms to a certain distribution, here it is a normal distribution. By estimating the P value of a certain data distribution, it is determined whether the assumption that the data conforms to a normal distribution is true. If P If the value is greater than the significance level, the hypothesis is considered to be true, otherwise it is not true. The specific process is as follows:

(1) Use the Kolmogorov-Smirnov test method to detect whether the time deviation conforms to the normal distribution. For N groups of time deviation data, calculate the mean value of the data as μ, the variance as σ ² , and set the detection significance level as α. Use the Kolmogorov-Smirnov test method to detect the probability that the set of data does not obey the normal distribution, that is, the P value. If the P value is less than or equal to the significance level, then the time deviation does not conform to the normal distribution. If the P value is greater than the significance level, then the time deviation Deviations fit a normal distribution. Among them, the N value is greater than or equal to 100.

(2) If the time deviation conforms to the normal distribution, its normal distribution expression can be recorded as X~N(μ, σ ² ). Among them, X represents these N sets of time deviation data.

(3) If the time deviation does not conform to the normal distribution, sort the time deviation data according to the numerical size from small to large, and for all data whose numerical size is between the second quartile and the third quartile, Calculate the median and variance.

Step 2: Obtain point cloud and image data in real time during the continuous driving of the connected vehicle on the park road, and detect the center point, heading angle, confidence score, instantaneous speed, and image connected vehicle target of the point cloud connected vehicle target. license plate number.

Specifically, the CenterPoint three-dimensional target detection algorithm based on the point cloud data is used to detect the center point, orientation angle, and confidence score of the connected vehicle target, and based on the center point movement distance and frame of the connected vehicle target in the two frames before and after the nearest neighbor. The interval time ratio calculates the instantaneous speed of the connected vehicle; the OCR recognition method is used to identify the connected vehicle target in the image data and identify the license plate number of the connected vehicle.

The above three-dimensional target detection algorithms include image-based generation, point cloud-based generation, and image-point cloud fusion generation. Here, the target detection algorithm based on the CenterPoint network model is used, which is only based on point cloud generation and labels a large amount of collected point cloud data. The annotated data is divided into a training set, a verification set, and a test set. The model trained on the training set has an accuracy mAP value of up to 91% on the test set, within a 50m (unit: meter) range of the point cloud data center. The detection rate of targets within is 95%. The detection accuracy of the OCR recognition method reaches 99%. The orientation angle value range is (-π, π), and the confidence score value range is (0, 1).

Based on the confidence filtering method, the detected center point position of the point cloud network vehicle target is filtered and corrected. That is, the confidence score and time deviation distribution parameters of the connected vehicle target detected by the detection algorithm are used to calculate the confidence gain, and the optimal position of the connected vehicle target is estimated by re-filtering based on the confidence gain. The specific process is as follows:

(1) For the kth point cloud network connected vehicle target, assume that the center point position coordinates detected using the deep learning detection algorithm are (x _k , y _k , z _k ), the orientation angle is γ _k , and the confidence score is c , the calculated instantaneous speed is v _k .

(2) Calculate the confidence gain based on the time deviation distribution parameters, specifically:

(2.1) If the time deviation conforms to the normal distribution, assume that the parameter mean in the normal distribution expression is u _t and the variance is σ _t ² . Then the confidence gain ε is:

Then based on the confidence gain re-filtering, the horizontal and vertical coordinates x _k ′ and y _k ′ of the point cloud center point of the connected vehicle target are estimated as:

x _k ′＝x _k *ε+(x _k +v _k *u _t *cosγ _k )(1-ε)

y _k ′＝y _k *ε+(y _k +v _k *u _t *sinγ _k )(1-ε)

方差为τ _t ²，则置信增益ε′为： (2.2) If the time deviation does not conform to the normal distribution, assuming that the time deviation is sorted from small to large, the median of the data between the second quartile and the third quartile is

The variance is τ _t ² , then the confidence gain ε′ is:

Then based on the confidence gain filter, the horizontal and vertical coordinates x _k ′ and y _k ′ of the point cloud center point of the connected vehicle target are re-estimated as:

(3) Since the connected vehicle is a rigid object, its position movement will not change its value on the vertical coordinate, that is, the z-axis, that is, z _k ′ = z _k , then the estimated maximum value of the connected vehicle target after re-filtering will not be changed. The coordinates of the optimal location center point are (x _k ′, y _k ′, z _k ′).

Step 3: Map the filtered and corrected point cloud connected vehicle targets to the corresponding image frames one by one, and calculate the mapping point coordinates of the center point of each point cloud connected vehicle target in the image and any connected vehicle in the image. The distance difference between the coordinates of the target center point is the smallest and less than the threshold, which is the corresponding matching target in the image. In this way, the mapping, matching and alignment of all point clouds and connected vehicle targets in the image is completed. As shown in Figure 3, it is a schematic diagram of the pseudo 3D frame mapped from the position-corrected point cloud reference connected vehicle to the image. The specific process of this step is as follows:

(1) For the filtered and corrected point cloud connected vehicle target at any position, use the affine transformation matrix to map the coordinates of the point cloud center point to the image.

(2) Calculate the distance difference between the coordinates of the mapping point of each point cloud connected vehicle target in the image and the coordinates of the center point of any connected vehicle target in the image.

Assume that the corrected point cloud center point coordinates are (x′, y′, z), and the mapping point coordinates mapped to the image are (p, q), then the mapping point coordinates and the point cloud center point coordinates satisfy the following relationship:

Among them, H is the affine transformation matrix from point cloud to image. The dimension of H matrix is 3*4. It can be obtained by joint external parameter calibration of lidar and camera and internal parameter calibration of camera. It can be expressed as follows, where h _rs are the elements in the H matrix. , all are real numbers, 1≤r≤4, 1≤s≤4:

Assume that the center point coordinates of the i-th connected vehicle target in the image are (mi _, n _i ), and 1≤i≤N _{img_obj} , N _{img_obj} is the total number of connected vehicle targets in the image. Then the distance difference between the coordinates of the mapping point and the coordinates of the center point of the i-th connected vehicle target in the image is:

(3) Calculate the minimum distance difference and determine whether the minimum distance difference is less than the set threshold Δ. The minimum distance difference is:

If the minimum distance difference d _min is less than the threshold Δ, then the connected vehicle target in the corresponding image is the matching target.

(4) According to steps (1)-(3), complete the mapping, matching and alignment of all point cloud connected vehicle targets and image network connected vehicle targets.

Step 4: Fusion of the sensed information of the matched and aligned connected vehicle targets in point clouds and images to obtain the license plate number and instantaneous speed information of the same target connected vehicle, and identify the license plate of the connected vehicle whose instantaneous speed exceeds the maximum speed limit. The information is reported to the connected vehicle cloud control platform and an overspeed warning is issued, and the connected vehicle is remotely controlled to slow down to a normal speed. The maximum speed limit is the maximum speed limit for connected vehicles specified in the park, which is 30km/h, and the regular speed is 25km/h.

The connected vehicle cloud control platform is based on a cloud server, provides connected vehicle management and control functions, contains information about each connected vehicle and can remotely control specific connected vehicles.

On the other hand, as shown in Figure 4, the present invention also provides a network-connected vehicle speed warning system based on filter correction. The system includes a time deviation distribution parameter determination module, a filter correction module, a matching alignment module and a perception information fusion module. ;

The time deviation distribution parameter determination module is used to select a reference connected vehicle, and obtain the center of the point cloud and image data of several frames of the reference connected vehicle during continuous driving through the laser radar and camera with data frame time synchronization. Point coordinates, use the affine transformation matrix to map the center point of the reference connected vehicle in the point cloud to the image, measure the position deviation between the mapping point on the image and the coordinates of the center point of the reference connected vehicle on the image, and estimate the reference Generate time deviations of connected vehicle targets in point clouds and images, and calculate time deviation distribution parameters; for the specific implementation process of the time deviation distribution parameter determination module, refer to a speed warning method for connected vehicles based on filter correction provided by the present invention Detailed description of step one.

The filter correction module is used to obtain point cloud and image data in real time during the continuous driving of road Internet-connected vehicles, and use the confidence filtering method to determine the position of the point cloud center point for the point cloud Internet-connected vehicle target detected in any point cloud frame. Filter correction; specifically: use the confidence score of the point cloud network vehicle target detected by the detection algorithm and the time deviation distribution parameter obtained by the time deviation distribution parameter determination module to calculate the confidence gain, and re-filter the estimated point cloud network connection based on the confidence gain The optimal position of the vehicle target; for the specific implementation process of the filter correction module, refer to the detailed description of step 2 in the speed warning method for connected vehicles based on filter correction provided by the present invention.

The matching and alignment module is used to map the point cloud network vehicle targets corrected by the filter correction module to the corresponding image frames one by one, and calculate the mapping point coordinates and image coordinates of the center point of each point cloud network vehicle target in the image. The distance difference between the coordinates of the center point of any connected vehicle target. The one with the smallest distance difference and less than the threshold is its corresponding matching target in the image. In this way, the mapping, matching and alignment of all point clouds and connected vehicle targets in the image is completed. ; For the specific implementation process of the matching and alignment module, refer to the detailed description of step three in the network-connected vehicle speed warning method based on filter correction provided by the present invention.

The perceptual information fusion module is used to fuse the perceptual information of the connected vehicle target in the point cloud and image after matching and alignment by the matching alignment module, so as to obtain the license plate number and instantaneous speed of the connected vehicle. If the instantaneous speed exceeds the maximum speed limit, The license plate number information of the connected vehicle is reported to the connected vehicle cloud control platform, and an overspeed warning is issued, and the connected vehicle is remotely controlled to slow down to a safe speed. For the specific implementation process of the perceptual information fusion module, refer to the detailed description of step 4 in the network-connected vehicle speed warning method based on filter correction provided by the present invention.

Corresponding to the aforementioned embodiments of the overspeed warning method for connected vehicles based on filter correction, the present invention also provides embodiments of an overspeed warning device for connected vehicles based on filter correction.

Referring to Figure 5, an embodiment of the present invention provides a network-connected vehicle speed warning device based on filter correction, including a memory and one or more processors. The memory stores executable code, and the processor executes the When the code is executable, it is used to implement the network-connected vehicle speed warning method based on filter correction in the above embodiment.

The embodiments of the network-connected vehicle speed warning device based on filter correction of the present invention can be applied to any device with data processing capabilities, and any device with data processing capabilities can be a device or device such as a computer. The device embodiments may be implemented by software, or may be implemented by hardware or a combination of software and hardware. Taking software implementation as an example, as a logical device, it is formed by reading the corresponding computer program instructions in the non-volatile memory into the memory and running them through the processor of any device with data processing capabilities. From the hardware level, as shown in Figure 5, it is a hardware structure diagram of any device with data processing capabilities where the networked vehicle speed warning device based on filter correction of the present invention is located. In addition to the processor and memory shown in Figure 5 , network interfaces, and non-volatile memory, any device with data processing capabilities where the device in the embodiment is located may also include other hardware based on the actual functions of any device with data processing capabilities. This will not be discussed here. Repeat.

For details on the implementation process of the functions and effects of each unit in the above device, please refer to the implementation process of the corresponding steps in the above method, and will not be described again here.

As for the device embodiment, since it basically corresponds to the method embodiment, please refer to the partial description of the method embodiment for relevant details. The device embodiments described above are only illustrative. The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in One location, or it can be distributed across multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of the present invention. Persons of ordinary skill in the art can understand and implement the method without any creative effort.

Embodiments of the present invention also provide a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the speed warning method for connected vehicles based on filter correction in the above embodiments is implemented.

The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities as described in any of the foregoing embodiments, such as a hard disk or a memory. The computer-readable storage medium can also be an external storage device of any device with data processing capabilities, such as a plug-in hard disk, smart memory card (Smart Media Card, SMC), SD card, flash memory card equipped on the device (Flash Card) etc. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and can also be used to temporarily store data that has been output or is to be output.

The above embodiments are used to illustrate the present invention, rather than to limit the present invention. Within the spirit of the present invention and the protection scope of the claims, any modifications and changes made to the present invention fall within the protection scope of the present invention.

Claims (9)

A speed warning method for connected vehicles based on filter correction, characterized in that the method includes the following steps: Step 1: Select a reference connected car, collect several frames of point cloud and image data of the reference connected car during continuous driving through the lidar and camera with data frame time synchronization, and mark the point where the reference connected car is For the center point coordinates in the cloud and image data, use the affine transformation matrix to map the center point of the reference connected vehicle in the point cloud to the image, and measure the mapping point on the image and the coordinates of the center point of the reference connected vehicle on the image. Position deviation, estimate the generation time deviation of the reference connected vehicle target in the point cloud and image, and calculate the time deviation distribution parameters; Step 2: Acquire point cloud and image data in real time during the continuous driving of the road connected vehicle, and use the confidence filtering method to filter and correct the position of the point cloud center point for the point cloud connected vehicle target detected in any point cloud frame; specifically Calculate the confidence gain using the confidence score and time deviation distribution parameters of the point cloud network target detected by the detection algorithm, and re-filter to estimate the optimal position of the point cloud network target based on the confidence gain; Step 3: Map the filtered and corrected point cloud connected vehicle targets to the corresponding image frames one by one, and calculate the mapping point coordinates of the center point of each point cloud connected vehicle target in the image and any connected vehicle in the image. The distance difference between the target center point coordinates. The one with the smallest distance difference and less than the threshold is the corresponding matching target in the image. In this way, the mapping, matching and alignment of all point clouds and connected vehicle targets in the image is completed; Step 4: Fusion of the sensed information of the matched and aligned connected vehicle targets in point clouds and images to obtain the license plate number and instantaneous speed of the connected vehicle, and report the license plate number of the connected vehicle whose instantaneous speed exceeds the maximum speed limit to The connected vehicle cloud control platform also issues speeding warnings and remotely controls the connected vehicle to slow down to a safe speed. A network-connected vehicle speed warning method based on filter correction according to claim 1, characterized in that in step one, a hardware line control method is used to control the lidar and camera data frame time synchronization. A speed warning method for connected vehicles based on filter correction according to claim 1, characterized in that in step one, assuming that the point cloud to be estimated and the reference connected vehicle target generation time deviation in the image are t, mark The center point coordinates of the reference connected car in the point cloud are (x, y, z), the center point coordinates in the image are (u′, v′), the heading angle is γ, and the calculated instantaneous speed is v_t , the center point of the point cloud is mapped to the mapping point coordinates on the image (u, v), the measured reference connected car is mapped to the mapping point coordinates (u, v) on the image and the center of the reference connected car in the image The position deviation of point coordinates (u′, v′) is d; Then the center point coordinates (x, y, z) of the reference connected car in the point cloud and the mapped point coordinates (u, v) mapped to the image satisfy the following relationship:

Among them, H is the affine transformation matrix from point cloud to image. The dimension of H matrix is 3*4, which is obtained by the joint external parameter calibration of lidar and camera and the internal parameter calibration of camera. It is expressed as follows: where h _rs is the element in the H matrix, All are real numbers, 1≤r≤4, 1≤s≤4;

According to the instantaneous driving speed of the reference connected vehicle, the coordinates x′, y′, and z′ of the reference connected vehicle’s position in the point cloud after moving within the time deviation t are obtained: x′＝x+v_t*t*cosγ y′＝y+v_t*t*cosγ z′=z where v_t is the instantaneous speed of the reference connected vehicle, (x′, y′, z′) is the position coordinate of the reference connected vehicle after movement; Then the position coordinates (x′, y′, z′) of the reference connected vehicle after movement and the center point coordinates (u′, v′) in the image satisfy the following relationship:

Therefore, according to the measured position deviation d of the reference connected car mapped to the mapping point coordinates (u, v) on the image and the reference connected car’s center point coordinates (u′, v′) in the image, it is listed as follows Equation:

It can be deduced that the time deviation t is a value related to the known affine transformation matrix, point cloud center point coordinates, heading angle, instantaneous speed, and position deviation, expressed as follows:

Among them, the capital letters A and B are:

A speed warning method for connected vehicles based on filter correction according to claim 3, characterized in that the instantaneous driving speed of the reference connected vehicle is determined by the moving distance of the center point of the target connected vehicle in the two frames before and after the nearest neighbor. Calculated from the ratio of frame interval time. A speed warning method for connected vehicles based on filter correction according to claim 1, characterized in that the specific process of calculating the time deviation distribution parameters is as follows: (1) Use the Kolmogorov-Smirnov test method to detect whether the time deviation conforms to the normal distribution; assuming that the estimated time deviation data is N groups, calculate the mean of the data as μ, the variance as σ ² , and set the detection significance level as α; Use the Kolmogorov-Smirnov test method to detect the probability that the data does not obey the normal distribution, that is, the P value. If the P value is less than or equal to the significance level, the time deviation does not conform to the normal distribution. If the P value is greater than the significance level, the time deviation conforms to the normal distribution. normal distribution; ( ² ) If the time deviation conforms to the normal distribution, its normal distribution expression is recorded as (3) If the time deviation does not conform to the normal distribution, sort the time deviation data according to the numerical size from small to large, and for all data whose numerical size is between the second quartile and the third quartile, Calculate the median and variance. A network-connected vehicle speed warning method based on filter correction according to claim 5, characterized in that step two includes the following steps: (1) For the k-th point cloud network connected vehicle target, the center point position coordinates detected by the deep learning detection algorithm are (x _k , y _k , z _k ), the orientation angle is γ _k , and the confidence score is c, The calculated instantaneous speed is v _k ; (2) Calculate the confidence gain based on the time deviation distribution parameters, specifically: (2.1) If the time deviation conforms to the normal distribution, assuming that the parameter mean in the normal distribution expression is u _t and the variance is σ _t ² ; then the confidence gain ε is:

Then based on the confidence gain re-filtering, the horizontal and vertical coordinates x _k ′ and y _k ′ of the point cloud center point of the connected vehicle target are estimated as: x _k ′＝x _k *ε+(x _k +v _k *u _t *cosγ _k )(1-ε) y _k ′＝y _k *ε+(y _k +v _k *u _t *sinγ _k )(1-ε) (2.2) If the time deviation does not conform to the normal distribution, assuming that the time deviation is sorted from small to large, the median of the data between the second quartile and the third quartile is

The variance is τ _t ² , then the confidence gain ε′ is:

Then based on the confidence gain filter, the horizontal and vertical coordinates x _k ′ and y _k ′ of the point cloud center point of the connected vehicle target are re-estimated as:

(3) Since the connected car is a rigid object, its position movement will not change its value on the vertical coordinate, that is, the z-axis. That is, the vertical coordinate z _k ′ = z _k re-estimated based on the confidence gain filter, then re-filtering The estimated optimal position center point coordinates of the connected vehicle target are (x _k ′, y _k ′, z _k ′). A network-connected vehicle speed warning method based on filter correction according to claim 1, characterized in that step three includes the following steps: (1) For the filtered and corrected point cloud connected vehicle target at any position, use the affine transformation matrix to map the coordinates of the point cloud center point to the image; (2) Calculate the distance difference between the mapping point coordinates of the center point of each point cloud connected vehicle target in the image and the coordinates of the center point of any connected vehicle target in the image; Assume that the coordinates of the corrected point cloud center point mapped to the mapped point on the image are (p, q), and the coordinates of the center point of the i-th connected vehicle target in the image are (mi _, n _i ), and 1≤i≤N _{img_obj} , N _{img_obj} is the total number of connected vehicle targets in the image; Then the distance difference d _i between the coordinates of the mapping point and the coordinates of the center point of the i-th connected vehicle target in the image is:

(3) Calculate the minimum distance difference and determine whether the minimum distance difference is less than the set threshold Δ; where the minimum distance difference is:

If the minimum distance difference d _min is less than the threshold Δ, then the connected vehicle target in the corresponding image is the matching target; (4) According to steps (1)-(3), complete the mapping, matching and alignment of all point cloud connected vehicle targets and image network connected vehicle targets. A speed warning method for connected vehicles based on filter correction according to claim 1, characterized in that in step four, the OCR license plate number recognition method is used to identify the license plate number of the connected vehicle target in the image based on the image data. A network-connected vehicle speed warning system based on filter correction, characterized in that the system includes a time deviation distribution parameter determination module, a filter correction module, a matching alignment module and a perception information fusion module; The time deviation distribution parameter determination module is used to select a reference connected vehicle, collect several frames of point cloud and image data of the reference connected vehicle during continuous driving through the laser radar and camera with data frame time synchronization, and mark them Determine the coordinates of the center point of the reference connected vehicle in the point cloud and image data, use the affine transformation matrix to map the center point of the reference connected vehicle in the point cloud to the image, and measure the mapping points on the image and the reference connected vehicle. Based on the position deviation of the center point coordinates on the image, estimate the generation time deviation of the reference connected vehicle target in the point cloud and image, and calculate the time deviation distribution parameters; The filter correction module is used to obtain point cloud and image data in real time during the continuous driving of road Internet-connected vehicles, and use the confidence filtering method to determine the position of the point cloud center point for the point cloud Internet-connected vehicle target detected in any point cloud frame. Filter correction; specifically: use the confidence score of the point cloud network vehicle target detected by the detection algorithm and the time deviation distribution parameter obtained by the time deviation distribution parameter determination module to calculate the confidence gain, and re-filter the estimated point cloud network connection based on the confidence gain The optimal position of the car target; The matching and alignment module is used to map the point cloud network vehicle targets corrected by the filter correction module to the corresponding image frames one by one, and calculate the mapping point coordinates and image coordinates of the center point of each point cloud network vehicle target in the image. The distance difference between the coordinates of the center point of any connected vehicle target. The one with the smallest distance difference and less than the threshold is its corresponding matching target in the image. In this way, the mapping, matching and alignment of all point clouds and connected vehicle targets in the image is completed. ; The perceptual information fusion module is used to fuse the perceptual information of the connected vehicle target in the point cloud and image after matching and alignment by the matching alignment module, so as to obtain the license plate number and instantaneous speed of the connected vehicle. If the instantaneous speed exceeds the maximum speed limit, The license plate number information of the connected vehicle is reported to the connected vehicle cloud control platform, and an overspeed warning is issued, and the connected vehicle is remotely controlled to slow down to a safe speed.

Images