CN114022563B - A dynamic obstacle detection method for autonomous driving - Google Patents

A dynamic obstacle detection method for autonomous driving Download PDF

Info

Publication number
CN114022563B
CN114022563B CN202111238591.2A CN202111238591A CN114022563B CN 114022563 B CN114022563 B CN 114022563B CN 202111238591 A CN202111238591 A CN 202111238591A CN 114022563 B CN114022563 B CN 114022563B
Authority
CN
China
Prior art keywords
point
dynamic
points
feature
static
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202111238591.2A
Other languages
Chinese (zh)
Other versions
CN114022563A (en
Inventor
田炜
文永琨
邓振文
黄禹尧
谭大艺
韩帅
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tongji University
Original Assignee
Tongji University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tongji University filed Critical Tongji University
Priority to CN202111238591.2A priority Critical patent/CN114022563B/en
Publication of CN114022563A publication Critical patent/CN114022563A/en
Application granted granted Critical
Publication of CN114022563B publication Critical patent/CN114022563B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/08—Projecting images onto non-planar surfaces, e.g. geodetic screens
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20092—Interactive image processing based on input by user
    • G06T2207/20104—Interactive definition of region of interest [ROI]

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Molecular Biology (AREA)
  • Artificial Intelligence (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Health & Medical Sciences (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Image Analysis (AREA)
  • Traffic Control Systems (AREA)
  • Image Processing (AREA)

Abstract

本发明提供一种用于自动驾驶的动态障碍物检测方法，包括以下步骤：通过车载传感器设备获取车辆运行过程中的环境图像流、点云数据和定位数据；将所述环境图像流分成连续的帧图像，对每一对连续的帧图像依次进行2D特征点的检测和匹配、2D特征点的3D位置的还原和恢复、3D特征点的动静态点的甄别，以及构建动静二值标签；将所有帧图像的动静二值标签作为神经网络的输入，训练神经网络；实时采集车辆运行过程中的环境图像流，以点云数据和定位数据，并输入训练好的神经网络，获取动静二值分割图像，进而实现动态障碍物的检测。与现有技术相比，该方法能够通过对动态目标和静态目标进行分割，进而实现动态障碍物检测，并提高障碍物检测的准确性。

The present invention provides a dynamic obstacle detection method for automatic driving, comprising the following steps: obtaining an environmental image stream, point cloud data and positioning data during vehicle operation through an on-board sensor device; dividing the environmental image stream into continuous frame images, sequentially performing 2D feature point detection and matching, restoration and recovery of the 3D position of the 2D feature points, identification of dynamic and static points of the 3D feature points, and construction of dynamic and static binary labels for each pair of continuous frame images; using the dynamic and static binary labels of all frame images as inputs of a neural network to train the neural network; collecting the environmental image stream during vehicle operation in real time, using point cloud data and positioning data, and inputting the trained neural network to obtain dynamic and static binary segmentation images, thereby realizing dynamic obstacle detection. Compared with the prior art, the method can realize dynamic obstacle detection by segmenting dynamic targets and static targets, and improve the accuracy of obstacle detection.

Description

Dynamic obstacle detection method for automatic driving
Technical Field
The invention relates to the field of intelligent network automobiles and the field of machine learning, in particular to a dynamic obstacle detection method for automatic driving.
Background
Autopilot is a technology that controls a vehicle to automatically travel on a road through computer equipment, and the implementation of autopilot relies on cooperation of artificial intelligence, visual computing, radar, and positioning components. Because the actual road condition is complex, not only static obstacles such as trees, green belts and the like are present, but also dynamic obstacles such as a large number of pedestrians, vehicles and the like are present, so that how to realize dynamic obstacle recognition, and further planning a driving route for avoiding the obstacles becomes a key of automatic driving.
However, in the prior art, detection of dynamic obstacles is still blank, most of processing methods depend on labels, but the labeling work is very tedious and time-consuming, and the cost is high, so that the development of the field of obstacle detection is greatly limited.
Currently, a laser radar or a camera is generally provided on a vehicle supporting automatic driving. When the laser radar is used for detecting the obstacle, the laser radar emits laser beams, and a laser point cloud is constructed according to reflected beams formed by the laser beams on the surface of the object, so that the obstacle in the environment is identified according to the laser point cloud; when the camera is used for detecting the obstacle, the camera acquires an environment image and recognizes the obstacle in the environment image through an image recognition technology. But is limited by the working principles of the laser radar and the camera, the error of obstacle distance detection is larger during recognition, and the accuracy of obstacle detection is affected.
Disclosure of Invention
The invention aims to overcome the defects of the prior art and provide a dynamic obstacle detection method for automatic driving, which can realize dynamic obstacle detection by dividing a dynamic target and a static target and improve the accuracy of obstacle detection.
The aim of the invention can be achieved by the following technical scheme:
the invention provides a dynamic obstacle detection method for automatic driving, which comprises the following steps:
s1, acquiring an environment image stream, point cloud data and positioning data corresponding to the environment image stream in the running process of a vehicle through vehicle-mounted sensor equipment;
S2, dividing the environment image stream into continuous frame images, sequentially detecting and matching 2D characteristic points, restoring and recovering 3D positions of the 2D characteristic points, discriminating dynamic and static points of the 3D characteristic points and constructing dynamic and static binary labels for each frame image according to point cloud data and positioning data;
s3, taking dynamic and static binary labels constructed by all continuous frame images as the input of a neural network, and training the neural network by combining a cross entropy loss function;
And S4, acquiring an environment image stream, point cloud data and positioning data corresponding to the environment image stream in the running process of the vehicle in real time, inputting a trained neural network, acquiring a dynamic and static binary segmentation image, and further realizing the detection of dynamic obstacles.
Preferably, the vehicle-mounted sensor comprises a camera for acquiring an ambient image stream, a lidar for acquiring point cloud data, and an RTK positioning device for acquiring positioning data.
Preferably, the step S2 includes the steps of:
s2.1, detecting and matching 2D feature points of each pair of continuous frame images, and acquiring a 2D feature matching point set;
S2.2, according to the point cloud data, restoring and recovering the 3D position of each feature point of each pair of matching points in the 2D feature matching point set, and further obtaining a 3D feature matching point set;
s2.3, establishing a motion constraint condition equation according to the 3D feature matching point set and the positioning data, and screening dynamic and static points of each 3D feature point in the 3D feature point set according to the residual value;
And S2.4, constructing a dynamic and static binary label according to the discrimination result of the dynamic and static points.
Preferably, the step of restoring and recovering the 3D position of each feature point in S2.2 includes the following steps:
s2.2.1 projecting the laser radar 3D point onto the frame image according to the point cloud data;
s2.2.2, defining a region of interest on the frame image by taking each characteristic point as a center, and collecting laser radar 3D points projected into the region of interest;
S2.2.3, judging whether the number of the laser radar 3D points in the region of interest is smaller than a specified threshold, if so, directly discarding the region, otherwise, carrying out S2.2.4;
s2.2.4 fitting a plane according to a laser radar 3D point in the region of interest by adopting a least square method, evaluating the quality of the plane according to the average value of plane residual errors, and reserving a plane with good quality for S2.2.5;
s2.2.5 recovering the 3D position of the point according to the intersection equation of the ray and the local plane.
Preferably, in S2.2, if the 3D position recovery of any one of the pair of matching points fails, the pair of matching points is not reserved.
Preferably, the formula for projecting the lidar 3D point onto the frame image is:
Wherein: For the laser to camera extrinsic matrix, Is an extrinsic rotation matrix of the camera,The camera is an external reference translation matrix, K is a camera internal reference matrix, P j∈pcl0/1,pcl0 and pcl 1 are both point cloud data, P i is 2D homogeneous coordinates of a laser radar 3D point projected onto an environment image, and D is Z coordinates of the laser radar 3D point under a camera coordinate system.
Preferably, the set of lidar 3D points projected into the region of interest is:
in the formula, AndRespectively as characteristic pointsP x and p y are the abscissa and ordinate, respectively, of point p, which is the point in the region of interest, w x and w y are the lateral and longitudinal dimensions, respectively, of the ROI region, Ω roi is the set of all points in the region of interest, Ω pcl is the set of all laser points projected into the region of interest.
Preferably, the fitted plane equation is specifically:
in the formula, Is an extrinsic rotation matrix of the camera,Is an external parameter translation matrix of the camera, n is a normal vector, and P is a three-dimensional coordinate matrix of the laser radar 3D point.
Preferably, the motion constraint equation is specifically:
in the formula, AndThe j pairs of matching points of the k frame and the k+1 frame images respectively, wherein Ω 3D is a 3D feature matching point set, res j is a residual, if Res j is smaller than a threshold value, the static feature points are considered, and otherwise, the dynamic feature points are considered.
Preferably, the step S2.4 includes the steps of:
S2.4.1 traversing all dynamic feature points and voting on pixel points in the interested area;
s2.4.2 traversing all static feature points, and setting 0 for the number of tickets of pixel points in the interested area;
s2.4.3, counting the total ticket number of each pixel point in each frame of image, judging whether the total ticket number is larger than a set threshold value, if so, considering the pixel point as a dynamic point, otherwise, considering the pixel point as a static point;
s2.4.4, binarizing the total number of tickets of each pixel point to construct a dynamic and static binarization label.
Compared with the prior art, the invention has the following advantages:
1. according to the invention, plane fitting is performed by adopting a least square method, and the intersection point is calculated by using the rays of the 2D points and the plane to finish estimation, so that 3D position recovery is accurately performed on the detected and matched 2D characteristic points, and a foundation is laid for accurately discriminating dynamic and static points.
2. According to the invention, dynamic points and static points of the recovered 3D characteristic points are discriminated, binary labels are constructed to perform neural network training, and the dynamic and static objects can be segmented more efficiently without depending on labeling data, so that dynamic obstacle detection is realized.
Drawings
Fig. 1 is a flow chart of a dynamic obstacle detection method for automatic driving according to the present embodiment;
FIG. 2 is a schematic diagram of 3D restoration of 2D feature points in the embodiment shown in FIG. 1;
Fig. 3 is a schematic flow chart of S2.2 in the embodiment shown in fig. 1.
Detailed Description
The invention will now be described in detail with reference to the drawings and specific examples.
Referring to fig. 1, the present invention provides a dynamic obstacle detection method for automatic driving, comprising the steps of:
s1, acquiring an environment image stream in the running process of a vehicle and short-time accurate point cloud data and positioning data corresponding to the environment image stream through vehicle-mounted sensor equipment;
The vehicle-mounted sensor device comprises a camera for acquiring an environment image, a laser radar for acquiring point cloud data and an RTK positioning device for acquiring positioning data.
S2, dividing the environment image stream into continuous frame images, sequentially detecting and matching 2D characteristic points, restoring and recovering 3D positions of the 2D characteristic points, discriminating dynamic and static points of the 3D characteristic points and constructing dynamic and static binary labels for each frame image according to point cloud data and positioning data;
s2.1, detecting and matching 2D feature points of each pair of continuous frame images, and acquiring a 2D feature matching point set;
Taking a pair of continuous frame images I 0,I1 as an example, detecting and matching 2D feature points, and acquiring a 2D feature point matching point set omega 2D;
Where nums i is the number of matching points for the I-th pair of previous frame image I 0 and the subsequent frame image I 1, Ω 2D is the 2D feature point matching point set, AndThe j-th pair of matching points of the frame image I 0 and the frame image I 1 are respectively, and j is more than or equal to 0 and less than or equal to nums i.
As an alternative embodiment, the feature point detection method Superpoint based on deep learning is used to perform feature point detection on the 2D feature point.
As an alternative embodiment, the RGB three-channel image data is normalized, i.e. each pixel value of the frame image divided by 255, before the detection and matching of the 2D feature points is performed.
S2.2, according to the point cloud data, restoring and recovering the 3D position of each feature point of each pair of matching points in the 2D feature matching point set, and further obtaining a 3D feature matching point set;
s2.2.1, according to the point cloud data, projecting the 3D point of the laser radar onto a frame image, wherein the projection formula is as follows:
Wherein: For the laser to camera extrinsic matrix, Is an extrinsic rotation matrix of the camera,The camera is an external reference translation matrix, K is a camera internal reference matrix, P j∈pcl0/1,pcl0 and pcl 1 are both point cloud data, P j is 2D homogeneous coordinates of a laser radar 3D point projected onto an environment image, and D is Z coordinates of the laser radar 3D point under a camera coordinate system. The projected lidar 3D points are shown as gray points inside the circle in fig. 2.
S2.2.2, defining a region of interest on the frame image by taking each characteristic point as a center, and collecting laser radar 3D points projected into the region of interest;
A certain pair of matching points for the kth frame and the (k+1) th frame Their 3D positions are restored separately, and if any one of the 3D positions fails to be restored, the pair of matching points is not reserved.
By characteristic pointsTaking the recovery procedure of (a) as an exampleFor the center, a region of interest (ROI) is defined on the frame image, resulting in a set:
in the formula, AndRespectively isP x and p y are the horizontal and vertical coordinates, respectively, of a point p on the image, point p being a point within the region of interest, w x and w y being the horizontal and vertical dimensions, respectively, of the ROI region, Ω roi being the set of all points within the region of interest.
Collecting laser 3D points projected into the region of interest:
Where Ω pcl is the set of all lidar 3D points projected into the region of interest.
S2.2.3, judging whether the number of the laser radar 3D points in the region of interest is smaller than a specified threshold, if so, directly discarding the region, otherwise, carrying out S2.2.4;
A first threshold thre n is set, and if the number of laser points in the region of interest is less than thre n, i.e., |Ω pcl|＜thren, the region is directly discarded.
The first threshold thre n is a super parameter, and related to the internal parameters of the camera, the manual adaptation needs to be performed according to different sensors. As an alternative embodiment, the first threshold thre n is set to 20 pixels.
S2.2.4, fitting a plane by adopting a least square method, evaluating the quality of the plane according to the average value of plane residual errors, and reserving the plane with good quality for S2.2.5;
If the number of the 3D points of the laser radar in the region of interest is greater than or equal to thre n, a plane is fitted by a least square method using the points in the region of interest, as shown in the lower right corner of fig. 2. The fitted plane equation is:
Wherein P is a three-dimensional coordinate matrix of the laser radar 3D point, wherein an ith column P col[i] in the matrix is a three-dimensional coordinate of the ith laser 3D point in the region of interest, namely P col[i]∈Ωpcl, and n is a normal vector of a plane.
The quality of the plane can be evaluated by calculating the plane residual:
in the formula, And r is a plane residual error which is an estimated value of the normal vector n, the average value of the normal vector n is used as an evaluation index, and if the quality is too poor, namely the average value of the normal vector n exceeds a second threshold value, the plane is directly abandoned.
The second threshold is set according to different sensors, as an alternative embodiment the second threshold is 0.05 meters in size.
S2.2.5 recovering the 3D position of the point according to the intersection equation of the ray and the local plane.
According to the plane equation andThe 3D position of the recovery point, as shown in fig. 2, is available from the ray intersecting the local plane:
Wherein, the pill is a coefficient for solving, and the 3D position of the intersection point is
Thus, the 2D feature point matching point set omega 2D is converted into the 3D feature point matching point set omega 3D
S2.3, establishing a motion constraint condition equation according to the 3D feature matching point set and the positioning data, and screening dynamic and static points of each 3D feature point in the 3D feature point set according to the residual value;
In training the segmentation model, continuous image streams are required, which correspond to short-time accurate positioning data and point cloud data. And for images of the front frame and the rear frame, acquiring a 3D characteristic point matching point set omega 3D by adopting the method described in S2. Due to the existence of short-time accurate positioning data, a position transformation matrix of each frame is obtained While matching points on the static object conform to the motion constraint that is used to construct the moving object tag.
Specifically, for a pair of 3D feature matching points
In the formula,AndThe j-th pair of matching points of the k-th frame and the k+1-th frame images respectively.
According to the motion constraint, there is the following relationship:
in the formula, AndThe position transformation matrices of the k-th frame and the k+1-th frame images, respectively.
However, due to errors in the actual motion, residuals will appear at both ends of the above formula, namely:
Where Res j is a residual, if Res j is smaller than the third threshold, that is, res j＜thresholddynamic, then the point is considered to be a static feature point and is classified as set D s, otherwise, it is considered to be a dynamic feature point and is classified as set D d.
The third threshold is a manually adjustable parameter according to the sensor, as an alternative embodiment, set to 0.07 meters.
And S2.4, constructing a dynamic and static binary label according to the discrimination result of the dynamic and static points.
S2.4.1 traversing all dynamic feature points and voting on pixel points in the interested area;
s2.4.2 traversing all the static feature points and setting 0 for the number of tickets of the pixel points in the region of interest, wherein the purpose of traversing all the static points is to suppress noise.
S2.4.3, counting the total ticket number of each pixel point in each frame of image, judging whether the total ticket number is larger than a fourth threshold value, if so, considering the pixel point as a dynamic point, otherwise, considering the pixel point as a static point;
as an alternative embodiment, the fourth threshold is set to 5.
S2.4.4, binarizing the total number of tickets of each pixel point to construct a dynamic and static binarization label.
The pseudo code for this process is shown below, where (h, w) is the height and width of the image, and p [0] and p [1] are the x and y coordinates of the p-point, respectively:
Vote=np.zeros(h,w)
for p in Ds:
Vote[max(0,p[0]-ws):min(w,p[0]+ws),max(0,p[1]-hs):min(h,p[1]+hs)]+=1
for p in Dd:
Vote[max(0,p[0]-wS):min(w,p[0]+ws),max(0,p[1]-hs):min(h,p[1]+hs)]=0
Vote[Vote>thre]=1
Vote[Vote<=thre]=0
s3, taking dynamic and static binary labels constructed by all continuous frame images as the input of a neural network, and training the neural network by combining a cross entropy loss function;
as an alternative embodiment, the neural network is selected from FCN network or Unet network.
The loss function (loss) of the network is set to cross entropy loss as follows:
loss=CrossEntropy(Vote,output)
where output is the output of the network and is a tensor with channel 2. By the above way, a dynamic object segmentation module can be trained unsupervised. Meanwhile, the unsupervised training mode is also suitable for the lightweight network.
S4, inputting real-time collected continuous image data, point cloud data and positioning data, and dividing a dynamic target and a static target by adopting a trained neural network to obtain a dynamic and static binary divided image so as to realize dynamic obstacle detection.
According to the method for detecting the dynamic obstacle for automatic driving, provided by the invention, plane fitting is carried out by adopting a least square method, and the intersection point is obtained by using the ray of the 2D point and the plane to finish estimation, so that 3D position recovery is accurately carried out on the detected and matched 2D characteristic point, and a foundation is laid for accurately discriminating the dynamic and static point. And then, the dynamic points and the static points of the recovered 3D characteristic points are discriminated, binary labels are constructed for neural network training, and the dynamic and static objects can be segmented more efficiently without depending on labeling data, so that the detection of dynamic obstacles is realized.
The previous description of the embodiments is provided to facilitate a person of ordinary skill in the art in order to make and use the present invention. It will be apparent to those skilled in the art that various modifications can be readily made to these embodiments and the generic principles described herein may be applied to other embodiments without the use of the inventive faculty. Therefore, the present invention is not limited to the above-described embodiments, and those skilled in the art, based on the present disclosure, should make improvements and modifications without departing from the scope of the present invention.

Claims (7)

1.一种用于自动驾驶的动态障碍物检测方法，其特征在于，包括以下步骤：1. A dynamic obstacle detection method for autonomous driving, comprising the following steps: S1：通过车载传感器设备获取车辆运行过程中的环境图像流，以及环境图像流对应的点云数据和定位数据；S1: Obtain the environment image stream during vehicle operation, as well as the point cloud data and positioning data corresponding to the environment image stream through the vehicle-mounted sensor device; S2：将所述环境图像流分成连续的帧图像，并根据点云数据和定位数据，对每一对连续的帧图像依次进行2D特征点的检测和匹配、2D特征点的3D位置的还原和恢复、3D特征点的动静态点的甄别，以及为每一帧图像构建动静二值标签；S2: Divide the environment image stream into continuous frame images, and perform 2D feature point detection and matching, 3D position restoration and recovery, dynamic and static point identification of 3D feature points, and dynamic and static binary labels for each frame image in sequence according to the point cloud data and positioning data; S3：将所有连续的帧图像所构建的动静二值标签作为神经网络的输入，并结合交叉熵损失函数训练神经网络；S3: The binary labels of motion and stillness constructed from all consecutive frame images are used as the input of the neural network, and the neural network is trained in combination with the cross entropy loss function; S4：实时采集车辆运行过程中的环境图像流，以及环境图像流对应的点云数据和定位数据，并输入训练好的神经网络，获取动静二值分割图像，进而实现动态障碍物的检测；S4: Real-time collection of environmental image streams during vehicle operation, as well as point cloud data and positioning data corresponding to the environmental image streams, and input into the trained neural network to obtain dynamic and static binary segmentation images, thereby realizing the detection of dynamic obstacles; 所述S2包括以下步骤：The S2 comprises the following steps: S2.1：对每一对连续的帧图像进行2D特征点的检测与匹配，并获取2D特征匹配点集；S2.1: Detect and match 2D feature points for each pair of continuous frame images, and obtain a 2D feature matching point set; S2.2：根据点云数据，对所述2D特征匹配点集中的每一对匹配点的每一特征点进行3D位置的还原和恢复，进而获取3D特征匹配点集；S2.2: According to the point cloud data, restore and recover the 3D position of each feature point of each pair of matching points in the 2D feature matching point set, thereby obtaining a 3D feature matching point set; S2.3：根据所述3D特征匹配点集和定位数据，建立运动约束条件方程，并根据残差值的大小对3D特征点集中的每一个3D特征点进行动静态点的甄别；S2.3: establishing a motion constraint condition equation according to the 3D feature matching point set and the positioning data, and identifying a dynamic or static point for each 3D feature point in the 3D feature point set according to the size of the residual value; S2.4：根据动静态点的甄别结果，构建动静二值标签；S2.4: Construct dynamic and static binary labels based on the identification results of dynamic and static points; 所述S2.2中对每一特征点进行3D位置的还原和恢复，包括以下步骤：The restoration and recovery of the 3D position of each feature point in S2.2 includes the following steps: S2.2.1：根据点云数据，将激光雷达3D点投影到帧图像上；S2.2.1: Project the LiDAR 3D points onto the frame image based on the point cloud data; S2.2.2：以每一特征点为中心，在帧图像上划定一个感兴趣区域，并收集投影到感兴趣区域内的激光雷达3D点；S2.2.2: Define a region of interest on the frame image with each feature point as the center, and collect the lidar 3D points projected into the region of interest; S2.2.3：判断感兴趣区域内的激光雷达3D点的数量是否小于指定阈值，若小于，则直接抛弃该区域，否则进行S2.2.4；S2.2.3: Determine whether the number of LiDAR 3D points in the region of interest is less than a specified threshold. If so, the region is discarded directly, otherwise proceed to S2.2.4; S2.2.4：根据感兴趣区域内的激光雷达3D点并采用最小二乘法拟合平面，根据平面残差的均值评估该平面的质量，并保留质量好的平面进行S2.2.5；S2.2.4: Fit a plane using the least squares method based on the lidar 3D points in the region of interest, evaluate the quality of the plane based on the mean of the plane residuals, and retain the plane with good quality for S2.2.5; S2.2.5：根据射线与局部平面相交方程恢复点的3D位置；S2.2.5: Recover the 3D position of the point based on the equation of intersection of the ray and the local plane; 所述S2.4包括以下步骤：The S2.4 comprises the following steps: S2.4.1：遍历所有的动态特征点，并对其感兴趣区域中的像素点进行投票；S2.4.1: Traverse all dynamic feature points and vote for the pixels in their area of interest; S2.4.2：遍历所有的静态特征点，并对其感兴趣区域中的像素点的票数置0；S2.4.2: Traverse all static feature points and set the votes of the pixels in their area of interest to 0; S2.4.3：统计每一帧图像中每一像素点的总票数，并判断总票数的大小是否大于设定阈值，若是，则认为该像素点为动态点，否则认为该像素点为静态点；S2.4.3: Count the total number of votes for each pixel in each frame of the image, and determine whether the total number of votes is greater than a set threshold. If so, the pixel is considered to be a dynamic point, otherwise, the pixel is considered to be a static point; S2.4.4：将每一像素点的总票数进行二值化，构建动静二值化标签。S2.4.4: Binarize the total number of votes for each pixel and construct a dynamic or static binary label. 2.根据权利要求1所述的一种用于自动驾驶的动态障碍物检测方法，其特征在于，所述车载传感器包括用于采集环境图像流的相机、用于采集点云数据的激光雷达和用于采集定位数据的RTK定位设备。2. According to a dynamic obstacle detection method for autonomous driving according to claim 1, it is characterized in that the vehicle-mounted sensor includes a camera for collecting environmental image streams, a laser radar for collecting point cloud data, and an RTK positioning device for collecting positioning data. 3.根据权利要求1所述的一种用于自动驾驶的动态障碍物检测方法，其特征在于，所述S2.2中，若一对匹配点中的任意一个特征点的3D位置恢复失败，则不保留这对匹配点。3. A dynamic obstacle detection method for autonomous driving according to claim 1, characterized in that, in S2.2, if the 3D position recovery of any one feature point in a pair of matching points fails, the pair of matching points is not retained. 4.根据权利要求1所述的一种用于自动驾驶的动态障碍物检测方法，其特征在于，所述将激光雷达3D点投影到帧图像上的公式为：4. A dynamic obstacle detection method for autonomous driving according to claim 1, characterized in that the formula for projecting the laser radar 3D point onto the frame image is: 式中：为激光到相机的外参矩阵，为相机的外参旋转矩阵，为相机的外参平移矩阵，K为相机内参矩阵，Pj∈pcl0/1，pcl0和pcl1均为点云数据，pj为激光雷达3D点投影至环境图像上的2D齐次坐标，d为激光雷达3D点在相机坐标系下的Z坐标。Where: is the external parameter matrix from laser to camera, is the camera's extrinsic rotation matrix, is the camera’s extrinsic translation matrix, K is the camera’s intrinsic matrix, Pj∈pcl0 /1 , pcl0 and pcl1 are both point cloud data, pj is the 2D homogeneous coordinate of the lidar 3D point projected onto the environment image, and d is the Z coordinate of the lidar 3D point in the camera coordinate system. 5.根据权利要求4所述的一种用于自动驾驶的动态障碍物检测方法，其特征在于，投影到感兴趣区域内的激光雷达3D点的集合为：5. A dynamic obstacle detection method for autonomous driving according to claim 4, characterized in that the set of laser radar 3D points projected into the region of interest is: 式中，和分别为特征点的横、纵坐标，px和py分别为点p的横、纵坐标，点p为感兴趣区域内的点，wx和wy分别为ROI区域的横向尺寸和纵向尺寸，Ωroi为感兴趣区域内所有点的集合，Ωpcl为投影至感兴趣区域内所有激光点的集合。In the formula, and The feature points , px and py are the horizontal and vertical coordinates of point p, respectively. Point p is a point in the region of interest. wx and wy are the horizontal and vertical sizes of the ROI area, respectively. Ωroi is the set of all points in the region of interest. Ωpcl is the set of all laser points projected into the region of interest. 6.根据权利要求5所述的一种用于自动驾驶的动态障碍物检测方法，其特征在于，拟合的平面方程具体为：6. A dynamic obstacle detection method for autonomous driving according to claim 5, characterized in that the fitted plane equation is specifically: 式中，为相机的外参旋转矩阵，为相机的外参平移矩阵，n为法向量，P为激光雷达3D点的三维坐标矩阵。In the formula, is the camera's extrinsic rotation matrix, is the camera's extrinsic translation matrix, n is the normal vector, and P is the three-dimensional coordinate matrix of the lidar 3D point. 7.根据权利要求1所述的一种用于自动驾驶的动态障碍物检测方法，其特征在于，所述运动约束条件方程具体为：7. The dynamic obstacle detection method for autonomous driving according to claim 1, wherein the motion constraint condition equation is: 式中，和分别为第k帧和第k+1帧图像的第j对配点，Ω3D为3D特征匹配点集，Resj为残差，若Resj小于阈值，则认为是静态特征点，否则认为是动态特征点。In the formula, and are the j-th pair of matching points of the k-th frame and the k+1-th frame image respectively, Ω 3D is the 3D feature matching point set, Res j is the residual, if Res j is less than the threshold, it is considered to be a static feature point, otherwise it is considered to be a dynamic feature point.
CN202111238591.2A 2021-10-25 2021-10-25 A dynamic obstacle detection method for autonomous driving Active CN114022563B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111238591.2A CN114022563B (en) 2021-10-25 2021-10-25 A dynamic obstacle detection method for autonomous driving

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111238591.2A CN114022563B (en) 2021-10-25 2021-10-25 A dynamic obstacle detection method for autonomous driving

Publications (2)

Publication Number Publication Date
CN114022563A CN114022563A (en) 2022-02-08
CN114022563B true CN114022563B (en) 2025-06-03

Family

ID=80057231

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111238591.2A Active CN114022563B (en) 2021-10-25 2021-10-25 A dynamic obstacle detection method for autonomous driving

Country Status (1)

Country Link
CN (1) CN114022563B (en)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115331199A (en) * 2022-08-10 2022-11-11 阿波罗智联(北京)科技有限公司 Obstacle detection method, obstacle detection device, electronic device, and storage medium
CN116824544B (en) * 2023-06-21 2026-01-30 重庆长安汽车股份有限公司 A method, device, and vehicle-mounted terminal for determining dynamic obstacles based on point clouds.
CN116883496B (en) * 2023-06-26 2024-03-12 小米汽车科技有限公司 Coordinate reconstruction method, device, electronic equipment and storage medium of traffic elements
CN117115426A (en) * 2023-08-23 2023-11-24 合众新能源汽车股份有限公司 3D-based obstacle labeling methods, devices and electronic equipment

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110097047B (en) * 2019-03-19 2021-10-08 同济大学 A vehicle detection method based on deep learning using single-line lidar
CN111274976B (en) * 2020-01-22 2020-09-18 清华大学 Lane detection method and system based on multi-level fusion of vision and laser radar
CN112037280A (en) * 2020-08-17 2020-12-04 北京声智科技有限公司 Object distance measuring method and device
CN112085801B (en) * 2020-09-08 2024-03-19 清华大学苏州汽车研究院（吴江） Calibration method for 3D point cloud and 2D image fusion based on neural network

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
"Mapping with Monocular Camera Sensor under Adversarial Illumination for Intelligent Vehicles";Wei Tian;《Sensors》;20230321;第23卷(第6期);第3296页 *

Also Published As

Publication number Publication date
CN114022563A (en) 2022-02-08

Similar Documents

Publication Publication Date Title
CN114022563B (en) A dynamic obstacle detection method for autonomous driving
CN111448478B (en) System and method for correcting high-definition maps based on obstacle detection
CN114359181B (en) Intelligent traffic target fusion detection method and system based on image and point cloud
Bar Hillel et al. Recent progress in road and lane detection: a survey
CN106919915B (en) Map road marking and road quality acquisition device and method based on ADAS system
US10290219B2 (en) Machine vision-based method and system for aircraft docking guidance and aircraft type identification
CN115717894A (en) A high-precision vehicle positioning method based on GPS and common navigation maps
Shi et al. Fast and robust vanishing point detection for unstructured road following
CN109460709A (en) The method of RTG dysopia analyte detection based on the fusion of RGB and D information
CN107992819B (en) Method and device for determining vehicle attribute structural features
CN107491720A (en) A kind of model recognizing method based on modified convolutional neural networks
CN114155511B (en) A method for collecting environmental information for autonomous vehicles on public roads
CN102629326A (en) Lane line detection method based on monocular vision
CN114299247B (en) Road traffic signs and markings rapid detection and troubleshooting methods
CN105404857A (en) Infrared-based night intelligent vehicle front pedestrian detection method
CN112598743B (en) Pose estimation method and related device for monocular vision image
CN119229221B (en) A container damage information detection and counting method based on yolov9 and DeepSort
CN108520528A (en) Based on the mobile vehicle tracking for improving differential threshold and displacement field match model
Mouzinho et al. Hierarchical semantic segmentation based approach for road surface damages and markings detection on paved road
CN111950478B (en) Method for detecting S-shaped driving behavior of automobile in weighing area of dynamic flat-plate scale
Liu et al. Research on security of key algorithms in intelligent driving system
WO2025124953A1 (en) Method of and device for identifying road distress on a road based on imagery of the road
Wu et al. Enhanced roadway geometry data collection using an effective video log image-processing algorithm
CN105447496A (en) Docking airplane model identification and verification method and system based on machine vision
Malik High-quality vehicle trajectory generation from video data based on vehicle detection and description

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant