CN111145545B - UAV monitoring system and method for road traffic behavior based on deep learning - Google Patents

Abstract

The invention discloses a road traffic behavior unmanned aerial vehicle monitoring system and method based on deep learning, wherein the system comprises an acquisition module, a single-camera processing module, a cross-camera matching module and a traffic parameter extraction module; the single-camera processing module creates a single-camera processing submodule for each path of video output by the acquisition module to perform data processing; the single-camera processing submodule comprises an image preprocessing module, a vehicle detection module and a vehicle tracking module. The method uses an unmanned aerial vehicle to shoot road traffic conditions and analyzes the traffic state of vehicles in a monitoring area, and comprises the steps of calibrating a monitoring picture, a video-based vehicle detection and tracking technology, a camera-crossing multi-target track matching method based on geographic positions, an algorithm for analyzing the traffic state according to vehicle motion tracks and the like.

Description

UAV monitoring system and method for road traffic behavior based on deep learning

technical field

The invention belongs to the field of intelligent transportation, and in particular relates to a road traffic behavior unmanned aerial vehicle monitoring system and method based on deep learning.

Background technique

Intelligent Transportation System (ITS) is the development direction of future transportation system, and its purpose is to improve the safety and efficiency of transportation. The system adopts various relevant advanced science and technology, integrates the people, vehicles, roads and environment involved in the traffic system to make it play an intelligent role, so that the traffic system can achieve a safe, smooth, low pollution and low energy consumption. Target. The real-time detection and tracking of moving vehicles is one of the core parts of the intelligent transportation system. In recent years, with the rapid development of computer hardware technology and digital image processing technology, although the detection and tracking technology for moving vehicles has also been continuously improved, the direct use of ITS technology to solve traffic problems still faces huge challenges. The detection and tracking of moving vehicles is to use computer vision technology to obtain sequence images of vehicles, and then detect vehicle motion information, such as vehicle type, vehicle speed, traffic flow, and lane occupancy. According to these data, macro-control of traffic is carried out, and then traffic congestion is reduced, so as to improve the traffic environment, increase the utilization rate of roads, and reduce the incidence of traffic accidents. In addition, the correct and real-time recording of the driving conditions of vehicles on traffic roads, such as illegal overtaking, vehicle collisions, etc., can quickly deal with traffic accidents that occur, thereby greatly improving the efficiency of troubleshooting, making the road fast and smooth, and for Handling traffic incidents improves robust and reliable evidence.

At present, there are still many problems to be solved in the detection and tracking of vehicles in the intelligent transportation system:

First, the detection of moving vehicles in dynamic scenes makes the extraction of vehicles more difficult due to the existence of two independent movements of the vehicle and the background. The accuracy of the current mainstream target detection technology needs to be improved in this task;

Second, the staggering and occlusion between vehicles in the actual traffic environment are too frequent, the deformation of vehicles under different viewing angles and changes in camera viewing angles bring huge challenges to the vehicle retrieval and tracking technology under multi-cameras;

Third, for the current common roadside image collection, the effective distance of a single camera is limited and cannot cover the entire road section, and the information collected by multiple cameras is also difficult to correlate with each other. It is very difficult to obtain long-distance and accurate vehicle trajectory information.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a road traffic behavior UAV monitoring system and method based on deep learning, using UAV to shoot road traffic conditions, analyze the traffic conditions of vehicles in the monitoring area, including calibration of monitoring images, video-based Vehicle detection and tracking technology based on geographic location, cross-camera multi-target trajectory matching method, and algorithm for analyzing traffic status based on vehicle motion trajectory, etc.

The present invention is achieved through the following technical solutions:

The UAV monitoring system for road traffic behavior based on deep learning includes an acquisition module, a single-camera processing module, a cross-camera matching module and a traffic parameter extraction module; among them,

The acquisition module is used for the acquisition of aerial vehicle video data, and the collected multi-channel video data is input into the single-camera processing module for single-camera multi-target tracking processing of each channel of video;

The single-camera processing module is used to perform multi-target tracking processing on the multi-channel video data output by the acquisition module, obtain the vehicle trajectory set of each channel of video and input it to the cross-camera matching module;

The cross-camera matching module is used to perform information matching on the vehicle trajectory set of each video input by the single-camera processing module, and obtain the complete trajectory of all vehicles in the global video and input it to the traffic parameter extraction module;

The traffic parameter extraction module is used to analyze the vehicle trajectory information input by the cross-camera matching module, and extract its traffic parameters, thereby obtaining the traffic state of the target vehicle.

A further improvement of the present invention is that the single-camera processing module creates a single-camera processing sub-module for each video output by the acquisition module to perform data processing;

The single-camera processing sub-module includes an image preprocessing module, a vehicle detection module and a vehicle tracking module; the image preprocessing module is used to manually preprocess the output video of the acquisition module assigned by the single-camera processing submodule to obtain image sequences, lane information, Rotation adjustment matrix sequence and homography matrix, input lane information, image sequence and rotation adjustment matrix sequence into vehicle detection module; input homography matrix into vehicle tracking module; vehicle detection module is used for image sequence output by image preprocessing module Perform target detection processing to obtain information such as the position coordinates, pixel size and vehicle type of all vehicles in the image, and adjust and supplement the detection results according to the rotation adjustment matrix sequence and lane information output by the image preprocessing module to obtain target detection. The frame set sequence is input to the vehicle tracking module; the vehicle tracking module is used to perform target tracking processing on the target detection frame set sequence output by the vehicle detection module to obtain the complete trajectory information of all targets in the video, and according to the single output of the image preprocessing module. The adaptive matrix adjusts the tracking results to obtain the target trajectory information set; the target trajectory information set obtained by each single-camera processing sub-module is integrated, and then input to the cross-camera matching module.

The road traffic behavior UAV monitoring method based on deep learning is characterized in that, the method is based on the above-mentioned deep learning-based road traffic behavior UAV monitoring system, including the following steps:

1) Use several unmanned aerial vehicles to form a swarm of unmanned aerial vehicles to collect aerial vehicle video data, and store the collected data synchronously, and input it to the single camera processing module;

2) The single-camera processing module creates sub-modules for each channel of video to perform single-camera multi-target tracking processing according to the output video of the acquisition module, obtains the vehicle track set of each channel of video, and then integrates the vehicle track sets of all videos into the cross-camera. matching module;

3) The cross-camera matching module performs pairwise matching and information fusion on the vehicle trajectory data of each video according to the integrated data output by the single-camera processing module to obtain the complete trajectory information of each vehicle in the global video, and then compares the trajectory data of all vehicles. The information is integrated into a global trajectory set and input to the traffic parameter extraction module;

4) According to the global trajectory set input by the cross-camera matching module, the traffic parameter extraction module uses the traffic parameter calculation formula to calculate the trajectory information of each vehicle in the set to obtain a single traffic parameter of the target; Calculation is performed to obtain the interactive traffic parameters of the two targets; according to the calculated traffic parameters, the traffic state is evaluated for each vehicle.

A further improvement of the present invention is that the concrete realization method of step 1) is as follows:

401) Assign numbers to all unmanned aerial vehicles, arrange them from small to large, and take off in sequence at the leftmost end of the monitoring area according to the numbering sequence;

402) For No. 1 UAV, lift off vertically, hover at an altitude of about 250 meters from the ground, and then adjust the position so that the left side of the shooting screen is the leftmost end of the monitoring area;

403) For No. 2 UAV, after hovering at an altitude of about 250 meters above the ground, adjust the position so that the left area of the shooting picture and the right area of the No. 1 drone shooting picture have a real world length of about 30 meters overlap;

404) and so on, until the shooting picture of the drone group covers the entire monitoring area, and at the same time, the video recording function of all drones is turned on, and the video data is collected and stored synchronously. After the data collection is completed, the collected data is transferred from the drone to the database. .

The further improvement of the present invention is, the concrete realization method of step 2) is as follows:

501) read the video collected by the drone numbered 1 from the database, process the video into a frame-based image sequence, and select one of the frames as the calibration image H to characterize the scene of the video; in the calibration image H Select more than 20 marked points. The marked points should be at the same level, and the entire monitoring screen should be distributed as evenly as possible, and then record the image coordinates of all marked points. Use Google Maps or GPS to collect the latitude and longitude position data of the above marked points, and convert them into Northeast sky coordinate information; arrange the image coordinates and northeast sky coordinates in corresponding order and input python's cv2.findHomography function to fit, and calculate the homography matrix K that maps the image coordinate system of the calibration frame to the world coordinate system; in the calibration image On H, for each lane area, mark several points along the edge of the lane area and save the image coordinates of these points; integrate the point set generated by each lane area into the npy file to represent the scene lane information; use ORB Algorithm to extract the position information L and feature information F of several feature points P of the calibration image H;

502) use ORB algorithm, extract the position information L _j and characteristic information F _j of several characteristic points P _j of the jth frame image H _j of this video; Calculate characteristic point P according to the characteristic information F and characteristic information F _j of 501) One-to-one correspondence with the feature point P _j , and according to the correspondence and the position information L and position information L _j of 501), use python's cv2.findHomography function to fit, and calculate the image coordinate system of the currently processed image. The rotation adjustment matrix M _j of the image coordinate system mapping of the frame; According to the above method, traverse all the frames of the video according to the frame number, and obtain the rotation adjustment matrix sequence M;

503) Use the target detector based on deep learning to perform frame-by-frame target detection on the image sequence of 501), and obtain the detection frame of all vehicles in each frame of image, that is, the image position information and the corresponding vehicle model information; According to the lane information of 501), analyze each frame. The lane area where each detection frame is located; the above information is combined to obtain a detection frame set sequence D, whose unit d _j represents the detection frame set of the jth frame picture, including the image coordinates, vehicle type and lane of each detection frame;

504) According to the detection frame set sequence D obtained in 503), for the detection frame coordinates of each d _j , use the homography matrix M _j corresponding to the jth frame in the rotation adjustment matrix sequence M of 502) to perform homography transformation. , get the transformed coordinate and update it to d _j , get d _j ′; after traversing and updating all units, the detection frame set sequence is updated to D ′;

505) According to the detection frame set sequence D′ obtained in 504), use the Sort multi-target tracking algorithm based on Kalman filtering and Hungarian matching to calculate the vehicle trajectory set T, and its unit t _i represents the trajectory information of the i-th vehicle, It includes the image coordinate sequence (x _ij , y _ij ) of the target, the vehicle type, and the lane sequence r _ij where it is located, where j is the frame number corresponding to the parameter;

并更新到t_i，得到w_i；遍历更新所有单元后，车辆轨迹集合更新为W₁；506) According to the vehicle trajectory set T obtained in 505), perform homography transformation on the coordinate sequence (x _ij , y _ij ) of each t _i using the homography matrix K of 501) to obtain the transformed coordinates

And update to t _i to get _wi ; after traversing and updating all units, the vehicle trajectory set is updated to W ₁ ;

507) Read the video collected by the drone numbered 2 from the database, repeat steps 501)-506) to obtain a vehicle trajectory set updated to W ₂ , and so on, traverse all videos to obtain a vehicle trajectory set sequence W.

A further improvement of the present invention is that the concrete realization method of step 3) is as follows:

601) Obtain the vehicle trajectory set sequence W from the single-camera processing module, wherein W _n is the vehicle trajectory information set of the n th video, and its unit w _ni is the trajectory information of the ith target of the n th video, and establishes a global vehicle Trajectory information set W _total ={}, set n=1;

602) Take W _n as the target, use the data of W _n+1 to perform matching, establish a matching set m _ni for each w _ni in W _n , and set i=1;

603) Traverse the data of W _n+1 , and classify w _n+1i into the set m _ni according to the following rules: it is the same type of vehicle; there is an overlapping valid time, and the valid time refers to the time period from the appearance of the target to the disappearance; The lanes are the same within the valid time of , and after the traversal is over, calculate the Euclidean distance between each element in m _ni and the trajectory of w _ni within the valid time of overlap, and keep the element with the smallest distance and lower than the set threshold in the set, and the rest elements Delete; if m _ni is empty, add w _ni to the global vehicle trajectory information set W _total , otherwise, update the information of w _ni to the unit corresponding to the remaining elements of m _ni in W _n+1 ;

604) make i=i+1, and then continue to loop from 603) until all elements in W _n are traversed;

605) make n=n+1, if at this moment n is equal to the total number of video group videos, then add all elements of W _n to W _total , otherwise, continue to loop from 602);

606) Sort and assign numbers to all the elements in the set W _total according to the order of appearance time. The set W _total contains the trajectory information of all targets in the global monitoring range of the video group, and summarizes the information of the same vehicle under different videos into the same one. unit.

A further improvement of the present invention is that the concrete realization method of step 4) is as follows:

701) obtain the global information set W _total from the cross-camera matching module; take any unit or specify a unit from the set, and this unit includes the position information sequence 1 and the lane information sequence r of the corresponding vehicle in the monitoring range monitoring time; use a low-pass The filtering method filters the sequence information to obtain the position information sequence l′; set the distance parameter x, for the position information sequence l′, starting from the xth node, calculate the Euclidean distance between the node and the first x-1th node, and then divide the Calculate the speed of the node target by the time difference between the two nodes; continue to traverse the sequence l' until all remaining nodes are calculated, and obtain the speed information sequence v; according to the speed information sequence v, calculate the speed change process, through Set a threshold to determine whether the target is overspeeding during driving; according to the lane information sequence r, count the number of lane changes and lane change time of the target within a unit time, and by setting the threshold, you can determine whether the target is in the process of driving. Bad lane changing behavior occurs;

计算车头时距TH，根据MOR公式

计算两车碰撞时间TTC，从而判断两车在行驶过程中是否存在碰撞危险；702) Select two targets with contextual relationship, extract the two units from the global information set W _total , and perform the data processing of 701) respectively to obtain the position information sequences l ₁ , l ₂ and speed information sequences v ₁ , 1 , v ₂ ; Calculate the Euclidean distance according to the corresponding time data of l ₁ and l ₂ , and obtain the front and rear interval between the two, that is, the vehicle following distance sequence D(t), and determine whether the latter has followed during the driving process by setting a threshold Near behavior; according to the MOR formula

Calculate the headway TH, according to the MOR formula

Calculate the collision time TTC of the two vehicles, so as to determine whether the two vehicles are in danger of collision during the driving process;

703) Select two targets in a side-by-side relationship, extract the two units from the global information set W _total , and perform the data processing of 701) respectively to obtain the two position information sequences l ₁ , l ₂ , according to l ₁ , l ₂ The Euclidean distance is calculated according to the data at the time, and the side-by-side interval sequence D(h) of the two can be obtained. By setting the threshold, it can be judged whether the two have bad parallel behavior during the driving process.

The present invention at least has the following beneficial technical effects:

The invention takes the video of the traffic section of the aerial photography of the drone as the input, takes the cross-camera multi-target tracking technology based on deep learning as the core technology, and completes the real-time detection and tracking of the moving vehicles within the monitoring range based on the task requirements of the intelligent transportation system. The traffic state of the vehicle is analyzed in real time through the obtained motion parameters. By constructing the mapping relationship between world coordinates and image coordinates, the invention applies the image-based target detection and tracking technology to the task of actual traffic parameter extraction, and provides a new and effective solution for the acquisition of target traffic parameters of an intelligent transportation system. In general, the present invention has the following advantages:

First: Through parameter adjustment and data enhancement technology, the current mainstream target detection model based on deep learning has been improved, and its detection ability of vehicles in the test environment has been greatly improved;

Second: Use the drone to hover over the shooting area and collect data from an angle perpendicular to the ground, eliminating the impact of obstacles on the road and the occlusion of the target caused by dense vehicles;

Third: Using the method of cross-camera multi-target trajectory matching based on geographic location, the area captured by each camera can be mapped to the same world coordinates, which can easily and accurately associate cross-camera data and realize the expansion of the collection range.

Further, through the cross-camera multi-target tracking technology, long-distance driving trajectory information of the target vehicle can be obtained, and accurate and abundant data can be provided for the calculation of vehicle traffic parameters.

Further, according to the vehicle traffic parameters obtained by the analysis, it is possible to monitor the running state of the vehicle in real time, give timely early warning to bad behaviors, and give timely feedback to the traffic conditions of the environment.

Description of drawings

Fig. 1 is the key technology relation diagram of the present invention.

FIG. 2 is a schematic diagram of data collection in the real environment of the present invention.

FIG. 3 is a demonstration diagram of the traffic parameter extraction result of the present invention.

Detailed ways

The present invention will be further described below with reference to the accompanying drawings and embodiments.

The deep learning-based UAV monitoring method for road traffic behavior according to the present invention, as shown in FIG. 1 , includes four modules, which are an acquisition module, a single-camera processing module, a cross-camera matching module, and a traffic parameter extraction module, respectively. The single-camera processing module is composed of multiple single-camera processing sub-modules, and each sub-module includes an image preprocessing module, a vehicle detection module, and a vehicle tracking module. The details of each module are as follows.

1. Collection module.

The collection module is used for the collection of aerial photography vehicle video data, and the collected multi-channel video data is input into the single-camera processing module for single-camera multi-target tracking processing of each channel of video. The main body of the acquisition module is a drone group composed of several drones, and the number of drones depends on the scope of the monitoring area. First, assign numbers to all drones, and arrange them from small to large. According to the numbering sequence, they will lift off at the leftmost end of the monitoring area. As shown in Figure 2, during the collection process, the drone group will hover at a distance from the ground. Data collection is carried out at a height of about 250 meters, and they are arranged side by side along the long side of the monitoring area according to the number. The collection range of each drone is about 270 meters, and there is an overlapping monitoring area of at least 30 meters between adjacent drones. The drone swarm will collect data synchronously and save the data. After the collection is completed, the drone swarm will be recovered, and the data will be saved to the database according to the drone number.

Combined with the example description, as shown in Figure 2, the target vehicle A and B appear from the collection area of the No. 1 drone, and then leave the collection area of the No. 1 drone in the overlapping area of the No. 1 and No. 2 drones. , enter the collection area of the No. 2 UAV; and so on, the two targets will appear in the collection area of each UAV in sequence, and finally leave the monitoring area, and each UAV will collect the two targets at different time periods. The motion information is waiting for subsequent modules to process.

2. Single camera processing module.

The single-camera processing module is used to perform multi-target tracking processing on the multi-channel video information output by the acquisition module, obtain the vehicle trajectory set of each channel of video and input it to the cross-camera matching module. The single-camera processing module consists of several single-camera processing sub-modules, and each sub-module corresponds to the video collected by a drone. Each single-camera processing sub-module consists of an image preprocessing module, a vehicle detection module and a vehicle tracking module. For complex video acquisition, multiple sub-modules are used for parallel processing. Each sub-module will output a set of vehicle trajectories corresponding to the video, and the output sets of all sub-modules will be integrated as the output of the single-camera processing module and input to the cross-camera matching module. The functional modules of the sub-modules are further described below.

1) Image preprocessing module.

The image preprocessing module is used to perform manual preprocessing on the output video of the acquisition module assigned by the sub-module, which is mainly composed of four preprocessing techniques: data read-in, camera calibration, lane extraction and image rotation adjustment. The function of data read-in is to process the input video captured by a single camera into a frame-based image sequence, and select one of the frames as a calibration frame to represent the scene of the video. The function of camera calibration is to obtain the mapping relationship between image coordinates and world coordinates. Using the method of transforming image coordinates and world coordinates based on homography matrix, the homography matrix between the image coordinates of the calibration frame and the real world coordinate system is obtained. to characterize the mapping relationship. The function of lane extraction is to extract the lane information in the monitoring area, and use the image-based lane area representation method to artificially plan the lane area in the calibration frame image to provide support for the subsequent extraction of vehicle lane information and lane changing behavior. The function of image rotation adjustment is to calculate the mapping relationship between the current test image and the reference image, and correct the slight rotation and offset of the image in the video at different times through the compensation method of screen rotation distortion based on feature point matching, so as to overcome the camera due to environmental factors ( (such as breeze, drift, etc.) the measurement deviation of traffic parameters caused by displacement. The specific steps are further described below.

Step 1: After the sub-module receives the video information, it processes the video into a frame-based image sequence, and selects one of the frames as the calibration image H to represent the scene of the video. Select more than 20 marking points in the calibration image H, and the marking points require the same level of height and evenly distribute the entire monitoring screen as much as possible, and then record the image coordinates of all marking points.

Step 2: Use Google Maps or GPS to collect the latitude and longitude position data of the above marked points, and convert them into the northeast sky coordinate information. Arrange the image coordinates and the northeast sky coordinates in the corresponding order, input python's cv2.findHomography function to fit, and calculate the homography matrix K that maps the image coordinate system of the calibration frame to the world coordinate system

Step 3: On the calibration image H, for each lane area, mark several points along the edge of the lane area and save the image coordinates of these points. The point set generated by each lane area is integrated into the npy file to represent the scene lane information.

Step 4: Using the ORB algorithm, extract the position information L and the feature information F of several (about 1000) feature points P of the calibration image H. Using the ORB algorithm, the position information L _j and the feature information F _j of several (about 1000) feature points P _j of the jth frame image H _j of the video are extracted. Calculate the one-to-one correspondence between the feature point P and the feature point P _j according to the feature information F and the feature information F _j , and use the cv2.findHomography function of python to fit according to the corresponding relationship and the position information L and the position information L _j , A rotation adjustment matrix M _j that maps the image coordinate system of the currently processed image to the image coordinate system of the calibration frame is calculated. According to the above method, all frames of the video are traversed according to the frame number, and the rotation adjustment matrix sequence M is obtained.

Step 5: Input the image sequence obtained in step 1, the lane information obtained in step 3 and the rotation adjustment matrix sequence M obtained in step 4 into the vehicle detection module; input the homography matrix K obtained in step 2 into the vehicle tracking module.

2) Vehicle detection module.

The function of the vehicle detection module is to perform target detection processing on the image sequence output by the image preprocessing module, obtain the position coordinates, pixel size and vehicle type of all vehicles in the image, and adjust the matrix according to the rotation output by the image preprocessing module. The sequence and lane information are used to adjust and supplement the detection results, obtain the target detection frame set sequence and input it into the vehicle tracking module. The invention adopts the yolov3 target detection model as the baseline model, and optimizes the model on the data set and training parameters. In addition to using the public vehicle dataset, the training dataset also adds collected and manually labeled images, expands the amount of image data such as large vehicles and vehicles with shadows, enriches the types of data, and enhances the recognition ability of the model. In addition, the present invention utilizes data enhancement technology, transforms data for angle, saturation, exposure and hue, etc. to achieve data set expansion, and enhances data by rotating angle, adjusting saturation, adjusting exposure and hue , and then improve the generalization ability of the vehicle detection model. Before training, using the k-means clustering algorithm, the detection frames of the training set are roughly clustered into several iconic ones, which are used as the adjustment benchmark for the anchor parameters of the detection model to optimize the regression loss of the detection model and improve the detection accuracy. During training, the learning rate is initially set to 0.01 for faster convergence in the early stages of training. When the iteration reaches 40,000 times, the learning rate is reduced to one-tenth of the original; when the iteration reaches 45,000 times, it is reduced to one-tenth of the original, and finally iterates to 85,000 times to obtain the final model. The specific steps are further described below.

Step 1: According to the image sequence output by the image preprocessing module, the target detector based on deep learning is used to perform frame-by-frame target detection, and the detection frame of all vehicles in each frame of image, that is, the image position information and the corresponding vehicle model information are obtained. At the same time, according to the lane information output by the image preprocessing module, the lane area where each detection frame is located is analyzed. Based on the above information, a detection frame set sequence D is obtained, and its unit d _j represents the detection frame set of the jth frame picture, including the image coordinates, vehicle type and lane of each detection frame.

Step 2: Based on the detection frame set sequence D obtained in Step 1, for the detection frame coordinates of each d _j , use the homography matrix M _j corresponding to the jth frame in the rotation adjustment matrix sequence M output by the image preprocessing module to perform homography. Responsive transformation to get the transformed coordinates and update to d _j to get d _j . After traversing and updating all units, the detection frame set sequence is updated to D'. The purpose of this step is to rotate and correct all the detected coordinates of the corresponding frame according to the rotation adjustment matrix of each frame and the calibration frame calculated by the image preprocessing module, so that the image coordinates of the corresponding frame are mapped to the image coordinates of the calibration frame. Through such correction, all the calculated coordinates are based on the same image coordinate system, thereby eliminating the error caused by the coordinate offset caused by the shaking of the screen.

Step 3: Input the detection frame set sequence D' obtained in Step 2 into the vehicle tracking module.

3) Vehicle tracking module

The vehicle tracking module is used to perform target tracking processing on the target detection frame set sequence output by the vehicle detection module. According to the target detection frame set sequence, obtain the position, size, lane and other information of all vehicles in each frame of the video. The information belonging to the same target is correlated and iteratively performed to obtain the complete trajectory information of all targets in the video. After that, the tracking result is mapped and transformed according to the homography matrix output by the image preprocessing module, and the trajectory is mapped from the image coordinates to World coordinates to get the target trajectory information set. The present invention uses the sort multi-target tracking algorithm to realize the function of the module. The sort multi-target tracking algorithm will use the Kalman filter method to estimate the position and size information of the next frame target according to the position and size information of the target in the current frame. After the Kalman filter predicts the motion state of the target, the Hungarian algorithm determines whether the detection frame is successfully associated with the target according to the IOU between the prediction frame of the previous frame and the detection frame of the frame. If the data association is successful, the detection frame is used to update the state of the target; if the data association fails, the linear model is used to predict the target. When the IOU between the detection frame in a certain frame and the prediction frame generated in the previous frame is less than a threshold, it can be considered that a new target has appeared, a new label is assigned to it, and the detection frame is used to predict the state of the target in the next frame . When the prediction frame of a target has no detection frame matching it in multiple frames, it can be considered that the target has disappeared and the prediction can be stopped. The specific steps are further described below.

Step 1: According to the detection frame set sequence D′ output by the vehicle detection module, use the Sort multi-target tracking algorithm based on Kalman filter and Hungarian matching to calculate the vehicle trajectory set T, and its unit t _i represents the trajectory information of the ith vehicle , including the image coordinate sequence of the target (x _ij , y _ij ), the vehicle type, and the lane sequence r _ij , where j is the frame number corresponding to this parameter, which is equivalent to recombining the target detection frame set sequence according to the IDs of different vehicles distribution to obtain the target image coordinate system trajectory information set.

并更新到t_i，得到w_i。遍历更新所有单元后，车辆轨迹集合更新为W。这一步实现了将所有车辆的轨迹坐标从标定帧的图像坐标系映射至真实世界坐标系，从而为后续跨摄像头轨迹匹配、交通参数的计算等提供了数据支持。Step 2: Based on the trajectory information set T output in step 1, perform homography transformation on the coordinate sequence (x _ij , y _ij ) of each t _i using the homography matrix K output by the image preprocessing module to obtain the transformed coordinates

And update to t _i , get _wi . After traversing and updating all units, the vehicle trajectory set is updated to W. This step realizes the mapping of the trajectory coordinates of all vehicles from the image coordinate system of the calibration frame to the real world coordinate system, thus providing data support for subsequent cross-camera trajectory matching and calculation of traffic parameters.

Step 3: The set of vehicle trajectories obtained from step 2 is updated to W, which is the output of the vehicle tracking module and the final output of the corresponding single-camera processing sub-module, which is subsequently integrated by the single-camera processing module.

After all sub-modules output the vehicle trajectory set, the single-camera processing module integrates the set output by each sub-module into a sequence in the order of the sub-modules, and inputs it to the cross-camera matching module. Combined with the example, during the data collection process, each UAV will capture the pictures of targets A and B traveling in the UAV collection area in different time periods. After processing, a separate set of vehicle trajectories for each video is extracted. Each set contains the trajectories of two targets, A and B, and these trajectories are in the same order as the sets in terms of time series. We need to integrate the trajectories of the same target in different sets to form a complete trajectory for the global monitoring range, so cross-camera matching technology is required for further processing.

4. Cross-camera matching module

The cross-camera matching module is used to perform information matching on the trajectory sets of multiple videos input by the single-camera processing module, and obtain the complete trajectories of all vehicles in the global video and input them to the traffic parameter extraction module. When there are multiple cameras to collect data, the data collected by each camera will be separately processed by the image preprocessing module, vehicle detection module, and vehicle tracking module. Camera data is matched to obtain global data. The present invention uses the method of cross-camera multi-target trajectory matching based on geographic location. According to a priori operation, there is a certain spatial overlap area between adjacent cameras. By comparing all the overlapping areas captured by two adjacent cameras at the same time If the world coordinates of the vehicle are very close, the two can be considered as the same target, so as to match the two targets from different sets, that is, the information of the same target in multiple target trajectory sets can be summarized and integrated into A unit to obtain the complete information of the target in the global monitoring scope. The specific steps are further described below.

Step 1: Obtain the vehicle trajectory set sequence W from the single-camera processing module, where Wn is the vehicle trajectory information set of the _nth video, and its unit _wni is the trajectory information of the ith target of the nth video. A global vehicle trajectory information set W _total ={} is established. Let n=1.

Step 2: Target W _n and use the data of W _n+1 for matching. A matching set m _ni is established for each w _ni in W _n . Let i=1.

Step 3: Traverse the data of W _n+1 , and classify w _n+1i into the set m _ni according to the following rules: it is the same type of vehicle; there are overlapping valid times, and the valid time refers to the time period from the appearance to the disappearance of the target; The lanes are the same during the effective time of the overlap. After the traversal is over, calculate the Euclidean distance between each element in m _ni and the trajectories of w _ni in the overlapping effective time, keep the elements with the smallest distance and lower than the set threshold in the collection, and delete the remaining elements. If m _ni is empty, add w _ni to the global vehicle trajectory information set W _total , otherwise, update the information of w _ni to the unit corresponding to W _n+1 for the remaining elements of m _ni .

Step 4: i=i+1, and then continue the loop from step 3 until all elements in W _n are traversed.

Step 5: n=n+1, if n is equal to the total number of videos in the video group, add all elements of W _n to W _total , otherwise, continue the loop from step 2.

Step 6: Sort and assign numbers to all elements in the set W _total according to their appearance time order. The set W _total contains the trajectory information of all targets in the global monitoring range of the video group, and summarizes the information of the same vehicle under different videos into the same unit. Finally, the global information set W _total is input into the traffic parameter extraction module.

The working process of the cross-camera matching module firstly obtains several target trajectory information sets obtained from the analysis of data collected from different cameras from the vehicle tracking module, and sorts the several target trajectory information sets according to the spatial order of the cameras. According to the method of cross-camera multi-target trajectory matching based on geographic location, first establish a global information set, which is an empty set, starting from the first set, and matching with the data of the second set according to the position of the trajectory in the overlapping area. After the matching is completed, the information of the successfully matched units is updated from the first set to the matched units in the second set, and the units that do not match successfully are updated to the global information set, indicating that the target corresponding to this unit is only in the first set. A camera appears, then the second set and the third set are matched, and so on, until the last set, all units in the last set are updated to the global information set, and all units in the global information set are updated Assign a global ID, so far, cross-camera matching is complete. The information of each unit of the global information set is the complete information of the unit corresponding to the target within the global monitoring range, including vehicle model information, position information and lane information at each moment.

Combined with the example, after the processing of the cross-camera matching module, the trajectory information of the A and B targets originally scattered in the corresponding trajectory sets of each video will be spliced one by one in chronological order to obtain a complete trajectory information unit, which includes each The information of the vehicle from entering the first drone collection area to leaving the last drone collection area. The advantage of this method is that by establishing the relationship between the image coordinate system and the world coordinate system, multiple original different image coordinate systems can be mapped to the same world coordinate system after matrix transformation, so as to realize the difference between different image coordinate systems. The integration of information. In the subsequent calculation of traffic parameters, since the data is based on the world coordinate system, the authenticity and reliability of the parameters are improved.

5. Traffic parameter extraction module

Traffic parameter extraction is the back-end module of the vehicle behavior analysis system. The function of this module is to calculate the trajectory information of each vehicle in the set based on the global trajectory set output by the cross-camera matching module and use the traffic parameter calculation formula to obtain the target. The single traffic parameter is calculated; the trajectory information of the two vehicles in the set is calculated to obtain the interactive traffic parameters of the two targets. According to the calculated traffic parameters, the traffic status of each vehicle is evaluated to determine whether the vehicle has bad behavior. At present, the traffic parameters that can be analyzed by the present invention include: vehicle speed, speeding parameters, and lane changing parameters for a single target; Users can also customize the calculation formula of traffic parameters according to the extracted data to enrich the output of this module. The calculation process of the different parameters is further described below.

1) The speed, overtaking parameters and lane change parameters of a single target.

A unit is arbitrarily selected from the global information set W _total or a unit is designated, and the unit includes the position information sequence l and the lane information sequence r corresponding to the monitoring time of the vehicle in the monitoring range. A low-pass filtering method is used to filter the sequence information to obtain the position information sequence l'. Set the distance parameter x, for the position information sequence l', starting from the xth node, calculate the Euclidean distance between the node and the first x-1th node, and then divide by the time difference between the two nodes, that is, calculate the node target. speed. Continue to traverse the sequence l' until all remaining nodes are calculated, and obtain the speed information sequence v. During the test, x is the frame rate of the video, which means that the speed of each point is calculated from the Euclidean distance between the position of the point and the position of the target one second ago. According to the speed information sequence v, we can know the change process of the target speed. By setting the threshold, we can judge whether the target is overspeeding during the driving process; according to the lane information sequence r, we can count the number of lane changes and The lane change time, by setting a threshold value, can determine whether the target has bad lane changing behavior during the driving process.

2) The front and rear car-following parameters, the head-to-head distance and the collision time of the two front and rear relationship targets.

计算车头时距TH，根据MOR公式

计算两车碰撞时间TTC，从而判断两车在行驶过程中是否存在碰撞危险。Selecting two targets with contextual relationship, extracting the two units from the global information set W _total and performing the data processing described in 1) respectively to obtain the two position information sequences l ₁ , l ₂ and velocity information sequences v ₁ , v ₂ . Calculate the Euclidean distance according to the corresponding time data of l ₁ and l ₂ , and the front and rear interval between the two can be obtained, that is, the vehicle following distance sequence D(t). By setting the threshold, it can be judged whether the latter has followed too closely during the driving process. behavior; according to the MOR formula

Calculate the headway TH, according to the MOR formula

Calculate the collision time TTC of the two vehicles, thereby judging whether the two vehicles are in danger of collision during the driving process.

3) Side-by-side driving parameters of two side-by-side relationships.

Select two targets in a side-by-side relationship, extract the two units from the global information set W _total , and perform the data processing described in 1) respectively to obtain the two position information sequences l ₁ , l ₂ , according to l ₁ , l ₂ The Euclidean distance is calculated according to the data at the time, and the side-by-side interval sequence D(h) of the two can be obtained. By setting the threshold, it can be judged whether the two have bad parallel behavior during the driving process.

With reference to an example, as shown in FIG. 3 , in the cross-camera matching module, the system assigns the ID of the target A vehicle as 11, and the ID of the target B vehicle as 13. Figure 3 records the speed of the two vehicles at this moment, the current lane, the number of lane changes per unit time, the following distance D of the two vehicles, the headway TH of the rear vehicle, and the collision time TTC. (Because the two vehicles are in a front-to-back state, the side-by-side driving parameters are not calculated).

Claims (3)

1. the road traffic behavior UAV monitoring method based on deep learning, is characterized in that, the method is based on the described road traffic behavior UAV monitoring system based on deep learning, comprises acquisition module, single camera processing module, cross-camera matching module and a traffic parameter extraction module; wherein, the acquisition module is used for the collection of aerial vehicle video data, and the collected multi-channel video data is input into the single-camera processing module to perform single-camera multi-target tracking processing for each channel of video; the single-camera processing module, It is used to perform multi-target tracking processing on the multi-channel video data output by the acquisition module, and obtain the vehicle trajectory set of each channel of video and input it to the cross-camera matching module; the cross-camera matching module is used to process each channel of video input by the single camera module The information matching is performed on the set of vehicle trajectories, and the complete trajectories of all vehicles in the global video are obtained and input to the traffic parameter extraction module; the traffic parameter extraction module is used to analyze the vehicle trajectory information input by the cross-camera matching module and extract its traffic parameters. , so as to obtain the traffic state of the target vehicle; The method includes the following steps: 1) Use several unmanned aerial vehicles to form a swarm of unmanned aerial vehicles to collect aerial vehicle video data, and store the collected data synchronously and input it to the single camera processing module; the specific implementation method is as follows: 401) Assign numbers to all unmanned aerial vehicles, arrange them from small to large, and take off in sequence at the leftmost end of the monitoring area according to the numbering sequence; 402) For No. 1 UAV, lift off vertically, hover at an altitude of about 250 meters from the ground, and then adjust the position so that the left side of the shooting screen is the leftmost end of the monitoring area; 403) For No. 2 UAV, after hovering at an altitude of about 250 meters above the ground, adjust the position so that the left area of the shooting picture and the right area of the No. 1 drone shooting picture have a real world length of about 30 meters overlap; 404) and so on, until the shooting picture of the drone group covers the entire monitoring area, and at the same time, the video recording function of all drones is turned on, and the video data is collected and stored synchronously. After the data collection is completed, the collected data is transferred from the drone to the database. ; 2) The single-camera processing module creates sub-modules for each channel of video to perform single-camera multi-target tracking processing according to the output video of the acquisition module, obtains the vehicle track set of each channel of video, and then integrates the vehicle track sets of all videos into the cross-camera. Matching module; the specific implementation method is as follows: 501) read the video collected by the drone numbered 1 from the database, process the video into a frame-based image sequence, and select one of the frames as the calibration image H to characterize the scene of the video; in the calibration image H Select more than 20 marked points. The marked points should be at the same level, and the entire monitoring screen should be distributed as evenly as possible, and then record the image coordinates of all marked points. Use Google Maps or GPS to collect the latitude and longitude position data of the above marked points, and convert them into Northeast sky coordinate information; arrange the image coordinates and northeast sky coordinates in corresponding order and input python's cv2.findHomography function to fit, and calculate the homography matrix K that maps the image coordinate system of the calibration frame to the world coordinate system; in the calibration image On H, for each lane area, mark several points along the edge of the lane area and save the image coordinates of these points; integrate the point set generated by each lane area into the npy file to represent the scene lane information; use ORB Algorithm to extract the position information L and feature information F of several feature points P of the calibration image slice; 502) use ORB algorithm, extract the position information L _j and characteristic information F _j of several characteristic points P _j of the jth frame image H _j of this video; Calculate characteristic point P according to the characteristic information F and characteristic information F _j of 501) One-to-one correspondence with the feature point P _j , and according to the correspondence and the position information L and position information L _j of 501), use python's cv2.findHomography function to fit, and calculate the image coordinate system of the currently processed image. The rotation adjustment matrix M _j of the image coordinate system mapping of the frame; According to the above method, traverse all the frames of the video according to the frame number, and obtain the rotation adjustment matrix sequence M; 503) Use the target detector based on deep learning to perform frame-by-frame target detection on the image sequence of 501), and obtain the detection frame of all vehicles in each frame of image, that is, the image position information and the corresponding vehicle model information; According to the lane information of 501), analyze each frame. The lane area where each detection frame is located; the above information is combined to obtain a detection frame set sequence D, whose unit d _j represents the detection frame set of the jth frame picture, including the image coordinates, vehicle type and lane of each detection frame; 504) According to the detection frame set sequence D obtained in 503), for the detection frame coordinates of each d _j , use the homography matrix M _j corresponding to the jth frame in the rotation adjustment matrix sequence M of 502) to perform homography transformation. , get the transformed coordinate and update it to d _j , get d _j ′; after traversing and updating all units, the detection frame set sequence is updated to D ′; 505) According to the detection frame set sequence D′ obtained in 504), use the Sort multi-target tracking algorithm based on Kalman filtering and Hungarian matching to calculate the vehicle trajectory set T, and its unit t _i represents the trajectory information of the i-th vehicle, It includes the image coordinate sequence (x _ij , y _ij ) of the target, the vehicle type, and the lane sequence r _ij where it is located, where j is the frame number corresponding to the parameter; 506) According to the vehicle trajectory set T obtained in 505), perform homography transformation on the coordinate sequence (x _ij , y _ij ) of each t _i using the homography matrix K of 501) to obtain the transformed coordinates

And update to t _i to get _wi ; after traversing and updating all units, the vehicle trajectory set is updated to W ₁ ; 507) Read the video collected by the unmanned aerial vehicle numbered 2 from the database, repeat steps 501)-506) to obtain a vehicle trajectory set and update it to W ₂ , and so on, traverse all videos to obtain a vehicle trajectory set sequence W; 3) The cross-camera matching module performs pairwise matching and information fusion on the vehicle trajectory data of each video according to the integrated data output by the single-camera processing module to obtain the complete trajectory information of each vehicle in the global video, and then compares the trajectory data of all vehicles. The information is integrated into a global trajectory set and input to the traffic parameter extraction module; the specific implementation method is as follows: 601) Obtain the vehicle trajectory set sequence W from the single-camera processing module, wherein W _n is the vehicle trajectory information set of the n th video, and its unit w _ni is the trajectory information of the ith target of the n th video, and establishes a global vehicle Trajectory information set W _total ={}, set n=1; 602) Take W _n as the target, use the data of W _n+1 to perform matching, establish a matching set m _ni for each w _ni in W _n , and set i=1; 603) Traverse the data of W _n+1 , and classify w _n+1i into the set m _ni according to the following rules: it is the same type of vehicle; there is an overlapping valid time, and the valid time refers to the time period from the appearance of the target to the disappearance; The lanes are the same within the valid time of , and after the traversal is over, calculate the Euclidean distance between each element in m _ni and the trajectory of w _ni within the valid time of overlap, and keep the element with the smallest distance and lower than the set threshold in the set, and the rest elements Delete; if m _ni is empty, add w _ni to the global vehicle trajectory information set W _total , otherwise, update the information of w _ni to the unit corresponding to the remaining elements of m _ni in W _n+1 ; 604) make i=i+1, and then continue to loop from 603) until all elements in W _n are traversed; 605) make n=n+1, if at this moment n is equal to the total number of video group videos, then add all elements of W _n to W _total , otherwise, continue to loop from 602); 606) Sort and assign numbers to all the elements in the set W _total according to the order of appearance time. The set W _total contains the trajectory information of all targets in the global monitoring range of the video group, and summarizes the information of the same vehicle under different videos into the same one. unit; 4) According to the global trajectory set input by the cross-camera matching module, the traffic parameter extraction module uses the traffic parameter calculation formula to calculate the trajectory information of each vehicle in the set to obtain a single traffic parameter of the target; Calculation is performed to obtain the interactive traffic parameters of the two targets; according to the calculated traffic parameters, the traffic state is evaluated for each vehicle. 2. the road traffic behavior UAV monitoring method based on deep learning according to claim 1, is characterized in that, the concrete realization method of step 4) is as follows: 701) obtain the global information set W _total from the cross-camera matching module; take any unit or specify a unit from the set, and this unit includes the position information sequence 1 and the lane information sequence r of the corresponding vehicle in the monitoring range monitoring time; use a low-pass The filtering method filters the sequence information to obtain the position information sequence l′; set the distance parameter x, for the position information sequence l′, starting from the xth node, calculate the Euclidean distance between the node and the first x-1th node, and then divide the Calculate the speed of the node target by the time difference between the two nodes; continue to traverse the sequence l' until all remaining nodes are calculated, and obtain the speed information sequence v; according to the speed information sequence v, calculate the speed change process, through Set a threshold to determine whether the target is overspeeding during driving; according to the lane information sequence r, count the number of lane changes and lane change time of the target within a unit time, and by setting the threshold, you can determine whether the target is in the process of driving. Bad lane changing behavior occurs; 702) Select two targets with contextual relationship, extract the two units from the global information set W _total , and perform the data processing of 701) respectively to obtain the position information sequences l ₁ , l ₂ and speed information sequences v ₁ , 1 , v ₂ ; Calculate the Euclidean distance according to the corresponding time data of l ₁ and l ₂ , and obtain the front and rear interval between the two, that is, the vehicle following distance sequence D(t), and determine whether the latter has followed during the driving process by setting a threshold Near behavior; according to the MOR formula

Calculate the headway TH, according to the MOR formula

Calculate the collision time TTC of the two vehicles, so as to determine whether the two vehicles are in danger of collision during the driving process; 703) Select two targets in a side-by-side relationship, extract the two units from the global information set W _total , and perform the data processing of 701) respectively to obtain the two position information sequences l ₁ , l ₂ , according to l ₁ , l ₂ The Euclidean distance is calculated according to the data at the time, and the side-by-side interval sequence D(h) of the two can be obtained. By setting the threshold, it can be judged whether the two have bad parallel behavior during the driving process. 3. the road traffic behavior unmanned aerial vehicle monitoring method based on deep learning according to claim 1, is characterized in that, single camera processing module creates single camera processing submodule to carry out data processing for every road video output by acquisition module; The single-camera processing sub-module includes an image preprocessing module, a vehicle detection module and a vehicle tracking module; the image preprocessing module is used to manually preprocess the output video of the acquisition module assigned by the single-camera processing submodule to obtain image sequences, lane information, Rotation adjustment matrix sequence and homography matrix, input lane information, image sequence and rotation adjustment matrix sequence into vehicle detection module; input homography matrix into vehicle tracking module; vehicle detection module is used for image sequence output by image preprocessing module Perform target detection processing to obtain information such as the position coordinates, pixel size and vehicle type of all vehicles in the image, and adjust and supplement the detection results according to the rotation adjustment matrix sequence and lane information output by the image preprocessing module to obtain target detection. The frame set sequence is input to the vehicle tracking module; the vehicle tracking module is used to perform target tracking processing on the target detection frame set sequence output by the vehicle detection module to obtain the complete trajectory information of all targets in the video, and according to the single output of the image preprocessing module. The adaptive matrix adjusts the tracking results to obtain the target trajectory information set; the target trajectory information set obtained by each single-camera processing sub-module is integrated, and then input to the cross-camera matching module.

Images