CN120220116B - A control system and method based on unmanned road patrol tow truck - Google Patents

Abstract

The invention provides a control system and a control method based on an unmanned road guard patrol wrecker, which relate to the technical field of automatic wrecker, and comprise a multi-mode sensing module consisting of an image sensor, a laser radar and a depth sensor, wherein two-dimensional image data, three-dimensional point cloud data and three-dimensional depth data are acquired, data characteristic fusion is carried out by utilizing a layered detection module combining a double-flow depth neural network and a self-attention mechanism, multi-level grading is output, accurate assessment of vehicle damage and intelligent selection of wrecker modes are realized, and a system is continuously optimized through self-adaptive learning. According to the invention, through the multi-sensor cooperation, double-flow deep neural network feature extraction and self-attention dynamic fusion mechanism, high-precision vehicle state analysis and intelligent obstacle clearance decision are realized, intelligent accurate judgment on road fault vehicle damage conditions is realized, secondary damage risks caused by damage evaluation errors are reduced, the frequency of manual intervention is reduced, and the efficiency of road obstacle clearance operation is remarkably improved.

Description

A control system and method based on unmanned road patrol tow truck

Technical Field

The present invention relates to the technical field of automatic obstacle removal, and in particular to a control system and method based on an unmanned road patrol obstacle removal vehicle.

Background Art

With the rapid development of artificial intelligence, autonomous driving, and smart transportation technologies, intelligent and automated road management systems centered around unmanned vehicles are gaining increasing attention. Among them, unmanned road patrol tow trucks, capable of autonomous patrols, timely detection of roadside vehicles, and rapid and effective disposal, have become a key development direction for intelligent transportation and smart cities. Tow trucks typically choose a towing method based on the damage to the vehicle. For example, for a minor tow truck, the tow truck tows the vehicle to a support structure using a tow hook. For a severely damaged tow truck, the tow truck connects to the vehicle's four wheels via a lifting mechanism and hoists it to a support structure. Alternatively, the tow truck lifts and secures one end of the vehicle through a lifting mechanism and then directly tows it for towing.

In recent years, researchers have proposed a number of automated or semi-automated tow truck removal solutions that rely on cameras or lidar to identify vehicles through optical visual recognition of their exterior features. However, current unmanned tow truck systems generally suffer from low recognition accuracy and limited data processing methods. This makes it difficult to accurately and effectively assess vehicle damage in complex environments, making it difficult to avoid manual intervention during actual tow truck removal operations. Furthermore, due to the lack of intelligence in data processing and decision-making, traditional unmanned tow trucks are unable to accurately distinguish between appropriate towing methods (such as towing, lifting, or hoisting) for different degrees of damage. This leads to frequent misjudgments, which reduces tow truck efficiency and can even cause secondary damage accidents.

Therefore, in order to meet the higher requirements of modern traffic management for efficiency, intelligence and reliability of obstacle removal operations, it is necessary to develop an unmanned obstacle removal control system and method with more accurate damage assessment capabilities, more flexible operation decision-making methods and more intelligent data fusion processing mechanisms.

Summary of the Invention

In order to overcome the defects of the prior art, the technical problem to be solved by the present invention is to propose a control system and method based on an unmanned road patrol tow truck, which adopts the following technical solutions:

On one hand, the present invention provides a control system based on an unmanned road patrol tow truck, comprising:

The multimodal perception module includes at least:

Image sensor, used to obtain two-dimensional image data of the faulty vehicle and extract vehicle body identification information through OCR;

A laser sensor for generating three-dimensional point cloud data of the faulty vehicle's appearance;

A depth sensor for generating three-dimensional depth data of the local appearance of the faulty vehicle;

Hierarchical detection module, including a deep neural network computing model with a two-stream architecture:

The first stream network uses a convolutional neural network to extract the image feature vector F _img of the two-dimensional image data and output a first confidence score C ₁ ;

The second stream network uses a point cloud processing algorithm to extract the point cloud feature vector _Fpc of the 3D point cloud data and the depth feature vector _Fdep of the 3D depth data; superimposes and fuses the 2D image data and the 3D point cloud data, and outputs a second confidence score _C2 ; if the second confidence score _C2 is ≤85, the 3D point cloud data and the 3D depth data are updated and fused, and a third confidence score _C3 is output;

The feature fusion unit uses a self-attention mechanism to weightedly fuse the image feature vector F _img , the point cloud feature vector F _{pc ,} and the depth feature vector F _dep to generate a fused feature vector F _fusion ;

The control processing unit is connected to the hierarchical detection module and selects the obstacle removal mode according to a preset threshold rule. When the third confidence score C ₃ ≤85, manual intervention is requested.

As a further improvement, the image sensor scans the front and rear of the faulty vehicle, and the calculation formula of the first confidence score _C1 is:

;

Where F _img represents the image feature vector output by the first-stream convolutional neural network;

W ₁ is the weight matrix, b ₁ is the bias term;

σ(·) is the Sigmoid activation function, which maps the above first confidence score _C1 to the range of 0~100.

Scan the front and rear of the faulty vehicle. Usually, the front and rear of the vehicle are marked with information such as the vehicle brand logo and vehicle model. The image sensor determines the damage to the front and rear of the vehicle on the one hand, and collects the vehicle information of the faulty vehicle through OCR on the other hand.

As a further improvement, the laser sensor scans both sides and the roof of the faulty vehicle, and the calculation formula of the second confidence score _C2 is:

;

Among them, W ₂ is the weight matrix and b ₂ is the bias term;

σ(·) is the Sigmoid activation function, which maps the second confidence score C ₂ to the range of 0~100;

The fusion feature vector F _{fusion (img, pc)} is generated by fusing the above two-dimensional image data and three-dimensional point cloud data through the self-attention mechanism. The specific formula is:

;

The weight coefficients γ1 and _γ2 satisfy γ1 + γ2 = 1, and when the first confidence score _C1 > ₇₅ , that is _, when the front and rear of _the faulty vehicle are slightly damaged , the weight coefficient γ1 is reduced and the weight coefficient γ2 _is increased _.

As a further improvement, the depth sensor scans the wheel and axle area of the faulty vehicle, and the calculation formula of the third confidence score _C3 is:

;

Among them, W ₃ is the weight matrix and b ₃ is the bias term;

σ(·) is the Sigmoid activation function, which maps the third confidence score C ₃ to the range of 0~100;

The fusion feature vector F _{fusion (pc, dep)} is generated by fusing the above two-dimensional image data and three-dimensional point cloud data through the self-attention mechanism. The specific formula is:

;

Wherein, M is a mask matrix, and the overlapping part of the above-mentioned three-dimensional point cloud data and the above-mentioned three-dimensional depth data is updated to the above-mentioned three-dimensional depth data through the mask matrix M;

The above weight coefficients γ ₂ ´ and γ ₃ satisfy γ ₂ ´ + γ ₃ = 1, and the weight coefficient γ ₃ > γ ₂ ´.

As a further improvement, it also includes an adaptive learning module, which is connected to the above-mentioned control processing unit and optimizes the weight coefficients in the above-mentioned feature fusion unit through offline learning and incremental learning.

As a further improvement, the offline learning of the above weight coefficients adopts the loss function of minimizing the mean square error:

;

Among them, F _target,i is the standard feature vector of the vehicle state marked after manual intervention.

As a further improvement, the incremental learning method of the above weight coefficients is:

When the recent system detection error shows a continuous increasing trend, the weight coefficient is updated using the following rules:

;

Accuracy is the percentage of tasks in the last five tasks where the error between the system and the test score and the manual verification score is within ±5;

Δ is the learning rate, initially limited to 0.01~0.05;

When Accuracy ≥ 95%, the weight coefficient update is stopped to maintain the parameter stability of the system.

As a further improvement, the preset rule for the control processing unit to select the obstacle removal mode is:

When the first confidence score C ₁ is greater than 75 and the second confidence score C ₂ is greater than 85, the control executes the towing mode to tow the faulty vehicle at a speed not exceeding 20 km/h;

When the first confidence score C ₁ >75 and the second confidence score C ₂ ≤85, if the third confidence score C ₃ >85, the lifting mode is executed; if the third confidence score C ₃ ≤85, manual intervention is requested;

When the first confidence score C ₁ ≤75 and the second confidence score C ₂ >85, the control executes the hoisting mode;

When the first confidence score C ₁ ≤75 and the second confidence score C ₂ ≤85, if the third confidence score C ₃ >85, the lifting mode is executed; if the third confidence score C ₃ ≤85, manual intervention is requested.

Another aspect of the present invention provides a control method based on an unmanned road patrol tow truck, which is applied to the control system proposed in any one of the above items, comprising the following steps:

S10: Control the image sensor to collect two-dimensional image data of the front and rear of the faulty vehicle, extract an image feature vector F _img of the two-dimensional image data through a first flow network, and generate a first confidence score C ₁ ;

S20: controlling the laser sensor to obtain three-dimensional point cloud data of both sides and the roof of the faulty vehicle, and extracting a point cloud feature vector F _pc of the three-dimensional point cloud data using a second stream network;

S21: superimposing and fusing the above-mentioned two-dimensional image data and three-dimensional point cloud data through a self-attention mechanism, and generating a second confidence score C ₂ ;

S30: Select the corresponding obstacle removal mode according to the scoring results:

When the first confidence score C ₁ >75 and the second confidence score C ₂ >85, the towing mode is selected to perform the towing operation by connecting the tow hook of the faulty vehicle;

When the first confidence score C ₁ ≤ 75 and the second confidence score C ₂ > 85, the hoisting mode is selected to lift the faulty vehicle by connecting the four wheels and perform the towing operation;

S40: If the second confidence score C ₂ ≤85, control the depth sensor to acquire three-dimensional depth data of the wheel and axle area of the faulty vehicle, and use the second flow network to extract a depth feature vector F _dep of the three-dimensional depth data;

S41: updating and fusing the above 3D point cloud data and 3D depth data through a self-attention mechanism, and generating a third confidence score C ₃ ;

S50: Select the corresponding obstacle removal mode based on the scoring results:

When the third confidence score C ₃ > 85, the lifting mode is selected to lift and fix one end of the faulty vehicle and tow the faulty vehicle for towing.

When the third confidence score C ₃ ≤85, manual intervention is requested;

S60: Records the scoring results of each task and actual clearance feedback in real time, and optimizes the model weight coefficients through online incremental learning and offline batch learning to improve the accuracy of subsequent detection tasks.

Compared with the prior art, the present invention has the following beneficial effects:

First, the technical solution of the present invention uses a multimodal perception module composed of image sensors, laser sensors and depth sensors to comprehensively perceive and collect data on faulty vehicles, and jointly constructs a multidimensional representation of vehicle damage with two-dimensional images, three-dimensional point clouds and three-dimensional depth data. The multimodal data acquisition method enables the system to overcome the problem of one-sided information recognition by a single sensor. Specifically, the image sensor obtains the front and rear damage of the vehicle body and vehicle identification information. The first confidence score _C1 is used to determine whether the towing conditions are met. The laser sensor further obtains three-dimensional point cloud data of the vehicle body, excluding the front and rear ends. This is superimposed with the two-dimensional image data to form a second confidence score _C2 , which represents the damage status of the entire vehicle body. If the towing conditions are not met, if the second confidence score _C2 is higher than a preset value, indicating that the vehicle body damage is minor, the four wheels of the faulty vehicle are connected through a lifting mechanism and the vehicle is lifted for towing. Furthermore, if the second confidence score _C2 is lower than the preset value, indicating that the vehicle body damage is severe, lifting the vehicle may overturn due to imbalance. Therefore, the depth sensor scans the wheel and axle area and outputs a third confidence score _C3 to determine whether the wheel can rotate. If it can, the towing mode is adopted for towing. The above technical solution effectively improves the accuracy and comprehensiveness of the assessment of damage to faulty vehicles. By judging the damage to the vehicle and selecting the appropriate clearance method, it significantly reduces the probability of misjudgment while effectively avoiding the risk of secondary damage caused by wrong decisions, thereby improving the overall operational efficiency and safety of traffic management, and has obvious practical application advantages.

Secondly, the technical solution of the present invention includes a hierarchical detection module based on a dual-stream architecture, which processes multiple modal data separately through a deep neural network to achieve an efficient feature extraction and feature fusion process. When the second confidence score _C2 is not enough to provide a clear judgment, the system will update the fused three-dimensional depth data, replace the three-dimensional depth data representing the wheels and axles with the original three-dimensional point cloud data, and generate a more refined third confidence score _C3 to enhance the damage identification accuracy of the wheel area. The step-by-step refinement of the evaluation method effectively avoids the misjudgment of the operation mode due to insufficient damage assessment. In the case of minor damage, the depth sensor is not called, which saves system resources, greatly improves the system's adaptability to complex scenarios, and improves the accuracy of decision-making and the reliability of obstacle clearance operation selection.

Third, the technical solution of the present invention also includes an adaptive learning module to achieve dynamic optimization of the weight coefficient. After manual intervention, the processing results are entered into the system, a standard feature vector is generated, and the weight coefficient is corrected through offline learning. On the other hand, when the deviation between the last five scoring results and manual verification is too large, the weight coefficient is corrected through incremental learning. The adaptive learning module effectively enhances the robustness and environmental adaptability of the system, ensuring the high precision and stability of the unmanned tow truck control system in long-term operation under actual complex road conditions.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

FIG1 is a schematic diagram of the framework of the system of the present invention;

FIG2 is a flow chart of the steps of the method of the present invention;

FIG3 is a flow chart of the obstacle removal mode selection rule in the present invention.

DETAILED DESCRIPTION

In order to facilitate understanding by those skilled in the art, the structure of the present invention is further described in detail with reference to the embodiments and the accompanying drawings:

In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the technical features indicated. Terms such as "portion," "side," and "end" indicate positions or relationships based on those shown in the accompanying drawings and are intended solely to facilitate and simplify the description of the present invention. They do not indicate or imply that the devices or components indicated must have, be constructed, or operate in a specific orientation. Therefore, they should not be construed as limitations on the present invention.

As shown in FIG1 , the present application provides a control system based on an unmanned road patrol tow truck, comprising a multimodal perception module, a hierarchical detection module, and a control processing unit.

As shown in Figure 1, the multimodal perception module includes at least an image sensor, a laser sensor, and a depth sensor. The image sensor, specifically an optical imaging device such as a high-resolution camera, is used to capture 2D image data of the faulty vehicle and extract vehicle identification information through optical character recognition (OCR). It can clearly capture the faulty vehicle's license plate number, vehicle model, and surface damage details. The laser sensor, specifically a device such as a 3D lidar, generates 3D point cloud data of the faulty vehicle's exterior. This device actively emits laser pulses and measures reflection time to obtain 3D point cloud data of the target object's surface. LiDAR offers high distance and angle measurement accuracy and is unaffected by ambient lighting variations. It can efficiently and stably acquire 3D structural data of the faulty vehicle's exterior and damaged areas. The depth sensor, specifically a depth imaging device such as a structured light scanner or time-of-flight camera, generates 3D depth data of the faulty vehicle's exterior. By calculating target depth information through light deformation, it is suitable for accurately capturing detailed damage information on the vehicle's chassis, wheels, and axles. This facilitates further assessment of the vehicle's mobility and avoids unsuccessful clearance operations due to missed underbody damage. The above-mentioned image sensors, laser sensors and depth sensors are controlled by the movement of the tow truck body and the body robotic arm to scan the front, rear, left and right sides, roof and other areas of the faulty vehicle, and align the data through spatiotemporal calibration. The data representation error is ≤2cm, so that subsequent data fusion is accurate.

Furthermore, as shown in Figure 1, the layered detection module includes a deep neural network computing model with a two-stream architecture. Specifically, the first-stream network uses a convolutional neural network, such as a ResNet or DenseNet model. Through multi-layer convolution operations, it gradually extracts image feature vectors F _img from the input two-dimensional image data to form a first confidence score C ₁ . This score is used to determine the extent of damage to the front and rear of the faulty vehicle and whether the faulty vehicle meets the towing conditions.

The second-stream network utilizes a point cloud processing neural network model based on PointNet or PointNet++ to fuse and extract features from the 3D point cloud data acquired by the laser sensor and the 3D depth data acquired by the depth sensor. The neural network's internal Multi-Layered Propagation (MLP) structure performs feature mapping, clustering, and dimensionality reduction on each point cloud data point, generating a point cloud feature vector F _pc for the 3D point cloud data and a depth feature vector F _dep for the 3D depth data. The 2D image data and 3D point cloud data are then superimposed and fused to output a second confidence score C ₂ . In one embodiment, the 2D image data represents damage information on the front and rear of the vehicle body, while the 3D point cloud data represents damage information on the sides and roof of the vehicle body. The superposition and fusion of the two represents damage information on the exterior of the vehicle body. If the second confidence score C ₂ ≤ 85, the 3D point cloud data and 3D depth data are updated and fused to output a third confidence score C ₃ , which is used to further determine the damage extent of the faulty vehicle and the applicable clearance method. Specifically, the 3D depth data represents local information about the wheels and axles, which overlaps with some of the representational information of the 3D point cloud data. Therefore, the 3D depth data is updated and incorporated into the 3D point cloud data. That is, the two are updated and fused to form new 3D data, which is used to further determine the damage extent of the faulty vehicle's wheels.

Furthermore, the hierarchical detection module includes a feature fusion unit. This unit utilizes a self-attention mechanism based on the Transformer architecture. When fusing feature vectors from multiple data modalities, this mechanism automatically generates attention weights based on the importance of each modal feature to highlight key damage information. This feature fusion process further mitigates the negative impact of missing or misjudged information from a single modality, improving the accuracy and robustness of overall vehicle damage assessment. Specifically, the self-attention mechanism is used to weightedly fuse the image feature vector F _img , the point cloud feature vector F _{pc ,} and the depth feature vector F _dep , generating the fused feature vector F _fusion .

As shown in FIG1 , the control processing unit is connected to the hierarchical detection module and selects the obstacle removal mode according to a preset threshold rule. When the third confidence score C ₃ ≤85, manual intervention is requested.

As shown in FIG3 , the preset rules for the control processing unit to select the obstacle removal mode are specifically:

When the first confidence score C ₁ >75 and the second confidence score C ₂ >85, the control executes the towing mode to tow the faulty vehicle at a speed not exceeding 20 km/h.

In this rule, when the first confidence score C ₁ > 75, it is determined that the damage to the front and rear of the vehicle is minor, meeting the conditions for installing a tow hook. Therefore, the damage to the vehicle body is further determined. If the second confidence score C ₂ > 85, it is determined that the towing mode can be executed.

When the first confidence score C ₁ ≤75 and the second confidence score C ₂ >85, the control executes the hoisting mode.

In this rule, when the first confidence score C ₁ ≤ 75, it indicates that the front and rear of the vehicle are severely damaged and do not meet the installation conditions for a tow hook. Therefore, the damage to the vehicle body is further determined. If the second confidence score C ₂ > 85, it indicates that the damage to the vehicle body is relatively minor. The lifting mode is executed. The faulty vehicle is fixed to the wheel area through the lifting mechanism, lifted, and placed on the load-bearing part of the tow truck for transportation.

When the first confidence score C ₁ >75 and the second confidence score C ₂ ≤85, if the third confidence score C ₃ >85, the control executes the lifting mode; if the third confidence score C ₃ ≤85, the control requests manual intervention.

When the first confidence score C ₁ ≤75 and the second confidence score C ₂ ≤85, if the third confidence score C ₃ >85, the control executes the lifting mode; if the third confidence score C ₃ ≤85, the control requests manual intervention.

In this rule, if the second confidence score C ₂ ≤ 85, it indicates significant vehicle damage. The damage could be to the vehicle or the wheels. If the vehicle is severely damaged, the imbalance of the vehicle could cause the vehicle to overturn during lifting, resulting in secondary damage. If the wheels are severely damaged, towing could result in unpredictable risks due to wheel loss of steering or directional deviation. Therefore, the third confidence score C ₃ is required for further analysis. If the third confidence score C ₃ > 85, the wheel damage is minor (the first and second confidence scores C ₁ and C ₂ are not considered in this case). Lifting is recommended for towing purposes: one end of the vehicle is lifted, the other end supported by the wheels, and the tow truck tows the vehicle. If the third confidence score C ₃ ≤ 85, the wheel area is severely damaged, requiring manual intervention and a more complex towing method, such as lifting the vehicle chassis or manually securing the vehicle for transportation.

In one embodiment, a tow truck uses information such as the vehicle brand, vehicle model, and vehicle identification code identified by an image sensor to move the faulty vehicle to a maintenance site or temporary storage point on a preset route, thereby completing an automated tow removal task.

The present invention proposes an unmanned tow truck control system based on multimodal perception and hierarchical confidence assessment. Through multi-sensor collaboration, dual-stream deep neural network feature extraction and self-attention dynamic fusion mechanism, high-precision vehicle status analysis and intelligent tow clearance decision-making are achieved, and intelligent and accurate judgment of the damage status of vehicles with road faults is realized. It can effectively reduce the risk of secondary damage caused by damage assessment errors and reduce the frequency of manual intervention. It has the advantages of high degree of automation, strong stability, and strong environmental adaptability. It can significantly improve the efficiency, safety and intelligence level of road towing operations, and has broad practical application value and good market prospects.

The working principles of the hierarchical detection module and the adaptive learning module are described below through a specific embodiment:

The image sensor scans the front and rear of the faulty vehicle, and the calculation formula for the first confidence score _C1 is:

;

Where F _img represents the image feature vector output by the first-stream convolutional neural network, W ₁ is the weight matrix, which is a vector of the same dimension as F _img and is used to map F _img to the score space, b ₁ is the bias term with an initial value of 0, and σ (·) is the Sigmoid activation function, whose original linear output mapping interval is [0, 1]. It is amplified by 100 times to map the first confidence score C ₁ to the range of 0 to 100.

The calculation formula of image feature vector F _img is:

;

Among them, Q _img , K _img , and V _img are the query matrix Q, key matrix K, and value matrix V in the self-attention mechanism, which are generated from two-dimensional image data.

The laser sensor scans both sides and the roof of the faulty vehicle. The second confidence score _C2 is calculated as follows:

;

Among them, W ₂ is the weight matrix, b ₂ is the bias term; σ (·) is the Sigmoid activation function, which is amplified 100 times to map the second confidence score C ₂ to the range of 0~100.

The fusion feature vector F _{fusion (img, pc)} is generated by fusing the two-dimensional image data and the three-dimensional point cloud data through the self-attention mechanism. The specific formula is:

;

Among them, Q _img , K _img , V _img are the query matrix Q, key matrix K, and value matrix V in the self-attention mechanism, which are generated from two-dimensional image data; Q _pc , K _pc , V _pc are the query matrix Q, key matrix K, and value matrix V generated from three-dimensional point cloud data.

The weight coefficients γ1 and _γ2 satisfy γ1 + γ2 = 1. When the first confidence score _C1 > ₇₅ , i.e., when _the front and rear of _the faulty vehicle are less damaged , the weight coefficient γ1 is reduced and the weight coefficient _γ2 is _increased . In a preferred embodiment, the initial values _of γ1 _and γ2 are 0.5 and 0.5, respectively. If _C1 > 75, γ1 is _changed to 0.3 and γ2 _is changed to 0.7.

The depth sensor scans the wheel and axle area of the faulty vehicle, and the third confidence score _C3 is calculated as follows:

;

Where W ₃ is the weight matrix, b ₃ is the bias term, and σ(·) is the Sigmoid activation function, which is amplified by 100 times to map the third confidence score C ₃ to the range of 0~100.

The fusion feature vector F _{fusion (pc, dep)} is generated by fusing the two-dimensional image data and the three-dimensional point cloud data through the self-attention mechanism. The specific formula is:

;

Where M is a mask matrix, and the overlapping portion of the 3D point cloud data and the 3D depth data is updated to 3D depth data using the mask matrix M. In this embodiment, the overlapping portion is the side surface of the wheel and the axle. In another embodiment, the depth sensor is also used to detect the vehicle chassis, in which case the 3D depth data of the vehicle chassis is not included in the update of the 3D point cloud data.

The weight coefficients γ ₂ ´ and γ ₃ satisfy γ ₂ ´ + γ ₃ = 1, and the weight coefficient γ ₃ > γ ₂ ´. Preferably, γ ₂ ´ = 0.3, γ ₃ = 0.7.

The system also includes an adaptive learning module connected to the control processing unit, which optimizes the weight coefficients in the feature fusion unit through offline learning and incremental learning.

The offline learning of weight coefficients adopts the loss function of minimizing the mean square error:

;

Here, F _target,i is the standard feature vector of the vehicle state annotated after manual intervention. This can be a standard feature vector converted from historical average data or manually calibrated by professionals. Using N sets of data, the fused feature vector F _{fusion (·)} is forward-calculated to update the aforementioned learnable variables, including the weight matrix and weight parameters.

The incremental learning method of weight coefficients is:

When the recent system detection error shows a continuous increasing trend, the weight coefficient is updated using the following rules:

;

Accuracy is the percentage of tasks in the last five tasks where the error between the system and the test score and the manual verification score is within ±5;

Δ is the learning rate, initially limited to 0.01~0.05;

When Accuracy ≥ 95%, the weight coefficient update is stopped to maintain the parameter stability of the system.

In some embodiments, the learning rate can be dynamically adjusted with the accuracy, for example:

If Accuracy = 93%, Δ is reduced to 0.01, stabilizing the system;

If Accuracy = 70%, Δ is increased to 0.05 to accelerate convergence.

As shown in FIG2 , another aspect of the present invention provides a control method based on an unmanned road patrol tow truck, which is applied to the above-mentioned control system and includes the following steps:

S10: Controlling the image sensor to collect two-dimensional image data of the front and rear of the faulty vehicle, extracting an image feature vector F _img of the two-dimensional image data through a first-stream network, and generating a first confidence score C ₁ ;

S20: Controlling the laser sensor to obtain three-dimensional point cloud data of both sides of the body and the roof area of the faulty vehicle, and using the second stream network to extract the point cloud feature vector F _pc of the three-dimensional point cloud data;

S21: superimpose and fuse the two-dimensional image data and the three-dimensional point cloud data through the self-attention mechanism, and generate a second confidence score C ₂ ;

S30: Select the corresponding obstacle removal mode according to the scoring results:

When the first confidence score C ₁ >75 and the second confidence score C ₂ >85, the towing mode is selected to perform the towing operation by connecting the tow hook of the faulty vehicle;

When the first confidence score C ₁ ≤ 75 and the second confidence score C ₂ > 85, the hoisting mode is selected to lift the faulty vehicle by connecting the four wheels and perform the towing operation;

S40: If the second confidence score C ₂ ≤85, control the depth sensor to acquire three-dimensional depth data of the wheel and axle area of the faulty vehicle, and use the second flow network to extract a depth feature vector F _dep of the three-dimensional depth data;

S41: updating and fusing the 3D point cloud data and the 3D depth data through a self-attention mechanism, and generating a third confidence score C ₃ ;

S50: Select the corresponding obstacle removal mode based on the scoring results:

When the third confidence score C ₃ > 85, the lifting mode is selected to lift and fix one end of the faulty vehicle and tow the faulty vehicle for towing.

When the third confidence score C ₃ ≤85, manual intervention is requested;

S60: Records the scoring results of each task and actual clearance feedback in real time, and optimizes the model weight coefficients through online incremental learning and offline batch learning to improve the accuracy of subsequent detection tasks.

The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims (9)

1. Control system based on unmanned road on duty patrol clearance car, characterized in that includes:

the multimode sensing module at least comprises:

the image sensor is used for acquiring two-dimensional image data of the fault vehicle and extracting vehicle body identification information through OCR;

A laser sensor for generating three-dimensional point cloud data of the appearance of the faulty vehicle;

a depth sensor for generating three-dimensional depth data of a local appearance of the faulty vehicle;

the layering detection module comprises a deep neural network calculation model with a double-flow architecture:

the first flow network is used for extracting an image feature vector F _img of the two-dimensional image data by adopting a convolutional neural network and outputting a first confidence score C ₁;

The second flow network adopts a point cloud processing algorithm to extract a point cloud characteristic vector F _pc of the three-dimensional point cloud data and a depth characteristic vector F _dep of the three-dimensional depth data, superimposes and fuses the two-dimensional image data and the three-dimensional point cloud data to output a second confidence score C ₂, and updates and fuses the three-dimensional point cloud data and the three-dimensional depth data to output a third confidence score C ₃ if the second confidence score C ₂ is less than or equal to 85;

The feature fusion unit adopts a self-attention mechanism to weight and fuse the image feature vector F _img, the point cloud feature vector F _pc and the depth feature vector F _dep to generate a fusion feature vector F _fusion;

And the control processing unit is connected with the layering detection module, selects an obstacle clearance mode according to a preset threshold rule, and requests manual intervention when the third confidence score C ₃ is less than or equal to 85.

2. The unmanned road patrol wrecker-based control system of claim 1, wherein the image sensor scans the head and tail of a faulty vehicle, and the first confidence score C ₁ is calculated as:

;

Wherein F _img represents an image feature vector output by the first stream convolutional neural network;

W ₁ is a weight matrix, and b ₁ is a bias term;

Sigma (·) is a Sigmoid activation function, mapping the first confidence score C ₁ to a range of 0-100.

3. The unmanned road guard patrol wrecker-based control system of claim 2, wherein the laser sensor scans both sides and roof of the body of the faulty vehicle, and the second confidence score C ₂ is calculated as:

;

wherein W ₂ is a weight matrix, and b ₂ is a bias term;

Sigma (·) is a Sigmoid activation function, so that the second confidence score C ₂ is mapped to a range of 0-100;

The fusion feature vector F _{fusion(img,pc)} is generated by fusion of the two-dimensional image data and the three-dimensional point cloud data through a self-attention mechanism, and the specific formula is as follows:

;

The weight coefficients gamma ₁ and gamma ₂ meet the requirement that gamma ₁+γ₂ =1, and when the first confidence score C ₁ is greater than 75, namely the head and tail of the fault vehicle are damaged slightly, the weight coefficient gamma ₁ is reduced, and the weight coefficient gamma ₂ is improved.

4. A control system based on an unmanned road duty patrol wrecker as claimed in claim 3, wherein the depth sensor scans the wheel and axle area of the faulty vehicle, and the third confidence score C ₃ is calculated as:

;

Wherein W ₃ is a weight matrix, and b ₃ is a bias term;

Sigma (·) is a Sigmoid activation function, so that the third confidence score C ₃ is mapped to a range of 0-100;

The fusion feature vector F _{fusion(pc,dep)} is generated by fusion of the two-dimensional image data and the three-dimensional point cloud data through a self-attention mechanism, and the specific formula is as follows:

;

Wherein M is a mask matrix, the overlapped part of the three-dimensional point cloud data and the three-dimensional depth data is updated into the three-dimensional depth data through the mask matrix M, as the laser sensor scans the two sides and the top of the vehicle body, the depth sensor scans the wheels and the wheel shafts, and certain overlapped areas exist in the two scanning areas, the mask matrix M with the same dimension as the data feature vector dimension is introduced, wherein the element is 0 or 1, specifically defined as that when one element of the mask matrix M is 1, the corresponding position is simultaneously scanned by the laser sensor and the depth sensor, the position is replaced by the three-dimensional depth data of the depth sensor,

The weight coefficients γ ₂ ́ and γ ₃ satisfy γ ₂´+γ₃ =1, and the weight coefficient γ ₃＞γ₂ ́.

5. The unmanned road patrol wrecker-based control system of claim 1, further comprising an adaptive learning module, in communication with the control processing unit, for optimizing the weight coefficients in the feature fusion unit by offline learning and incremental learning.

6. The unmanned road patrol wrecker-based control system of claim 5, wherein the offline learning of the weight coefficients employs a loss function that minimizes mean square error:

;

wherein F _target,i is a standard feature vector of the vehicle state noted after the human intervention.

7. The control system based on the unmanned road guard patrol wrecker as claimed in claim 6, wherein the incremental learning method of the weight coefficient is as follows:

When the recent system detection error has a continuous increasing trend, the weight coefficient is updated by adopting the following rule:

;

among the latest 5 tasks, accuracy is the task duty ratio that the error between the system and the measurement score and the artificial check score is within +/-5;

delta is learning rate and is initially limited to 0.01-0.05;

and stopping updating the weight coefficient when Accuracy is more than or equal to 95% so as to maintain the parameter stability of the system.

8. The control system based on the unmanned road guard patrol wrecker as claimed in claim 1, wherein the preset rules for selecting the wrecker mode by the control processing unit are:

When the first confidence score C ₁ >75 and the second confidence score C ₂ >85, controlling to execute a traction mode to traction the faulty vehicle at a speed of not more than 20 km/h;

when the first confidence score C ₁ is more than 75 and the second confidence score C ₂ is less than or equal to 85, controlling to execute a lifting mode if the third confidence score C ₃ is more than 85, and requesting manual intervention if the third confidence score C ₃ is less than or equal to 85;

When the first confidence score C ₁ is less than or equal to 75 and the second confidence score C ₂ is more than 85, controlling to execute a hoisting mode;

when the first confidence score C ₁ is less than or equal to 75 and the second confidence score C ₂ is less than or equal to 85, controlling to execute a lifting mode if the third confidence score C ₃ >85, and requesting manual intervention if the third confidence score C ₃ is less than or equal to 85.

9. A control method based on unmanned road patrol wrecker, which is applied to the control system as claimed in any one of claims 1-8, and is characterized by comprising the following steps:

S10, controlling the image sensor to acquire two-dimensional image data of the head and the tail of a fault vehicle, extracting an image feature vector F _img of the two-dimensional image data through a first flow network, and generating a first confidence score C ₁;

S20, controlling the laser sensor to obtain three-dimensional point cloud data of two sides of a vehicle body and a roof area of a fault vehicle, and extracting a point cloud feature vector F _pc of the three-dimensional point cloud data by using a second flow network;

S21, superposing and fusing the two-dimensional image data and the three-dimensional point cloud data through a self-attention mechanism, and generating a second confidence score C ₂;

s30, selecting a corresponding obstacle clearance mode according to the grading result:

When the first confidence score C ₁ is more than 75 and the second confidence score C ₂ is more than 85, selecting a traction mode, and performing obstacle clearing operation through a trailer hook connected with a fault vehicle;

When the first confidence score C ₁ is less than or equal to 75 and the second confidence score C ₂ is more than 85, selecting a hoisting mode, hoisting the fault vehicle by connecting four wheels of the fault vehicle, and performing obstacle clearing operation;

S40, if the second confidence score C ₂ is less than or equal to 85, controlling the depth sensor to acquire three-dimensional depth data of the wheel and axle area of the fault vehicle, and extracting a depth feature vector F _dep of the three-dimensional depth data by using a second flow network;

S41, updating and fusing the three-dimensional point cloud data and the three-dimensional depth data through a self-attention mechanism, and generating a third confidence score C ₃;

s50, selecting a corresponding obstacle clearance mode according to the grading result:

When the third confidence score C ₃ is more than 85, selecting a lifting mode, lifting and fixing one end of the fault vehicle, and dragging the fault vehicle to perform obstacle clearing operation;

When the third confidence score C ₃ is less than or equal to 85, requesting manual intervention;

And S60, recording the grading result of each task and the actual obstacle clearance operation feedback in real time, and optimizing the model weight coefficient in an online increment learning and offline batch learning mode so as to improve the accuracy of the subsequent detection task.