CN116653974A - A vehicle control method, device, vehicle-mounted equipment, and vehicle - Google Patents

Abstract

The disclosure relates to a vehicle control method and device, vehicle-mounted equipment and a vehicle, and particularly relates to the technical field of automation. The vehicle control method includes: synthesizing an original image of the bottom of the target vehicle according to the visual data of the target vehicle, wherein the visual data comprises the image of the bottom of the target vehicle and a looking-around image of the target vehicle; inputting the original image into a road condition recognition model to obtain road condition information output by the road condition recognition model; controlling the target vehicle to run based on the road condition information; the road condition recognition model is a model obtained based on training of a target training sample, wherein the target training sample comprises a plurality of images of the bottoms of vehicles and road condition information corresponding to the images of the bottoms of each vehicle. The method and the device are used for improving the accuracy of obstacle avoidance in the driving process.

Description

A vehicle control method, device, vehicle-mounted equipment, and vehicle

technical field

The present disclosure relates to the technical field of automation, and in particular to a vehicle control method, device, vehicle-mounted equipment and vehicle.

Background technique

Obstacle avoidance is very important in the process of driving a vehicle. At present, the vehicle's surround view camera is used to collect images, and then the images are stitched to obtain the front and side images of the vehicle, and based on these images to judge the obstacles around the vehicle, to realize the vehicle driving process. Aid in obstacle avoidance. However, at present, the road conditions at the bottom of the vehicle cannot provide road condition information to assist in avoiding obstacles during driving. Therefore, there is an urgent need for a method of controlling vehicle driving through the road condition information under the vehicle chassis to improve the accuracy of obstacle avoidance during driving.

Contents of the invention

In order to solve the above-mentioned technical problems or at least partly solve the above-mentioned technical problems, the present disclosure provides a vehicle control method, device, on-board equipment and vehicle, which can recognize the road condition information under the vehicle chassis through the original image of the vehicle bottom, and based on the road condition The information controls the driving of the vehicle, which can improve the accuracy of obstacle avoidance during driving.

In order to achieve the above purpose, the technical solutions provided by the embodiments of the present disclosure are as follows:

In a first aspect, a vehicle control method is provided, including:

Synthesizing the original image of the bottom of the target vehicle according to the visual data of the target vehicle, wherein the visual data includes an image of the bottom of the target vehicle and a surround view image of the target vehicle;

inputting the original image into the road condition recognition model, and obtaining the road condition information output by the road condition recognition model;

controlling the target vehicle to travel based on the road condition information;

Wherein, the road condition recognition model is a model trained based on target training samples, and the target training samples include images of a plurality of vehicle bottoms and road condition information corresponding to each vehicle bottom image.

As an optional implementation manner of an embodiment of the present disclosure, the road condition information includes at least one of the following:

The position of the obstacle, the size of the obstacle, the distance between the obstacle and the chassis of the target vehicle, the type of the obstacle, the position of the pothole, the size of the pothole, and the depth of the pothole.

As an optional implementation manner of the embodiment of the present disclosure, the road condition recognition model includes: an obstacle recognition model and a distance measurement model;

The step of inputting the original image into the road condition recognition model and obtaining the road condition information output by the road condition recognition model includes:

The original image is input into the obstacle recognition model, and the road condition image output by the obstacle recognition model is obtained, and the road condition image is an image marked with first information on the original image, and the first information includes the following at least one of:

Obstacle location, obstacle size, obstacle type, pothole location, pothole size;

The road condition image is input into the distance measurement model, and the second information output by the distance measurement model is obtained, and the second information includes at least one of the following:

The distance from the obstacle to the chassis of the target vehicle, the depth of potholes;

The first information and the second information are used as the road condition information.

As an optional implementation of the embodiment of the present disclosure, the obstacle recognition model is a model trained based on the first training sample, the first training sample includes images of multiple vehicle bottoms, and each vehicle bottom The first image corresponding to the image of the first image is an image marked with the first information on the image of the bottom of the vehicle;

and / or,

The distance measurement model is a model trained based on a second training sample, and the second training sample includes a plurality of the first images and the second information corresponding to each first image.

As an optional implementation manner of the embodiment of the present disclosure, before synthesizing the original image of the bottom of the target vehicle according to the visual data of the target vehicle, the method further includes:

determining that the target vehicle is in a driving state;

After said synthesizing the original image of the bottom of the target vehicle according to the visual data of the target vehicle, the method further includes:

Display the original image.

As an optional implementation manner of the embodiment of the present disclosure, after inputting the original image into the road condition recognition model and obtaining the road condition information output by the road condition recognition model, the method further includes:

generating a road condition display image based on the road condition information and the original image;

Display the traffic display image.

In a second aspect, a vehicle control device is provided, the vehicle control device is connected to visual sensors arranged at the bottom and around the target vehicle, including:

A synthesis module, configured to synthesize the original image of the bottom of the target vehicle according to the visual data of the target vehicle, wherein the visual data includes an image of the bottom of the target vehicle and a surround view image of the target vehicle;

an acquisition module, which inputs the original image into a road condition recognition model, and acquires road condition information output by the road condition recognition model;

a control module, configured to control the driving of the target vehicle based on the road condition information;

Wherein, the road condition recognition model is a model trained based on target training samples, and the target training samples include images of a plurality of vehicle bottoms and road condition information corresponding to each vehicle bottom image.

As an optional implementation manner of an embodiment of the present disclosure, the road condition information includes at least one of the following:

The position of the obstacle, the size of the obstacle, the distance between the obstacle and the chassis of the target vehicle, the type of the obstacle, the position of the pothole, the size of the pothole, and the depth of the pothole.

As an optional implementation manner of the embodiment of the present disclosure, the road condition recognition model includes: an obstacle recognition model and a distance measurement model;

The obtaining module is specifically configured to: input the original image into the obstacle recognition model, and obtain a road condition image output by the obstacle recognition model, and the road condition image is marked with the first information on the original image image, the first information includes at least one of the following:

Obstacle location, obstacle size, obstacle type, pothole location, pothole size;

The road condition image is input into the distance measurement model, and the second information output by the distance measurement model is obtained, and the second information includes at least one of the following:

The distance from the obstacle to the chassis of the target vehicle, the depth of potholes;

The first information and the second information are used as the road condition information.

As an optional implementation of the embodiment of the present disclosure, the obstacle recognition model is a model trained based on the first training sample, the first training sample includes images of multiple vehicle bottoms, and each vehicle bottom The first image corresponding to the image of the first image is an image marked with the first information on the image of the bottom of the vehicle,

As an optional implementation manner of this embodiment of the present disclosure, the distance measurement model is a model trained based on a second training sample, the second training sample includes a plurality of the first images, and each first The second information corresponding to the image.

As an optional implementation manner of the embodiment of the present disclosure, the acquiring module is further configured to:

determining that the target vehicle is in a driving state;

And after the original image of the bottom of the target vehicle is synthesized according to the visual data of the target vehicle, the original image is displayed.

As an optional implementation manner of the embodiment of the present disclosure, the acquiring module is further configured to:

Said inputting the original image into the road condition recognition model, after acquiring the road condition information output by the road condition recognition model, generating a road condition display image based on the road condition information and the original image; displaying the road condition display image.

In a third aspect, a vehicle-mounted device is provided, including: a processor, a memory, and a computer program stored on the memory and operable on the processor, when the computer program is executed by the processor, the following The vehicle control method described in one aspect or any optional implementation thereof.

In a fourth aspect, there is provided a vehicle, including: the vehicle control device according to the second aspect or any optional implementation thereof, or the vehicle-mounted device according to the third aspect.

In a fifth aspect, there is provided a computer-readable storage medium, including: storing a computer program on the computer-readable storage medium, and when the computer program is executed by a processor, the first aspect or any optional implementation thereof can be implemented. The vehicle control method described in the manner.

According to a sixth aspect, there is provided a computer program product, which is characterized by comprising: when the computer program product is run on a computer, the computer is enabled to realize the above-mentioned computer program described in the first aspect or any optional implementation manner thereof. vehicle control method.

Compared with the prior art, the technical solutions provided by the embodiments of the present disclosure have the following advantages: visual sensors are installed at the bottom and surroundings of the target vehicle to collect visual data under the chassis of the target vehicle, and synthesize the bottom of the target vehicle based on these visual data Based on the original image of the road condition recognition model, the original image is recognized to obtain the road condition information at the bottom of the target vehicle, and finally the driving of the target vehicle is controlled based on the road condition information. In this way, the road condition information at the bottom of the target vehicle is provided as a reference, and the driving process is improved. Accuracy of obstacle avoidance.

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description serve to explain the principles of the disclosure.

In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, In other words, other drawings can also be obtained from these drawings without paying creative labor.

FIG. 1 is a schematic diagram of an implementation scenario of a vehicle control method provided by an embodiment of the present disclosure;

FIG. 2 is a schematic flowchart of a vehicle control method provided by an embodiment of the present disclosure;

FIG. 3 is a schematic diagram of a scene provided by an embodiment of the present disclosure;

FIG. 4 is a schematic flow diagram of obtaining road condition information provided by an embodiment of the present disclosure;

FIG. 5 is a structural block diagram of a vehicle control device provided by an embodiment of the present disclosure;

FIG. 6 is a schematic structural diagram of a vehicle-mounted device provided by an embodiment of the present disclosure.

Detailed ways

In order to more clearly understand the above objects, features and advantages of the present disclosure, the solutions of the present disclosure will be further described below. It should be noted that, in the case of no conflict, the embodiments of the present disclosure and the features in the embodiments can be combined with each other.

In the following description, many specific details are set forth in order to fully understand the present disclosure, but the present disclosure can also be implemented in other ways than described here; obviously, the embodiments in the description are only some of the embodiments of the present disclosure, and Not all examples.

The terms "first" and "second" and the like in the specification and claims of the present disclosure are used to distinguish different objects, not to describe a specific order of objects. For example, the first information and the second information are used to distinguish different images, rather than describing a specific sequence of images.

The terms "comprising" and "having" and any variations thereof in the embodiments of the present invention are intended to cover a non-exclusive inclusion, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to expressly instead of those steps or elements listed, may include other steps or elements not explicitly listed or inherent to the process, method, product or apparatus.

It should be noted that, in the embodiments of the present invention, words such as "exemplary" or "for example" are used for example, illustration or description. Any embodiment or design solution described as "exemplary" or "for example" in the embodiments of the present invention shall not be construed as being more preferred or more advantageous than other embodiments or design solutions. Rather, the use of words such as "exemplary" or "such as" is intended to present related concepts in a concrete manner.

As a very important part of the driving process, obstacle avoidance is of great significance to the safety of vehicle driving. The obstacle avoidance technology mainly uses advanced sensor technology to enhance the car's perception of the driving environment, and the vehicle speed acquired by the perception system , location and other real-time information feedback to the system, and at the same time judge and analyze potential safety hazards based on the comprehensive information of road conditions and traffic flow, and remind the driver to avoid obstacles in an emergency, or automatically take alarm prompts, braking or steering and other measures to assist And control the car to actively avoid obstacles to ensure the safe, efficient and stable driving of the vehicle.

At present, road condition information cannot be provided for the road conditions at the bottom of the vehicle to assist in avoiding obstacles during driving. Therefore, there is an urgent need for a method of controlling vehicle driving through road condition information under the vehicle chassis to improve the accuracy of obstacle avoidance during driving.

In order to solve the above problems, an embodiment of the present disclosure provides a vehicle control method, in which visual sensors are arranged at the bottom and surroundings of the target vehicle to collect the visual data under the chassis of the target vehicle, and based on these visual data, synthesize the target vehicle bottom The original image, based on the road condition recognition model, recognizes the original image, obtains the road condition information at the bottom of the target vehicle, and finally controls the driving of the target vehicle based on the road condition information. In this way, since the road condition information at the bottom of the target vehicle is provided as a reference, the driving process is improved. Accuracy of obstacle avoidance.

In the embodiments of the present disclosure, the vision sensor may include one or more cameras. Optionally, the vision sensor may also include radar.

As shown in FIG. 1 , it is a schematic diagram of an implementation scenario of a vehicle control method provided by an embodiment of the present disclosure. As well as the surrounding visual sensors 112 , as shown in FIG. 1 , a plurality of visual sensors 112 may be provided at the bottom and surrounding of the vehicle 11 . Wherein, the vehicle-mounted device 111 is connected to the visual sensor 112, and the vehicle-mounted device 111 can obtain images or data collected by multiple visual sensors 112, and synthesize the original image of the bottom of the vehicle 11 according to these images or data, and obtain road condition information based on the original image , to assist in controlling the running of the vehicle 11.

The vehicle control method provided in the embodiments of the present disclosure may be implemented by a vehicle control device, a vehicle-mounted device, or a vehicle, wherein the vehicle control device may be a vehicle-mounted device or may be a functional module or function in a vehicle-mounted device for implementing the vehicle control method entity.

Exemplarily, for a new energy vehicle (such as an electric vehicle), the vehicle control device may be a vehicle control unit (Vehicle control unit, VCU), and the VCU is a core electronic control unit for implementing vehicle control decisions. The VCU judges the driver's driving intention by collecting signals such as the accelerator pedal, gear position, and brake pedal; by monitoring the vehicle status (such as vehicle speed, temperature, etc.) The power battery system sends the vehicle's running state control commands, and at the same time controls the working mode of the vehicle's accessory power system; the VCU has the functions of fault diagnosis, protection and storage of the vehicle system. The VCU judges the driver's driving intention by collecting signals such as the accelerator pedal, gear position, and brake pedal; by monitoring the vehicle status (vehicle speed, temperature, etc.), the VCU sends the vehicle's running status to the power system and power battery system State control command, while controlling the working mode of the power system of the vehicle accessories; VCU has the function of fault diagnosis protection and storage of the vehicle system. The main functions of the VCU include: receiving and processing the driver's driving operation instructions, and sending control instructions to the controllers of each component, so that the vehicle can drive according to the driver's expectations; Reliable communication with nickel-metal hydride battery packs, as well as state acquisition input and control instruction output for analog quantities of key information; the vehicle controller provides signal channels for direct control of corresponding components, including digital-analog (D/A ) conversion and digital output, etc.; receiving and processing the information of each component, combined with the energy management unit to provide current energy status information; judging and storing system faults, dynamically detecting system information, and recording faults; it has protection functions for the entire vehicle, According to the type of fault, the whole vehicle is protected in different levels. In an emergency, the generator can be turned off and the high-voltage system of the busbar can be cut off. The development of the vehicle controller includes software and hardware design. The core software is generally developed by the OEM, and the hardware and underlying driver software can be provided by the auto parts manufacturer.

Among them, the vehicle-mounted equipment can be an intelligent multimedia device integrated in the car center console, which can usually become an In-Vehicle Infotainment (IVI) system, also known as HU. With the evolution of related vehicle hardware and software and the form of car networking services Continuous innovation, the current IVI system has gradually covered navigation, music, video, voice recognition, telephone, information interaction and other content. At the same time, in terms of the evolution of the system architecture of the IVI system in the car, hardware integration and software interoperability are gradually emerging between the IVI system, body electronics, and Advanced Driver Assistance System (Advanced Driver Assistance System, ADAS) and other systems.

As shown in FIG. 2, it is a schematic flowchart of a vehicle control method provided by an embodiment of the present disclosure, the method includes:

201. Acquire visual data of a target vehicle.

Wherein, the above-mentioned visual data of the target vehicle includes an image of the bottom of the target vehicle and a surround-view image of the target vehicle, and the above-mentioned visual data may be collected by a visual sensor.

The visual sensor may include multiple cameras, and the visual data may be images collected by the multiple cameras. The vision sensor may also include radar, and the vision data may also include data detected by the radar.

In some embodiments, the vision data collected by the vision sensor may be acquired when the target vehicle is in a driving state. That is to say, before performing the above step 201, it needs to be determined that the target vehicle is in a driving state.

Wherein, it can be judged whether the target vehicle is in the driving state by the vehicle speed of the current target vehicle. When the vehicle speed of the current target vehicle is not 0, it is determined that the target vehicle is in the driving state; when the vehicle speed of the current target vehicle is 0, it is determined that the target vehicle is in the parking state. car status.

Because when the vehicle is in the driving state, it is necessary to know the road condition information under the current vehicle as reference information for obstacle avoidance in the assisted driving process, and when the vehicle is in the parking state, it is usually not necessary to obtain road condition information, so In the embodiments of the present disclosure, when the target vehicle is in a driving state, the visual data collected by the visual sensor can be obtained, which can avoid real-time collection of visual data by the visual sensor, so that the vehicle-mounted device has a large power consumption.

202. Synthesize the original image of the bottom of the target vehicle according to the visual data of the target vehicle.

Wherein, according to the visual data collected by the visual sensor, synthesizing the original image of the bottom of the target vehicle may include the following steps (a) to (c):

(a) Obtain the image data of the blind area below the bottom of the target vehicle through the cameras at the bottom or around the target vehicle.

In some embodiments, as shown in FIG. 4 , when the vehicle is in the driving state or in the parking state, the video stream data of the blind area under the bottom of the target vehicle can also be jointly acquired by using the surround view cameras around the target vehicle and the cameras at the bottom of the target vehicle. .

(b) Extract multiple key frame images from the blind area image data.

These blind spot image data may be video stream data. After the video stream data is acquired, as shown in FIG. 4 , multiple key frame images may be extracted from the video stream data through a key frame image extraction method.

(c) Image processing is performed on multiple key frame images to synthesize the original image of the bottom of the target vehicle.

As shown in Figure 4, multiple extracted key frame images can be processed through image distortion correction algorithm, image physical transformation algorithm, image key point detection and matching algorithm, and image stitching and cropping algorithm. The image under the chassis of the target vehicle is synthesized in real time (that is, the image of the original image spliced and cropped by the algorithm), and displayed on the vehicle device. Further, it can also detect whether the target vehicle is moving. When it is determined that the target vehicle is moving and is in a driving state, the real-time synthesized original image can be displayed in real time; when it is determined that the target vehicle is not moving and is in a driving state, it will not The update shows the original image composited in real time.

The above image distortion correction algorithm means that the original image we obtained is a distorted image, and to obtain an image without distortion, its mapping relationship needs to be deduced through a distortion model. Through the relationship between the real image and the distorted image, the coordinate position in the distorted image is calculated according to the coordinate position of the real image, and the corresponding pixel value is taken out, so that the pixel value of the real image can be obtained.

The above-mentioned image physical transformation algorithm may be processing such as rotation and scaling of the image. Further, the image physical transformation algorithm may be at least one of flipping, rotating, cropping, deforming, and scaling the image.

In the above image key point detection and matching algorithm, key points in an image can be detected, and image matching can be performed based on the key points.

Among them, the key point is a more abstract concept. For image processing, generally speaking, the points in the image that are more important to the analysis problem can be used as key points. When extracting key points, the edge of the object should be used as an important reference, but it must not be the only basis. For an object, the key point should be a point that expresses certain characteristics, not just an edge point. As long as it is helpful to analyze a specific problem, it can be called a key point. For example, after the image is classified and recognized, the points on the edge of the object can be used as key points. In the embodiment of the present disclosure, these points on the edge of the object may help to identify subsequent road condition information such as obstacles and potholes. The above-mentioned key points can also refer to the corner points in the image, and any of the following algorithms can be used to determine the key points in the image:

Harris (Harris) corner detection algorithm, Shi-Thomas (Shi-Tomasi) corner detection algorithm, accelerated segmentation test feature (FAST) corner detection algorithm.

Among them, image matching can include two methods of image matching based on grayscale and image matching based on features. Among them, there are many methods for image matching based on features, such as: Scale-invariant feature transform (Scale-invariant feature transform, SIFT ) algorithm, Speeded Up Robust Features (SURF) algorithm, Small univalue segment assimilating nucleus (SUSAN) algorithm, etc. for image matching.

Among them, the SIFT algorithm is used as a description in the field of image processing. This description has scale invariance and can detect key points in the image. It is a local feature descriptor. The SIFT feature is based on some local appearance interest points on the object independent of the size and rotation of the image. The tolerance to light, noise, and micro viewing angle changes is also quite high. Based on these characteristics, they are highly conspicuous and relatively easy to pick up, and objects are easily recognized with high accuracy. Using SIFT features to describe the detection accuracy of partial object occlusion is also quite high, and even more than 3 SIFT object features are enough to calculate the position and orientation. Under the conditions of current computer hardware speed and small feature database, the recognition speed can be close to real-time operation. The SIFT feature has a large amount of information and is suitable for quickly and accurately matching the above image stitching and cropping algorithms in a massive database.

Among them, SURF improves the feature extraction and description method, and uses a more efficient way to complete feature extraction and description compared to SIFT. The implementation process of SURF includes:

First construct the Hessian Matrix (Hessian Matrix) to generate all the points of interest for feature extraction. Among them, the Hessian Matrix is a square matrix composed of second-order partial derivatives of a multivariate function, which describes the local curvature of the function; then constructs the scale space , and then perform feature point positioning, which is to compare each pixel point processed by the Hessian Matrix with multiple points in the two-dimensional image space and scale space neighborhood, initially locate the key points, and then filter out the key points with weaker energy And the key points of wrong positioning, and screen out the final stable feature points; after that, the main direction of the feature points is assigned, that is, the harr wavelet in the circular neighborhood of the statistical feature points (it is a kind of Daubechies wavelet family) , the Dobesy wavelet is mainly used in discrete wavelet transform, which is the most commonly used wavelet transform, usually used in digital signal analysis, signal compression and noise removal) features. In the circular neighborhood of the feature points, count the sum of the horizontal and vertical harr wavelet features of all points in the 60-degree sector, then rotate the sector at a certain interval and count the harr wavelet feature values in the area again, and finally use the one with the largest value The direction of the fan is used as the main direction of the feature point, and finally the feature point descriptor is generated, and the feature point matching is performed according to the feature point descriptor. When performing feature point matching, the Euclidean distance between two feature points is calculated to determine the matching degree. The shorter the K-distance, the better the match between the two feature points.

Among them, the SUSAN algorithm is a method for processing grayscale images, which is mainly used to calculate the corner features in the image. The SUSAN algorithm is different from the square template of the conventional convolution algorithm. The circular template is selected, and the circular template is used to move on the image. The gray value of each image pixel inside the template is compared with the gray value of the central pixel of the template. If If the difference between the gray level of a certain pixel in the template and the gray level of the center pixel of the template is less than a certain value, it is considered that the point has the same (or similar) gray level as the nucleus. The area composed of pixels satisfying this condition is called the Univalue Segment Assimilating Nucleus (USAN), that is, the area whose brightness value is similar to the brightness of the core point is the USAN.

In the embodiment of the present disclosure, the original image of the bottom of the target vehicle can be synthesized in real time, and the original image can be updated and displayed in real time.

203. Input the original image into the road condition recognition model, and acquire the road condition information output by the road condition recognition model.

Wherein, the road condition recognition model is a model trained based on target training samples, and the target training samples include images of a plurality of vehicle bottoms and road condition information corresponding to each vehicle bottom image.

Optionally, the road condition information can be obtained when the target vehicle is determined to be in a driving state, that is, when the target vehicle is determined to be in a driving state, the original image is input to the road condition recognition model to obtain the road condition information output by the road condition recognition model .

The above road condition information includes but is not limited to one or more of the following (1) to (7):

(1) Obstacle location;

Wherein, the position of the obstacle may be the position of the obstacle relative to the target vehicle, specifically a reference point on the target vehicle may be set, and the position of the obstacle may be represented by the relative position of the obstacle from the reference point. Wherein, the reference point may be the center point of the chassis of the target vehicle.

By knowing the position of the obstacle and the direction of the current target vehicle, it is possible to know whether the obstacle will block the wheels of the target vehicle, so as to choose an appropriate way to control the target vehicle, such as slowing down, going straight, turning, etc.

(2) The size of the obstacle;

Wherein, the size of the obstacle may be the size of the area of the obstacle, or the size of the volume of the obstacle.

By knowing the size of the obstacle, you can know whether the obstacle will affect the driving of the target vehicle. For smaller obstacles, it may not affect the driving of the target vehicle, and there is no need to control the vehicle to turn around or slow down. If there are obstacles, it may be necessary to control the target vehicle to slow down or turn.

(3) The distance between the obstacle and the chassis of the target vehicle;

Exemplarily, as shown in FIG. 3 , it is a schematic diagram of a scene, and B in the scene shown in FIG. 3 represents the distance between the obstacle and the chassis of the target vehicle. It can be seen that the distance between the obstacle and the chassis of the target vehicle in the embodiments of the present disclosure may refer to the vertical distance between the obstacle and the chassis of the target vehicle.

By knowing the distance between the obstacle and the chassis of the target vehicle, it can be judged whether the target vehicle can normally cross the obstacle. When the distance between the obstacle and the chassis of the target vehicle is small, for example, when the distance is close to 0 or even a negative value, Then it can be determined that the target vehicle cannot normally cross the obstacle, there is a risk of scratching the chassis, and it needs to avoid.

(4) Obstacle type;

Wherein, the obstacle type may refer to an object type to which the obstacle belongs, such as a stone, a speed bump, a plastic bag, an animal, and the like.

By knowing the type of obstacle, it is also possible to determine whether the target vehicle crosses the obstacle, and whether it will block the wheels of the target vehicle, so as to choose an appropriate way to control the target vehicle, such as slowing down, going straight, turning, etc. Exemplarily, when it is determined that the type of obstacle is a speed bump, the vehicle will be controlled to decelerate; when it is determined that the type of obstacle is an animal, it can slow down, stop or turn, etc. to avoid colliding with the animal.

(5) Pothole location;

Wherein, the position of the pothole may be the position of the pothole relative to the target vehicle. Specifically, a reference point on the target vehicle may be set, and the position of the pothole may be represented by the relative position of the pothole from the reference point. The reference point may be the center point of the chassis of the target vehicle.

By knowing the position of the pothole and the direction of the current target vehicle, it is possible to know whether the pothole will affect the wheels of the target vehicle, so as to choose an appropriate way to control the driving of the target vehicle so that the wheels of the target vehicle avoid the potholes. For example, slow down, go straight, turn, etc.

(6) Pothole size;

Wherein, the size of the pothole may be the size of the surface area of the pothole on the road plane, or may be the size of the surface area of the pothole.

By knowing the size of the pothole, you can know whether the pothole will affect the driving of the target vehicle. For smaller potholes, it may not affect the driving of the target vehicle, and there is no need to control the vehicle to turn around or slow down. If there are large potholes, the wheels of the target vehicle may fall into the potholes. At this time, it is necessary to control the steering of the target vehicle so that the wheels of the target vehicle avoid the potholes.

(7) Pothole depth.

Exemplarily, as shown in FIG. 3 , it is a schematic diagram of a scene, and H in FIG. 3 represents the depth of potholes.

By knowing the depth of the pothole, you can know whether the pothole will affect the driving of the target vehicle. For shallow potholes, it may not affect the driving of the target vehicle, and there is no need to control the vehicle to turn around or slow down. If there are deep potholes, the wheels of the target vehicle may fall into the potholes. At this time, it is necessary to control the steering of the target vehicle so that the wheels of the target vehicle avoid the potholes.

Further, in addition to the above seven types of information, the above road condition information may also include information such as the distance (vertical distance) from the bottom of the pothole to the target vehicle. It should be noted that, the several types of information included in the above road condition information are only exemplary descriptions, and may also include other information, which is not limited by the embodiments of the present disclosure.

In some embodiments, the road condition recognition model includes: an obstacle recognition model and a distance measurement model.

The above-mentioned input of the original image to the road condition recognition model, and obtaining the road condition information output by the road condition recognition model may include the following steps 1) to 3):

1) Input the original image into the obstacle recognition model, and obtain the road condition image output by the obstacle recognition model.

Wherein, the road condition image is an image marked with first information on the original image, and the first information includes at least one of the following:

Obstacle location, obstacle size, obstacle type, pothole location, pothole size.

Wherein, the obstacle recognition model is a model trained based on the first training sample, which includes a plurality of images of the bottom of the vehicle, and a first image corresponding to each image of the bottom of the vehicle, the first image is the bottom of the vehicle An image marked with the first information on the image of .

2) input the road condition image to the distance measurement model, and obtain the second information output by the distance measurement model;

Wherein, the second information includes at least one of the following:

The distance between the obstacle and the chassis of the target vehicle, and the depth of the pothole.

Wherein, the distance measurement model may be a model of a monocular distance measurement algorithm, and the monocular distance measurement algorithm is also called a monocular distance measurement algorithm, or a monocular vision distance measurement algorithm.

The distance measurement model is a model obtained by training based on the second training sample, which includes a plurality of first images and the second information corresponding to each first image.

3) The first information and the second information are used as road condition information.

The road condition information can be obtained by combining the first information obtained by the obstacle recognition model and the second information obtained by the distance measurement model.

In some embodiments, the road condition recognition model may include: an obstacle recognition model, that is, only the first information can be obtained through the road condition recognition model, and at this time, the first information may also be used as the road condition information.

In some embodiments, after the original image is input into the road condition recognition model and the road condition information output by the road condition recognition model is obtained, a road condition display image can also be generated based on the traffic condition information and the original image, and the road condition display image can be displayed. The image can display traffic information in real time.

204. Control the target vehicle to travel based on the road condition information.

In some embodiments, when it is determined that there is a recognized obstacle based on the road condition information, it can be comprehensively judged whether it is necessary to slow down or turn to avoid danger based on the location, size, and type of the obstacle. Similarly, when it is determined that there is a pothole based on the road condition information, you can slow down, and according to the depth of the pothole, the position of the pothole, and the size of the pothole, determine the steering to avoid obstacles and avoid danger.

For example, when an obstacle of type stone is detected directly in front of the wheel of the target vehicle, the size of the obstacle is judged, and if the size of the obstacle exceeds the preset size, the vehicle is controlled to turn around so that the wheel will not be blocked by the stone .

Exemplarily, when it is detected that there is a relatively shallow pothole directly in front of a wheel of the target vehicle, the target vehicle is controlled to slow down to avoid bumping of the target vehicle.

Exemplarily, when it is detected that there is a deep pothole directly in front of the wheels of the target vehicle, the target vehicle is controlled to turn around so that the wheels will not be selected into the pothole.

In the vehicle control method provided by the embodiments of the present disclosure, a visual sensor is installed at the bottom of the target vehicle to collect visual data under the chassis of the target vehicle, and based on these visual data, the original image of the bottom of the target vehicle is synthesized, and based on the road condition recognition model, Identify the original image, get the road condition information at the bottom of the target vehicle, and finally control the driving of the target vehicle based on the road condition information, thus improving the accuracy of obstacle avoidance during driving because the road condition information at the bottom of the target vehicle is provided as a reference.

As shown in FIG. 5 , an embodiment of the present disclosure provides a vehicle control device, the vehicle control device is connected to visual sensors arranged at the bottom and around the target vehicle, and the device includes:

The synthesis module 501 is configured to synthesize the original image of the bottom of the target vehicle according to the visual data of the target vehicle, wherein the visual data includes an image of the bottom of the target vehicle and a surround view image of the target vehicle;

The acquiring module 502 is configured to input the original image into the road condition recognition model, and acquire the road condition information output by the road condition recognition model;

A control module 503, configured to control the target vehicle to travel based on the road condition information;

Wherein, the road condition recognition model is a model trained based on target training samples, and the target training samples include images of a plurality of vehicle bottoms and road condition information corresponding to each vehicle bottom image.

As an optional implementation manner of an embodiment of the present disclosure, the road condition information includes at least one of the following:

The position of the obstacle, the size of the obstacle, the distance between the obstacle and the chassis of the target vehicle, the type of the obstacle, the position of the pothole, the size of the pothole, and the depth of the pothole.

As an optional implementation manner of the embodiment of the present disclosure, the road condition recognition model includes: an obstacle recognition model and a distance measurement model;

The obtaining module 502 is specifically configured to: input the original image into the obstacle recognition model, and obtain a road condition image output by the obstacle recognition model, and the road condition image is marked with the first information on the original image image, the first information includes at least one of the following:

Obstacle location, obstacle size, obstacle type, pothole location, pothole size;

The road condition image is input into the distance measurement model, and the second information output by the distance measurement model is obtained, and the second information includes at least one of the following:

The distance from the obstacle to the chassis of the target vehicle, the depth of potholes;

The first information and the second information are used as the road condition information.

As an optional implementation of the embodiment of the present disclosure, the obstacle recognition model is a model trained based on the first training sample, the first training sample includes images of multiple vehicle bottoms, and each vehicle bottom The image corresponding to the first image is an image marked with the first information on the image of the bottom of the vehicle.

As an optional implementation manner of this embodiment of the present disclosure, the distance measurement model is a model trained based on a second training sample, the second training sample includes a plurality of the first images, and each first The second information corresponding to the image.

As an optional implementation manner of the embodiment of the present disclosure, the acquiring module 502 is further configured to:

determining that the target vehicle is in a driving state; and displaying the original image after synthesizing the original image of the bottom of the target vehicle according to the visual data of the target vehicle.

As an optional implementation manner of the embodiment of the present disclosure, the acquiring module 502 is further configured to:

Said inputting the original image into the road condition recognition model, after acquiring the road condition information output by the road condition recognition model, generating a road condition display image based on the road condition information and the original image; displaying the road condition display image.

As shown in FIG. 6 , an embodiment of the present disclosure provides a vehicle-mounted device, the vehicle-mounted device includes: a processor 601, a memory 602, and a computer program stored in the memory 602 and operable on the processor 601. When the computer program is executed by the processor 601, various processes of the vehicle control method in the above method embodiments are realized. And can achieve the same technical effect, in order to avoid repetition, no more details here.

An embodiment of the present invention provides a computer-readable storage medium, which is characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, each process of the vehicle control method in the above method embodiment is implemented, and The same technical effect can be achieved, so in order to avoid repetition, details will not be repeated here.

An embodiment of the present disclosure also provides a vehicle. The vehicle may include the above-mentioned vehicle control device or the above-mentioned on-board equipment. In addition, the vehicle may also include a visual sensor arranged at the bottom of the vehicle. Exemplarily, the vehicle may be as shown in Figure 1 Vehicle 11 is shown.

Wherein, the computer-readable storage medium may be a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, and the like.

An embodiment of the present invention provides a computer program product, the computer program product stores a computer program, and when the computer program is executed by a processor, each process of the vehicle control method in the above method embodiment can be achieved, and the same technical effect can be achieved. Repeat, no more details here.

Those skilled in the art should understand that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Accordingly, the present disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present disclosure may take the form of a computer program product embodied on one or more computer-usable storage media having computer-usable program code embodied therein.

In the present disclosure, the processor may be a central processing unit (Central Processing Unit, CPU), and may also be other general-purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC) , off-the-shelf programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, or the like.

In the present disclosure, memory may include non-permanent memory in computer-readable media, random access memory (RAM) and/or non-volatile memory, such as read-only memory (ROM) or flash memory (flash RAM). The memory is an example of a computer readable medium.

In this disclosure, computer-readable media includes both volatile and non-volatile, removable and non-removable storage media. The storage medium may store information by any method or technology, and the information may be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Flash memory or other memory technology, Compact Disc Read-Only Memory (CD-ROM), Digital Versatile Disc (DVD) or other optical storage, A magnetic tape cartridge, disk storage or other magnetic storage device or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media excludes transitory computer-readable media, such as modulated data signals and carrier waves.

It should be noted that in this article, relative terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply these No such actual relationship or order exists between entities or operations. Furthermore, the term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus comprising a set of elements includes not only those elements, but also includes elements not expressly listed. other elements of or also include elements inherent in such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

The above are only specific implementation manners of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to these embodiments herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims (10)

1. A vehicle control method characterized by comprising:

synthesizing an original image of the bottom of the target vehicle according to visual data of the target vehicle, wherein the visual data comprises the image of the bottom of the target vehicle and a looking-around image of the target vehicle;

inputting the original image into a road condition recognition model to acquire road condition information output by the road condition recognition model;

controlling the target vehicle to run based on the road condition information;

the road condition recognition model is a model obtained based on training of a target training sample, wherein the target training sample comprises a plurality of images of the bottoms of vehicles and road condition information corresponding to the images of the bottoms of each vehicle.

2. The method of claim 1, wherein the road condition recognition model comprises: an obstacle recognition model and a distance measurement model;

the step of inputting the original image into a road condition recognition model to obtain the road condition information output by the road condition recognition model comprises the following steps:

inputting the original image into the obstacle recognition model, and acquiring a road condition image output by the obstacle recognition model, wherein the road condition image is an image marked with first information on the original image, and the first information comprises at least one of the following:

Obstacle position, obstacle size, obstacle type, pothole position, pothole size;

inputting the road condition image into the distance measurement model, and acquiring second information output by the distance measurement model, wherein the second information comprises at least one of the following components:

the distance of the obstacle from the chassis of the target vehicle and the depth of the pit;

and taking the first information and the second information as the road condition information.

3. The method of claim 2, wherein the step of determining the position of the substrate comprises,

the obstacle recognition model is a model obtained based on training of a first training sample, the first training sample comprises a plurality of images of the bottoms of the vehicles and first images corresponding to the images of the bottoms of the vehicles, and the first images are images marked with the first information on the images of the bottoms of the vehicles.

4. The method of claim 3, wherein the step of,

the distance measurement model is a model obtained based on training of a second training sample, wherein the second training sample comprises a plurality of first images and the second information corresponding to each first image.

5. The method of claim 1, wherein the method further comprises, prior to synthesizing the original image of the bottom of the target vehicle from the visual data of the target vehicle:

Determining that the target vehicle is in a driving state;

after synthesizing the original image of the bottom of the target vehicle according to the visual data of the target vehicle, the method further comprises:

and displaying the original image.

6. The method of claim 1, wherein the inputting the original image into the road condition recognition model, and after obtaining the road condition information output by the road condition recognition model, the method further comprises:

generating a road condition display image based on the road condition information and the original image;

and displaying the road condition display image.

7. A vehicle control apparatus characterized by comprising:

the synthesis module is used for synthesizing an original image of the bottom of the target vehicle according to visual data of the target vehicle, wherein the visual data comprises the image of the bottom of the target vehicle and a look-around image of the target vehicle;

the acquisition module is used for inputting the original image into a road condition recognition model and acquiring road condition information output by the road condition recognition model;

the control module is used for controlling the target vehicle to run based on the road condition information;

the road condition recognition model is a model obtained based on training of a target training sample, wherein the target training sample comprises a plurality of images of the bottoms of vehicles and road condition information corresponding to the images of the bottoms of each vehicle.

8. An in-vehicle apparatus, characterized by comprising: a processor, a memory and a computer program stored on the memory and executable on the processor, which when executed by the processor implements the vehicle control method according to any one of claims 1 to 6.

9. A vehicle, characterized by comprising: the vehicle control apparatus according to claim 7, or the in-vehicle device according to claim 8.

10. A computer-readable storage medium, comprising: the computer-readable storage medium stores thereon a computer program which, when executed by a processor, implements the vehicle control method according to any one of claims 1 to 6.