DE102021004426A1 - Method for training an autonomous driving function - Google Patents

Abstract

The invention relates to a method for training an autonomous driving function. According to the invention, at least one vehicle is driven by a human driver classified as predefined safe, with data relating to the surroundings of the vehicle, driving dynamics data and driving commands (FB) from the driver being recorded and collected, with the recorded and a reward function (BF) is learned through Inverse Reinforcement Learning (IRL) and collected data, and the autonomous driving function is trained by means of the learned reward function (BF) through Reinforcement Learning (RL).

Description

The invention relates to a method for training an autonomous driving function.

From the prior art, as in the DE 10 2019 122 829 A1 described, a vehicle action control known. One or more target areas in the vicinity of a moving vehicle are identified. The vehicle is maneuvered to a target area selected according to a reinforcement learning reward function.

The invention is based on the object of specifying a novel method for training an autonomous driving function.

The object is achieved according to the invention by a method for training an autonomous driving function with the features of claim 1.

Advantageous refinements of the invention are the subject matter of the subclaims.

In a method for training an autonomous driving function, according to the invention, at least one vehicle is driven by a human driver classified as predefined safe, data on the surroundings of the vehicle, driving dynamics data and driving commands from the driver being recorded and collected, using the recorded and collected data by inverse reinforcement Learning a reward function is learned, and the autonomous driving function is trained by means of the learned reward function through reinforcement learning.

The solution according to the invention solves the problem that, due to the complexity of the driving task and a combinatorial amount of potential situations, it is not possible to program the autonomous driving function in a conventional manner, for example using behavioral cloning methods or, for example, exclusively using reinforcement learning, because the behavior an autonomous vehicle, in particular its autonomous driving function, is so complex that it cannot be statically programmed and unlimited amounts of training data cannot be used for training using neural networks.

Behavioral cloning processes learn to copy human decisions based on the input data and are therefore not able to scale to unknown situations. Classic reinforcement learning is a well-known approach to learning driving tasks. In contrast to the inverse reinforcement learning additionally used in the solution according to the invention, an agent, in this case the autonomous vehicle, more precisely its autonomous driving function, learns its decisions based on a predetermined reward function through its own experience. Due to the experience required for a reasonable performance, these agents have to be trained mainly in simulations. In addition to this, reward functions have to be continuously adapted, since modeling the driving task by means of a reward function is highly non-linear. The advantages of using environment detection and driving commands from a driver remain untapped when using classic reinforcement learning. Furthermore, the simulation has a gap, a so-called sim2real gap, to the real world, so that agents from the simulation cannot be used in the real world without additional experience. This can mean that a reward function in reality does not achieve the same performance as in the simulation.

These disadvantages of using only reinforcement learning are eliminated by the solution according to the invention, in particular by the combination of reinforcement learning and inverse reinforcement learning.

The vehicle used for the method has, in particular, sufficient software and hardware for perceiving the surroundings. Several vehicles and / or several drivers are advantageously used for the method. In addition, in particular a model for the abstract representation of the surroundings of the vehicle is provided for the method. For this purpose, there is in particular a preprocessing of data relating to the surroundings. The model is, for example, an occupancy grid.

For example, an algorithm is also provided which can extract features from the representation of the surroundings, for example an auto-encoder.

As described, the method comprises the inverse reinforcement learning algorithm for learning the reward function and the reinforcement learning algorithm, which can be trained using the reward function determined by the inverse reinforcement learning algorithm; more precisely, the autonomous driving function is trained.

Acquired data for the method are cached in particular in the vehicle. The vehicle has a sufficient buffer for this purpose. The recorded data is sent in particular to a backend, ie to a server external to the vehicle. The vehicle thus has a corresponding connection to the backend and hardware on. The data is sent in particular via a cellular connection.

Advantageously, the method comprises a method to ensure the security of the autonomous agent, i. H. the autonomous driving function. In particular, to evaluate the safety of the autonomous driving function, the performance of the autonomous driving function is compared with the performance of a human expert.

By combining the environment model, driving commands from the driver and inverse reinforcement learning, the method according to the invention can be used to learn a reward function based on the ambient conditions and the decisions / commands made by the driver. In contrast to classic reinforcement learning, which uses an already existing heuristic and selects the best action based on it, inverse reinforcement learning learns the heuristic through the decisions of the driver. As a result, the decision-making planning of the vehicle, in particular the autonomous driving function, can be learned much more efficiently and better. The calculated reward function is then used to train the reinforcement learning algorithm in order to achieve better behavior planning and trajectory planning. Since only the heuristics of human behavior are learned and a separate optimization process takes place, the algorithm can exceed the capabilities of a person with a sufficient amount of data by optimizing the reward function.

Embodiments of the invention are explained in more detail below with reference to drawings.

• 1 schematisch ein Ablaufdiagramm des Reinforcement Learning,
• 2 schematisch einen Softwarestack eines autonomen Fahrzeugs,
• 3 schematisch eine Inverse Reinforcement Learning Pipeline,
• 4 schematisch ein Occupancy Grid,
• 5 schematisch ein Training einer autonomen Fahrfunktion, und
• 6 schematisch ein Inverse Reinforcement Learning Verfahren.

Show:

• 1 a schematic flow diagram of reinforcement learning,
• 2 schematically a software stack of an autonomous vehicle,
• 3 schematically an inverse reinforcement learning pipeline,
• 4th schematically an occupancy grid,
• 5 schematically a training of an autonomous driving function, and
• 6th schematically an inverse reinforcement learning process.

Corresponding parts are provided with the same reference symbols in all figures.

Based on 1 until 6th In the following, a method for training an autonomous driving function is described, which can then be implemented in one or more vehicles. The respective vehicle, which this autonomous driving function and other necessary components for the implementation of the autonomous driving operation, for example a surroundings detection sensor system 3 , is then designed to carry out the autonomous driving mode.

Due to a reinforcement learning RL used for the method described here, the autonomous driving function is also used as a reinforcement learning agent 1 designated. Reinforcement Learning RL is also referred to as reinforcement learning or reinforcement learning, but the technical term Reinforcement Learning RL, which is known for this purpose, will continue to be used in the following.

Reinforcement Learning RL is an area of machine learning, ie machine learning that deals with it, like intelligent agents 1 Actions A in an environment 2 perform to maximize a reward B, also known as a reward, as in 1 shown schematically by means of a flow chart. The agent 1 learns a strategy S, also referred to as a policy, in order to select and carry out actions A according to the state of the environment and the reward B. To this end, he conducts an observation BA of his surroundings 2 by. The reward function BF corresponds to a kind of “label” and gives the agent 1 feedback on his behavior, for example by rewarding a good result, ie a good action A, with a high reward B and a bad result, ie a bad action A, being punished with a low or negative reward B. The reward function BF is usually determined by experts and can be improved through an iterative process. The agent's strategy S 1 is mostly learned through a neural network.

In the method described here, both this reinforcement learning RL and an inverse reinforcement learning IRL, i. H. inverse reinforcement learning or inverse reinforcement learning, also known as apprenticeship learning. In the following, the well-known technical term Inverse Reinforcement Learning IRL will continue to be used for this.

While in reinforcement learning RL a strategy S is to be learned with an already given reward function BF, inverse reinforcement learning IRL is motivated by the reverse process. The agent 1 , in this case mostly a human expert, executes actions A based on his strategy S. With the help of the given actions A and a model of the environment 2 a BF reward function can then be learned.

The human strategy S reflects what is known as “soft optimal” behavior, since it can be assumed that people behave close to the optimum in many areas, for example when driving a vehicle. The learned reward function BF reflects this optimal or almost optimal behavior and can be used to train a new reinforcement learning agent 1 , can thus be used in the method described here for training the autonomous driving function. The reinforcement learning agent 1 , here the autonomous driving function, is trained using the learned reward function BF. Through this process the Reinforcement Learning Agent 1 , here the autonomous driving function, thus being trained with the optimal or at least almost optimal reward function BF and through additional experience not only achieve the performance of the human expert, but even surpass human performance.

2 shows a software stack of an autonomous vehicle, in particular the autonomous driving function of the vehicle. A software stack, also known as a software stack, is a series of software components that build on one another and together form a platform. The software stack of the autonomous driving function usually consists of the in 2 shown blocks behavior V, prediction P, localization L, trajectory planning T, sensor fusion SF and motion control BS. The solution described here is limited to the areas of prediction P, behavior V and trajectory planning T. The basic idea behind it is that people usually solve these decisions in a kind of "end-to-end" approach, i.e. trajectory planning T using human prediction P takes place in one step based on the detected environmental conditions.

Thanks to advances in neural networks, algorithms in image processing already achieve better results than humans in a direct comparison. Due to the progress made in recent years, the recognition of the surroundings is no longer considered to be the greatest limiting factor for the implementation of autonomous driving. In contrast, the prediction P of the behavior of other road users and the behavior planning of the autonomously driving vehicle represent the greatest challenge for autonomous driving.

Because due to the high number of so-called edge cases, i. H. seldom occurring driving situations, it is not possible to drive vehicles, more precisely their autonomous driving function, through traditional feature engineering, i. H. To program by processing raw data in such a way that they can be processed directly by machine learning algorithms, so-called deep learning is usually also used in this area. H. deep neural networks. These methods scale with a number of driving situations seen and are therefore dependent on a high variance of data in order to achieve good performance. Because of the advantages in the area of data labeling, especially the low cost, and the low induced prior, i. H. Priority distribution, compared to so-called supervised learning methods, d. H. of supervised learning, Reinforcement Learning RL procedures are mostly used.

Reinforcement learning algorithms are trained by means of a reward function BF by collecting personal experience, i. H. the behavior V of the vehicle, more precisely the autonomous driving function, is adapted until the reward function BF is optimized. The problem here is that the reward function BF is set in advance by experts and therefore does not represent an optimized reward function BF.

Since human driving behavior is considered to be close to the optimum and distractions while driving represent the greatest reason for accidents, a safety-conscious driver is advantageously chosen as the benchmark. For example, safer drivers are classified using methods such as anomaly detection. These drivers and their respective vehicles are then used to collect data for training inverse reinforcement learning algorithms, as in 3 shown.

In addition, algorithms for recognizing the surroundings and the driving status are used to determine the current surroundings 2 and driving situation to be included in the optimization. Overall, a reward function BF can thereby be learned based on the experience of human experts. This is then used for training a reinforcement learning agent 1 , ie the autonomous driving function, and thereby enables better results to be achieved than pure reinforcement learning algorithms. Advantages are achieved above all in the case of the edge cases mentioned above.

In order to be able to use the advantages of Inverse Reinforcement Learning IRL, data on the current state of the vehicle, for example driving dynamics data such as accelerations, speed, yaw angle, which are determined by means of a corresponding driving dynamics sensor system 4th of the vehicle are detected, as well as travel commands FB by the driver, for example steering angle and a position of an accelerator pedal and / or brake pedal, and the environment 2 , ie the surroundings of the vehicle, for example static and dynamic objects and the Localization L of the vehicle, be known. I. E. This data is recorded for the procedure described here. The data on the surroundings of the vehicle are collected by means of its surroundings detection sensors 3 recorded.

These data are collected in order to ensure that a reward function BF is learned by means of the data points, that is to say by means of these data. 3 presents a pipeline for collecting the data, in which the data from various sensors in the vehicle are first preprocessed in order not to make the algorithm dependent on individual sensors, ie to optimize it only for one sensor class. Without this preprocessing, the learned reward function BF possibly generalizes worse to other vehicles with different sensor configurations and / or far more data would be required for the determination of a reward function BF.

As in 3 shown, an environment model UM can, for example, on an in 4th Occupancy Grid OG shown as an example can be reduced. An occupancy grid OG is an abstraction level of the environment model UM, which makes it possible to store an occupancy of cells relative to the autonomous vehicle. By means of the Occupancy Grid OG, the world around the vehicle is displayed in the area of sensor illumination, ie a detection area of the surroundings detection sensors 3 of the vehicle, discretized. Each cell can be marked as unoccupied or occupied. By means of a classification, a type of a respective recognized object, for example a pedestrian, car or cyclist, can also be stored.

By working on this model, a decoupling from the sensor set, ie from the environment detection sensors 3 , of the vehicle, so that it is irrelevant whether the respective vehicle operated with the autonomous driving function is equipped with lidar sensors or not, and how many cameras or radar sensors have been installed, as long as the respective vehicle has a sufficiently good environment detection in three-dimensional space enables, d. h a sufficient environment detection sensors 3 having.

The features of the Occupancy Grid OG, also known as features, can optionally be compressed by means of an architecture of the Autoencoder AE type in order to ensure the correct data dimensions for the input into the Inverse Reinforcement Learning IRL method.

The signals from the vehicle dynamics sensors are also used for the same reason 4th , ie the driving dynamics data, preprocessed in order to decouple them from the relevant sensor set, ie from the driving dynamics sensors 4th of the vehicle. By means of a method for driving dynamics estimation, for example by means of a Kalman filter KF or a type of Kalman filter for determining driving dynamics values FD, measurement inaccuracies of the driving dynamics sensors can also be detected 4th be intercepted, ie filtered out, and the estimation of the driving state can be improved. Optionally, other, unmeasured, variables such as sideslip angle, slip angle, slip stiffness or coefficient of friction can also be estimated and help improve the model. The result is the determined driving dynamics values FD, which flow into the inverse reinforcement learning IRL as actions or strategy SFa of the driver, ie are used as input for this. The reward function BF is then determined by means of the inverse reinforcement learning IRL, ie this is the output of the inverse reinforcement learning IRL.

For this purpose, the data of the environment model UM and the determined signals of the driving status as well as the driving commands FB, d. H. the driver's strategy SFa, to a backend, d. H. to a server external to the vehicle, because the inverse reinforcement learning IRL is carried out outside the vehicle and advantageously using this information from several vehicles, advantageously from a large number of vehicles, in particular also with a number of drivers classified as safer, in particular with a large number of such drivers . In addition, the vehicle or the respective vehicle is equipped with a sufficiently large memory to temporarily store the recorded data. This allows the algorithm, i.e. H. To continuously improve the Inverse Reinforcement Learning IRL using new data.

Using the preprocessed data, consisting of the environment model UM, the driving situation and commands from the driver, ie the driver's strategy SFa, the reward function BF is now learned using the inverse reinforcement learning IRL. This is again in 5 shown.

This is where the reinforcement learning agent works 1 , ie the autonomous driving function, trained by the learned reward function BF. The environment model UM and the driver's strategy SFa thus flow into the inverse reinforcement learning IRL, which is used to determine the reward function BF, which flows into the reinforcement learning RL, into which the environment model UM also flows. The result of the reinforcement learning RL is then the strategy S of the reinforcement learning agent 1 , ie the autonomous driving function.

The trained reinforcement learning agent 1 , ie the trained autonomous driving function, can then take on the tasks of prediction P, behavior V and trajectory planning T. To optimize the method, modern inverse reinforcement learning IRL approaches can also be used, for example guided cost learning, which make it possible to learn a strategy S for the autonomous driving function and thus for the autonomous vehicle during the inverse reinforcement learning IRL process. This strategy S can then be optimized by a reinforcement learning RL process. One principle of this procedure is in 6th shown. The method allows a required computing power to be reduced, since a learned strategy S is already available at the beginning of the reinforcement learning RL process and this is further optimized.

As in 6th As shown, an initial strategy IS is used first. A strategy example SB is generated from this. Using this example strategy SB and the driver's strategy SFa, an update of the reward UB is generated. This results in the learned reward B. The agent's learned strategy S results from the update of the reward UB and the strategy example SB 1 , ie the autonomous driving function. In this way, the reward B and the strategy S of the autonomous agent can thus simultaneously 1 , ie the autonomous driving function, can be learned.

Inverse reinforcement learning IRL is better suited for the complex application of autonomous driving than methods of the behavioral cloning type. Behavioral cloning processes only learn to copy human behavior and are therefore not able to react to situations that have not yet been seen and thus not learned. In contrast to this, an inverse reinforcement learning IRL method can reflect the heuristics of human behavior through the reward function BF and train a reinforcement learning RL approach. This reinforcement learning RL approach can now be trained using the learned reward function BF. The reinforcement learning agent 1 , ie the autonomous driving function, on the one hand, can be trained on the entered data, since the environment model UM is available, which can be used to simulate the driving behavior. On the other hand, the reinforcement learning agent 1 , ie the autonomous driving function, but can also be trained in synthetically generated simulation scenarios to determine the performance of the agent 1 , ie the autonomous driving function, to be further improved by means of new situations. Optionally, the recorded data can also be used to model an additional error relating to human driving behavior. The combined approach of Inverse Reinforcement Learning IRL and Reinforcement Learning RL can guarantee a significantly better generalization in situations that have not yet been seen than approaches of the behavioral cloning type.

wobei:

• E der Umgebung 2 entspricht,
• π der Strategie S oder SFa entspricht,
• R(s_t) der Belohnung B des Zustand s zum Zeitschritt t entspricht,
• γ^t einer Discount Rate zum Zeitschritt t entspricht.

To ensure the agent's performance 1 , ie the autonomous driving function, a so-called loss function can be used, which allows a comparison between the learned strategy S and strategy SFa of human behavior, ie the driver. For this, the value of a strategy is calculated: $${{\displaystyle V}}_r^{\pi }={\mathrm{E.}}_{\pi }\left[{\displaystyle {\sum }_{t=0}^{\infty }{\gamma }^t\ast \mathrm{R.}\left(s_t\right)}\right]$$

whereby:

• E the environment 2 is equivalent to,
• π corresponds to strategy S or SFa,
• R (s _t ) corresponds to the reward B of the state s at the time step t,
• γ ^t of a discount rate corresponding to the time step t.

wobei:

• ${{\displaystyle V}}_r^{\pi \ast }$
  
  dem Value der optimalen (menschlichen) Strategie π*, d. h. der Strategie SFa des Fahrers, entspricht,
• ${{\displaystyle V}}_r^{\mathrm{\pi }robot}$
  
  dem Value der erlernten Strategie S des Reinforcement Learning Agenten πrobot, d. h. der autonomen Fahrfunktion, entspricht, und
• ε einem Threshold, d. h. einem Schwellwert, für die Differenz zwischen optimaler Strategie SFa und erlernter Strategie S entspricht.

This function can be used to set a condition for the Loss strategy: $${\mathrm{L.}}_{\mathrm{S.}trateGie}={{\displaystyle V}}_r^{\pi \ast }-{{\displaystyle V}}_r^{\mathrm{\pi }\text{robot}}\le \epsilon$$

whereby:

• ${{\displaystyle V}}_r^{\pi \ast }$
  
  corresponds to the value of the optimal (human) strategy π *, i.e. the driver's strategy SFa,
• ${{\displaystyle V}}_r^{\mathrm{\pi }rObOt}$
  
  corresponds to the value of the learned strategy S of the reinforcement learning agent πrobot, ie the autonomous driving function, and
• ε corresponds to a threshold, ie a threshold value, for the difference between the optimal strategy SFa and the learned strategy S.

Based on the above condition of strategy loss, recorded data can be used to compare the performance of the human expert with that of the reinforcement learning agent 1 , ie the autonomous driving function. In this way, a statement can be made about the safety of the method and thus the autonomous driving function. Depending on this, safety requirements for the approval of the algorithm and thus the autonomous driving function for a closed test route, for a test vehicle in traffic, which is monitored by a trained safety driver, and for an entire vehicle fleet can be specified. In addition, the strategy loss can be calculated in different scenarios in order to limit the function to certain traffic scenarios, for example on motorway journeys.

QUOTES INCLUDED IN THE DESCRIPTION

This list of the documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent literature cited

• DE 102019122829 A1 [0002]

Claims (4)

Method for training an autonomous driving function, characterized in that - at least one vehicle is driven by a human driver classified as predetermined safe, - data of the surroundings of the vehicle, driving dynamics data and driving commands (FB) of the driver are recorded and collected, - by means of the recorded and collected data through Inverse Reinforcement Learning (IRL) a reward function (BF) is learned, - the autonomous driving function is trained by means of the learned reward function (BF) through Reinforcement Learning (RL). Procedure according to Claim 1 , characterized in that the data are preprocessed before learning the reward function (BF). Method according to one of the preceding claims, characterized in that the data is sent from the vehicle to a server external to the vehicle by means of a mobile radio connection and the learning of the reward function (BF) by inverse reinforcement learning (IRL) and the training of the autonomous driving function by means of the learned reward function ( BF) takes place through reinforcement learning (RL) outside the vehicle. Method according to one of the preceding claims, characterized in that, in order to evaluate the safety of the autonomous driving function, a performance of the autonomous driving function is compared with a performance of a human expert.

Images