DE102023102770A1 - METHOD AND DEVICE FOR TRAINING A MODEL BASED ON ARTIFICIAL INTELLIGENCE - Google Patents

Abstract

A computer-implemented method is provided for training a model based on artificial intelligence (AI model) for creating an environment model for an automated motor vehicle. The method comprises providing a training data set comprising labeled and unlabeled sensor data from environment sensors of the motor vehicle, and training the AI model with the labeled sensor data. The method comprises training an autoencoder with the labeled sensor data, generating an environment model based on the unlabeled sensor data using the trained AI model, generating grid representations of the environment model generated based on the unlabeled sensor data using the trained AI model, generating compressed representations of the grid representations using the trained autoencoder, and performing a core set selection based on the compressed representations in order to select at least one set of sensor data from the unlabeled sensor data for further training of the AI model.

Description

The present disclosure relates to a computer-implemented method for training a model based on artificial intelligence (AI model) and/or a device for data processing or a data processing device that is designed to carry out the method. Additionally or alternatively, a computer program is provided that includes instructions that, when the program is executed by a computer, cause the computer to at least partially carry out the method. Additionally or alternatively, a computer-readable medium is provided that includes instructions that, when the instructions are executed by a computer, cause the computer to at least partially carry out the method. The present disclosure relates, additionally or alternatively, to a use of an AI model trained with the method.

Driver assistance systems are increasingly being installed in modern motor vehicles, especially automobiles.

Driver assistance systems (FAS; (Advanced) Driver Assistance Systems, (A)DAS) are electronic, particularly mechatronic, devices in motor vehicles to support the driver in certain driving situations. The focus here is often on safety aspects, but also on increasing driving comfort.

Such driver assistance systems rely, for example, on an environment model to plan a trajectory of the automated motor vehicle, which includes a model of an environment of the motor vehicle comprising static and dynamic objects that are located in an environment of the motor vehicle.

To create such an environment model, object recognition algorithms based on artificial intelligence, generally referred to here as AI models, can be used, which in the trained state are designed to generate the environment model based on sensor data from an environment sensor system of the motor vehicle, e.g. comprising a radar sensor, a camera, a LiDar sensor and/or an infrared sensor.

Training such an AI model is usually done as part of supervised learning. The AI model learns a function from given pairs of inputs and outputs, the so-called labeled training data, which in this case is sensor data. The aim of supervised learning is that the AI model is trained to make associations after several runs with different inputs and outputs.

However, training such an AI model requires time-consuming and labor-intensive labeling of the sensor data used for training, especially when it is multidimensional data (as is the case with a LiDAR sensor, for example), so one of the reasons for this is to keep the amount of labeled sensor data required for training as low as possible.

One approach is so-called active learning, which is a sub-area of iterative supervised learning. Active learning is therefore a special case of machine learning in which a learning algorithm can interactively query an information source in order to obtain previously unlabeled training data in a labeled form for further training. In the statistics literature, it is sometimes referred to as optimal experimental design. The information source is also referred to as a teacher or oracle.

Because the AI model itself selects the sensor data to label, the amount of sensor data needed for training can often be much smaller than in traditional supervised learning, where all available sensor data is labeled.

However, active learning has so far been evaluated mainly for image classification tasks and showed insufficient performance on high-dimensional data.

Against the background of this prior art, the object of the present disclosure is to provide a device and/or a method which are each suitable for enriching the prior art.

The problem is solved by the features of the independent claims. The subordinate claims and the dependent claims each contain optional further developments of the disclosure.

The task is then solved by a computer-implemented method for training an artificial intelligence-based model (AI model) to create an environment model for an automated motor vehicle.

The method comprises providing a training data set comprising labeled and unlabeled sensor data from environmental sensors of the motor vehicle, and training the AI model with the labeled sensor data.

The method further comprises training an autoencoder with the labeled sensor data, generating an environment model based on the unlabeled sensor data using the trained AI model, generating mesh representations of the environment model generated based on the unlabeled sensor data using the trained AI model, generating compressed representations of the mesh representations using the trained autoencoder, and performing a core set selection.

Core set selection can be performed based on the compressed representations to select at least one set of sensor data from the unlabeled sensor data to label for further training of the AI model.

A computer-implemented method can be understood to mean a method in which at least one step of the method is carried out at least partially and/or temporarily by a computer or a data processing device.

The term training can be understood here as machine learning. Machine learning can be understood as the artificial generation of knowledge from experience. In machine learning, an artificial system, here referred to as the model based on artificial intelligence, learns from examples and can generalize them after the learning phase has ended. To do this, algorithms in machine learning build a statistical model that is based on training data and which is counter-tested. This means that the examples are not simply learned by heart, but patterns and regularities are recognized in the learning data. This enables the system to evaluate unknown data (so-called learning transfer). In this case, the active learning approach described at the beginning is chosen (see above for details).

The autoencoder can include or consist of an artificial neural network that is used to learn efficient encodings. The goal of an autoencoder is to learn a compressed representation (so-called encoding) for a set of data and thus to extract essential features. This allows the autoencoder to be used for dimensionality reduction.

The autoencoder can comprise several layers, in particular an input layer, some layers that are significantly smaller than the input layer and form the encoding, and an output layer in which each neuron has the same meaning as the corresponding one in the input layer. Intuitively, when training the autoencoder, the encoder projects the initial data (here the grid representations) into a space with usually deeper or smaller dimensions, and the decoder has the task of reconstructing the original data. The goal is now to choose the encoder and decoder so that the difference between the original data and the reconstructed data is as small as possible. The output layer is only used for training the autoencoder, whereas the smaller layers that form the encoding are used to compress the grid representations of the unlabeled sensor data, which in turn are used to select the sensor data to be labeled using core-set selection.

The idea behind core-set selection or the core-set approach is to select a subset s of data points from a total set S such that all remaining elements of S are within radius δ of the next element of s, i.e. s is a δ-cover of S. The set s should be selected for a given size such that the radius δ is minimal. As is usual in active learning, this process is also carried out iteratively and the subset of the individual selections s ⁱ (where i is the index of the run 1,2,3 ...) is accumulated. $$\underset{{\text{s}}^1:\left|{\text{<figure-callout id="1" label="s" filenames="DE102023102770A1\_0003.png" state="{{state}}">s</figure-callout>}}^1\le \text{b}\right|}{\text{min}}{\delta }_{{\text{s}}^0\cup {\text{<figure-callout id="1" label="s" filenames="DE102023102770A1\_0003.png" state="{{state}}">s</figure-callout>}}^1}$$

By formulating the problem as a k-center problem, a computationally efficient 2-OPT solution can be established using a greedy algorithm (https://de.wikipedia.org/wiki/Greedy-Protokoll), since it is equivalent to: $$\underset{{\text{s}}^1:\left|{\text{<figure-callout id="1" label="s" filenames="DE102023102770A1\_0003.png" state="{{state}}">s</figure-callout>}}^1\right|\le b}{\text{min}}\underset{i}{\text{Max}}\underset{j\in {\text{<figure-callout id="1" label="s" filenames="DE102023102770A1\_0003.png" state="{{state}}">s</figure-callout>}}^1\cup {\text{s}}^0}{\text{min}}\mathrm{\Delta }\left({\text{x}}_i,{\text{x}}_j\right)$$

The method according to the disclosure offers, inter alia, the advantage that active learning is performed on low-dimensional environment embeddings instead of multidimensional raw sensor data, in particular LiDAR raw data, and that the approach is independent of different hardware configurations and independent of the actual task of the AI model.

Possible further developments of the procedure described above are explained in detail below.

The environmental sensors have a LiDAR sensor and the labeled and unlabeled sensor data were at least partially recorded by the LiDAR sensor.

Generating the grid representations may include generating a two-dimensional bird's eye view based on the environment model, which is present as a three-dimensional model after it is generated by the AI model, and deriving the grid representations from the generated two-dimensional bird's eye view.

In other words, the environment model generated using the AI model is initially a spatial model of the environment of the motor vehicle, which is then converted into a two-dimensional model. The two-dimensional model represents the scene or the environment in a top view (i.e. looking from a Z direction from above) and is also known as a bird's eye view (BEV). A grid is placed over this top view, which can be made up of square cells of equal size, for example. Each cell is assigned a specific information content contained in the environment model at the location of the respective cells, so that a grid representation is created. Since the environment model changes over time, several chronologically successive grid representations can be derived from the environment model.

Deriving the grid representations from the generated two-dimensional bird's eye view may comprise, for each grid representation, deriving an occupancy grid that contains information about an occupancy of individual rectangles forming the occupancy grid based on the bird's eye view, and deriving a velocity grid and an acceleration grid for an X direction and a Y direction based on the bird's eye view and the occupancy grid.

This means that the occupancy grid can contain information for each cell about whether or not there is an object at the respective location in the environment model. The occupancy grid therefore contains the spatial information from the environment model. The associated speed grids and the associated acceleration grids now assign a speed and an acceleration in X and Y, i.e. in the horizontal plane, to these objects. The speed grids and the acceleration grids therefore contain the dynamic or temporal information from the environment model.

The autoencoder may comprise a space-time autoencoder configured to extract spatial information from the occupancy grids and to extract temporal information from the velocity grids and the acceleration grids.

The sensor data may have been recorded continuously during a journey in the motor vehicle.

Furthermore, a data processing device is provided, wherein the data processing device is designed to at least partially carry out the method described above.

The data processing device can be a computer having a processor and a memory, wherein a computer program is stored in the memory, which includes instructions or commands that cause the computer to at least partially carry out the method described above when the processor loads and executes the computer program stored in the memory.

What is described above with reference to the method also applies analogously to the data processing device and vice versa.

Furthermore, a computer program is provided, comprising instructions which, when the program is executed by a computer, cause the computer to at least partially carry out or implement the method described above.

What is described above with reference to the method and the data processing device also applies analogously to the computer program and vice versa.

Furthermore, a computer-readable medium, in particular a computer-readable storage medium, is provided. The computer-readable medium comprises instructions which, when executed by a computer, cause the computer to at least partially carry out the method described above.

That is, a computer-readable medium may be provided which comprises a computer program as defined above.

The computer-readable medium can be any digital data storage device, such as a USB stick, a hard disk, a CD-ROM, an SD card or an SSD card (or an SSD hard disk).

However, the computer program does not necessarily have to be stored on such a computer-readable storage medium in order to be made available to the computer, but can also be accessed via the Internet or otherwise externally. In other words, the computer-readable medium can also be a data signal.

What is described above with reference to the method, the data processing device and the computer program also applies analogously to the computer-readable medium and vice versa.

Furthermore, the disclosure relates to a use of an AI model trained with the method described above in the context of automated driving of an automated motor vehicle

The motor vehicle can comprise a control unit or control device. The motor vehicle can be a passenger car, in particular an automobile, or a commercial vehicle, such as a truck.

The motor vehicle can be automated. The motor vehicle can be designed to at least partially and/or at least temporarily assume longitudinal guidance and/or transverse guidance by means of the control device during automated driving of the motor vehicle.

For this purpose, the control device can use an AI model trained using the method described above, in particular an environment model generated thereby.

Automated driving can be carried out in such a way that the movement of the motor vehicle is (largely) autonomous. Automated driving can be controlled at least partially and/or temporarily by the control device.

It is conceivable that the motor vehicle intervenes in the lateral guidance of the motor vehicle through a driver assistance system actively, e.g. by adjusting an actual steering wheel position, and optionally passively, e.g. by displaying a turning instruction.

The motor vehicle can be a motor vehicle with autonomy level 0, i.e. the driver takes over the dynamic driving task, even if supporting systems (e.g. ABS or ESP) are present.

The motor vehicle may be a motor vehicle of autonomy level 1, i.e. it may have certain driver assistance systems that support the driver in operating the vehicle, such as adaptive cruise control (ACC).

The motor vehicle can be a motor vehicle of autonomy level 2, i.e. it can be so partially automated that functions such as automatic parking, lane keeping or lateral guidance, general longitudinal guidance, acceleration and/or braking are taken over by driver assistance systems.

The motor vehicle can be a level 3 autonomy motor vehicle, i.e. so automated that the driver does not have to continuously monitor the vehicle system. The motor vehicle independently carries out functions such as activating the indicator, changing lanes and/or keeping in lane. The driver can attend to other activities, but will be prompted by the system to take over control within a pre-warning period if necessary.

The motor vehicle can be a motor vehicle with autonomy level 4, i.e. it can be so highly automated that the vehicle's system permanently takes over control of the vehicle. If the system can no longer handle the driving tasks, the driver can be asked to take over control.

The motor vehicle can be a level 5 autonomy motor vehicle, i.e. so fully automated that the driver is not required to perform the driving task. No human intervention is required other than setting the destination and starting the system. The motor vehicle can operate without a steering wheel and pedals.

The control device or control unit can be part of a driver assistance system or represent this. The control device can be, for example, an electronic control unit (ECU). The electronic control unit can be an intelligent processor-controlled unit that can communicate with other modules via a central gateway (CGW), for example, and that can form the vehicle's on-board network via field buses such as the CAN bus, LIN bus, MOST bus, FlexRay and/or the automotive Ethernet, e.g. together with telematics control units and/or environmental sensors.

It is conceivable that the control device controls functions relevant to the driving behavior of the motor vehicle, such as the steering, engine control, power transmission and/or the braking system. In addition, driver assistance systems such as a parking assistant, adaptive cruise control (ACC), lane keeping assistant, lane change assistant, traffic sign recognition, light signal recognition, start-off assistant, night vision assistant and/or intersection assistant can be controlled by the control device.

What is described above with reference to the method, the apparatus, the computer program and the computer-readable medium also applies analogously to the use and vice versa.

• 1 zeigt schematisch ein Ablaufdiagramm eines computer-implementierten Verfahrens zum Trainieren eines KI-Modells zum Erstellen eines Umfeldmodells für ein automatisiertes Kraftfahrzeug.

Below, an optional embodiment is described with reference to 1 described.

• 1 shows a schematic flow chart of a computer-implemented method for training an AI model to create an environment model for an automated motor vehicle.

In a first step S1 of the method, a training data set is provided comprising labeled sensor data 1 and unlabeled sensor data 2, which were continuously recorded by environmental sensors of a motor vehicle, in this case a LiDAR sensor, during a (test) drive with the motor vehicle.

In a second step S2 of the method, the AI model 3, which after training is suitable for creating an environment model 5 based on the sensor data, is (initially) trained with the labeled sensor data 1, and in a third step S3 of the method, an autoencoder 4 is trained. For this purpose, an environment model 5, initially available as a 3D model, is created using the trained model 3 based on the labeled sensor data 1 (S31). A bird's eye view 7 is then created from this environment model 5 created based on the labeled sensor data 1 (S32), from which grid representations 6 are derived for training the autoencoder 4 (S33), which are then input into the autoencoder (S34). This procedure is analogous to steps S4 - S7 of the method, which are described in detail below with reference to the unlabeled sensor data 2, with the difference that in the seventh step an output of an output layer of the autoencoder 4 is no longer used, but the compressed data is further processed.

In a fourth step S4 of the method, a three-dimensional environment model 5 is generated based on the unlabeled sensor data 2 by means of the (initially) trained AI model 3.

Subsequently, grid representations 6 of the environment model 5 generated based on the unlabeled sensor data 2 using the trained AI model 3 are generated.

For this purpose, in a fifth step S5 of the method, a two-dimensional bird's eye view 7 is generated based on the three-dimensional environment model 5, and in a sixth step S6 of the method, the grid representations 6 are derived from the generated two-dimensional bird's eye view 7. The latter includes deriving an occupancy grid that contains information about the occupancy of individual rectangles forming the occupancy grid based on the bird's eye view 7, and deriving a speed grid and an acceleration grid for an X direction and a Y direction based on the bird's eye view and the occupancy grid. The grid representations 6 are derived for the environment model 5, which changes over time, or its bird's eye view 7, at constant intervals t, t+1 ... t+n.

In a seventh step S7, the derived grid representations 6 are input into the (initially) trained autoencoder 4 in order to generate compressed representations of the grid representations 6 by means of the trained autoencoder 4. In other words, the grid representations 6 are transferred to the latent space by means of the autoencoder 4. A latent space, also referred to as a latent feature space or embedding space, is an embedding of a set of objects in a structure in which objects that are similar to one another are closer to one another in the latent space. The position within the latent space can be viewed as being defined by a series of latent variables that result from the similarities of the object. The autoencoder 4 is embodied as a space-time autoencoder which is designed to extract spatial information from the occupancy grids and to extract temporal information from the velocity grids and the acceleration grids.

In an eighth step S8, a core set selection is carried out. This can be carried out based on the compressed representations in order to select several or a set of sensor data from the unlabeled sensor data for labeling for further training of the AI model 3. The process can then be started again or a new iteration can be carried out, whereby the selected unlabeled sensor data 2 is labeled and added to the pool of labeled sensor data 1 before the start of the next pass.

List of reference symbols

1

labeled sensor data

2

unlabeled sensor data

3

AI model

4

Autoencoder

5

Environment model

6

Bird's eye view of the environment model

7

Grid representations

S1 - S8

Process steps

Claims (10)

Computer-implemented method for training a model based on artificial intelligence (AI model) (3) for creating an environment model (5) for an automated motor vehicle, the method comprising: - providing (S1) a training data set comprising labeled and unlabeled sensor data (1, 2) from environment sensors of the motor vehicle, - training (S2) the AI model (3) with the labeled sensor data (1), characterized in that the method comprises: - training (S3) an autoencoder (4) with a grid representation generated based on the labeled sensor data (1) by means of the trained AI model (3), - generating (S4) an environment model (5) based on the unlabeled sensor data (2) by means of the trained AI model (3), - generating grid representations (7) of the environment model (5) generated based on the unlabeled sensor data (2) by means of the trained AI model (3), - generating (S7) compressed representations of the grid representations (6) by means of the trained autoencoder (4), and - performing (S8) a core set selection based on the compressed representations in order to select at least one set of sensor data for labeling for further training of the AI model (3) from the unlabeled sensor data (2). Computer-implemented method according to Claim 1 , characterized in that the environmental sensors have a LiDAR sensor and the labeled and unlabeled sensor data (1, 2) were at least partially recorded by the LiDAR sensor. Computer-implemented method according to Claim 2 , characterized in that the generation of the grid representations (6) comprises: - generating (S5) a two-dimensional bird's eye view (7) based on the environment model (5), which is present as a three-dimensional model after its generation by the AI model (3), and - deriving (S6) the grid representations (6) from the generated two-dimensional bird's eye view (7). Computer-implemented method according to Claim 3 , characterized in that deriving the grid representations (6) from the generated two-dimensional bird's eye view (7) comprises for each grid representation (6): - deriving an occupancy grid which contains information about an occupancy of individual rectangles forming the occupancy grid based on the bird's eye view (7), and - deriving a speed grid and an acceleration grid for an X direction and a Y direction based on the bird's eye view (7) and the occupancy grid. Computer-implemented method according to Claim 4 , characterized in that the autoencoder (4) comprises a space-time autoencoder which is designed to extract spatial information from the occupancy grid and to extract temporal information from the velocity grid and the acceleration grid. Computer-implemented method according to one of the Claims 1 until 5 , characterized in that the sensor data were recorded continuously during a journey with the motor vehicle. Device for data processing, characterized in that the device is designed to carry out the method according to one of the Claims 1 until 6 to execute. Computer program, characterized in that the computer program comprises instructions which, when the program is executed by a computer, cause the computer to carry out the method according to one of the Claims 1 until 6 to execute. Computer-readable medium, characterized in that the computer-readable medium comprises instructions which, when executed by a computer, cause the computer to carry out the method according to one of the Claims 1 until 6 to execute. Use of a device manufactured by the method according to any of the Claims 1 until 6 trained AI model (3) in the context of automated driving of an automated motor vehicle.

Images