DE102024111500A1 - Method and system for edge case detection in vehicle camera images - Google Patents

Abstract

The present invention relates to a method for edge case detection in vehicle camera images, in which a first large visual language model (22) is trained with image-word pairs, wherein a respective word is taken from a catalog of traffic scenarios (21), in which a second large visual language model (32) is trained with image-word pairs, wherein a respective word is taken from a general dictionary (31), in which, during a vehicle's journey, environmental images (11) are continuously recorded by at least one vehicle camera directed towards the vehicle's forecourt and transmitted to an evaluation unit arranged in the vehicle, in which the evaluation unit analyzes a respective environmental image according to the first and second large visual language models and assigns at least one image-word pair identified therein to a first and corresponding second analysis group, in which at least one image-word pair not included in the catalog of traffic scenarios is assigned to a difference group of the second analysis group, and in which respective analysis data and a respective environmental image at least one image-word pair of the difference group is transmitted as a marginal case scenario to a cloud server (50) (45). Furthermore, a system on which the procedure can be executed is presented.

Description

The present invention relates to a method for edge case detection in vehicle camera images. Furthermore, a system on which the method can be implemented is presented.

There are numerous atypical traffic scenarios in road traffic, such as a cow running across a highway. These cannot be included in scenario-based testing when creating a scenario and test catalog for an autonomous driving function because they are unknown until they occur. Identifying such edge cases, for example, in images or data provided by vehicle cameras, is not trivial. This is because object detectors are already trained on known scenarios to recognize specific objects, such as pedestrians or vehicles. Consequently, semantic information describing a previously unknown edge case—referred to as a corner case or edge case—is lacking. Furthermore, processing the sheer volume of images recorded by the individual vehicle cameras of, for example, a fleet of vehicles during journeys presents a significant data analysis challenge.

A visual language model, abbreviated VLM for "visual language model", or a so-called large visual language model with a vocabulary expanded compared to the VLM and referred to as LVLM for "large visual language model", are applications of machine learning in which a neural network is trained to recognize visual content from individual images or image sequences and assign it to linguistic terms.

The printed matter CN 117073701 A reveals a method for navigating by voice command using visual landmarks.

In the printed publication WO 2022/187063 A1 Visual input is described using a VLM.

The printed matter US 2023/0154213 A1 It reveals a method for image recognition. An LVM is trained with word-image pairs and learns to assign description classes to objects.

Against this background, an object of the present invention is to present a method for identifying edge case scenarios in which a reduced amount of data is transmitted to a cloud server. The identification of an edge case scenario in vehicle camera images is intended to take place directly in the vehicle itself, so that only the data necessary to describe the edge case is generated. Furthermore, a system with which the method can be carried out is presented.

To solve the aforementioned problem, a method for edge case detection in vehicle camera images is proposed. This method involves training a first large visual language model with image-word pairs, where each word is taken from a catalog of traffic scenarios. A second large visual language model is also trained with image-word pairs, where each word is taken from a general dictionary. During a vehicle's journey, at least one vehicle camera, focused on the area in front of the vehicle, continuously captures images of the surroundings and transmits them to an evaluation unit located within the vehicle. The evaluation unit analyzes each image of the surroundings according to the first large visual language model and assigns at least one identified image-word pair to a first analysis group. The evaluation unit then analyzes each image of the surroundings according to the second large visual language model and assigns at least one identified image-word pair to a second analysis group. At least one image-word pair from the second analysis group that is not included in the catalog of traffic scenarios is assigned to a difference group. Each analysis data set and corresponding environmental image for at least one image-word pair of the difference group are transmitted to a cloud server as a boundary case scenario. This advantageously reduces the number of environmental images to be transmitted by pre-evaluating them in the vehicle, focusing on images or data relevant to boundary cases.

Using the method according to the invention, unusual situations recorded in the vehicle's camera images can be identified in the vehicle's approach path. This significantly reduces the amount of recorded images that need to be sent back to the cloud server or from there to a central server, e.g., located at the vehicle manufacturer's premises or in a development center for autonomous driving systems or driver assistance systems, compared to the totality of all images recorded during journeys of, for example, a fleet of vehicles. Furthermore, this also eliminates the need for additional data analysis of the transmitted data to identify such edge case scenarios.

It is conceivable that the transmitted images or data relating to the edge case scenario could directly contribute to the development of a traffic scenario and the... The driving instructions for the autonomous driving system can be used. Furthermore, it is conceivable that these processes could be automated on a virtual test bench for the validation of autonomous driving systems.

It goes without saying that the training of the first and second major visual language models is conducted using a large number of words. This is done with the catalog of traffic scenarios, with the goal of being able to identify all known traffic scenarios through image-word pairs, and with the general dictionary, with the goal of being able to identify all objects that might occur in traffic situations, also through corresponding image-word pairs. These potentially occurring objects include, for example, all items that can be transported by trucks and that could become an obstacle on a road if cargo were to be lost. Conversely, large sections of the general dictionary can be excluded from the training of the second major visual language model, containing words describing objects that cannot be detected by vehicle cameras (e.g., due to their size) or words describing non-material things in general (e.g., philosophical terms).

In one embodiment of the method according to the invention, each word is assigned a respective coordinate in a phase space related to language. A distance measure is established between individual coordinates.

Two words are considered synonymous if the distance measure calculated for them falls below a predefined upper limit. This distance measure can be defined in a lexical-semantic word network, such as that provided by WordNet® (https://wordnet.princeton.edu/).

In a further embodiment of the method according to the invention, the edge case scenario transmitted by the vehicle is forwarded from the cloud server to an instance for the development of a corresponding driving function. This instance is, for example, a virtual test bench for the development of autonomous driving systems, on which a multitude of driving functions are tested in response to the occurrence of the edge case scenario, and only those that enable uninterrupted driving are retained for implementation in the autonomous driving system.

In a further embodiment of the method according to the invention, after the development of a driving function adapted to the edge case scenario, the catalog of traffic scenarios is extended to include the edge case scenario. The catalog of traffic scenarios, along with the corresponding driving functions assigned to each traffic scenario, is transmitted to the vehicle.

In a further embodiment of the method according to the invention, the vehicle is controlled by an autonomous driving system or a driver assistance system according to the catalog of traffic scenarios and the respective corresponding driving functions.

• • während einer Fahrt des Fahrzeugs von der mindestens einen Fahrzeugkamera Umgebungsbilder zu empfangen,
• • ein jeweiliges Umgebungsbild gemäß dem ersten großen visuellen Sprachmodell zu analysieren und mindestens ein darin identifiziertes Bild-Wort-Paar einer ersten Analysegruppe zuzuordnen,
• • das jeweilige Umgebungsbild gemäß dem zweiten großen visuellen Sprachmodell zu analysieren und mindestens ein darin identifiziertes Bild-Wort-Paar einer zweiten Analysegruppe zuzuordnen,
• • mindestens ein nicht im Katalog von Verkehrsszenarien enthaltenes Bild-Wort-Paar der zweiten Analysegruppe einer Differenzgruppe zuzuordnen, und
• • jeweilige Analysedaten und das jeweiliges Umgebungsbild zu dem mindestens einen Bild-Wort-Paar der Differenzgruppe als ein Randfall-Szenario per Funkkommunikation an den Cloud-Server zu übermitteln.

Furthermore, a system is claimed, comprising a vehicle with at least one vehicle camera oriented towards the area in front of the vehicle, an evaluation unit with a computer processor and a storage medium, a first and second major visual language model stored on the storage medium and executable on the computer processor, and means for radio communication, and a cloud server. The first major visual language model is trained with image-word pairs, where one word in each pair is taken from a catalog of traffic scenarios. The second major visual language model is trained with image-word pairs, where one word in each pair is taken from a general dictionary. The evaluation unit is configured to run continuously.

• • to receive images of the surroundings from at least one vehicle camera while the vehicle is in motion,
• • to analyze a given environmental image according to the first major visual language model and to assign at least one identified image-word pair to a first analysis group,
• • to analyze the respective environment according to the second major visual language model and to assign at least one image-word pair identified therein to a second analysis group,
• • to assign at least one image-word pair not included in the catalog of traffic scenarios to the second analysis group of a difference group, and
• • to transmit the respective analysis data and the respective environmental image to the cloud server via radio communication as a marginal case scenario for at least one image-word pair of the difference group.

The system according to the invention thus advantageously enables a reduction in the amount of data required for transmitting images from the at least one vehicle camera aimed at the area in front of the vehicle to the cloud server or central server for analyzing edge case scenarios. This is achieved by means of an evaluation unit embedded in the vehicle (English "embedding") for extracting semantic attributes from the camera images. This allows new, atypical objects (since they are not yet present in the catalog of traffic scenarios) to be recognized. In addition, objects that are already known (since they are present in the general dictionary) (e.g., a cow, see...) can be identified. 1 ), which are located in an atypical context (e.g. on the roadway), are identified.

In one embodiment of the system according to the invention, each word is assigned a specific coordinate in a language-related phase space. A distance measure is established between individual coordinates. The evaluation unit is configured to treat two words whose distance measure falls below a predefined upper limit as synonymous.

In a further embodiment of the system according to the invention, the cloud server is configured to forward the edge case scenario transmitted by the vehicle to an instance for the development of a corresponding driving function.

In a further embodiment of the system according to the invention, the cloud server is configured to transmit a driving function developed by the instance for the edge case scenario and the catalog of traffic scenarios extended by the edge case scenario with corresponding driving functions assigned to the respective traffic scenarios to the vehicle.

In a further embodiment of the system according to the invention, the system comprises an autonomous driving system or a driver assistance system. The autonomous driving system or the driver assistance system is configured to execute corresponding driving functions according to the catalog of traffic scenarios.

Further advantages and embodiments of the invention will become apparent from the description and the accompanying drawing.

• 1 zeigt ein Ablaufschema zu einer Ausführungsform des erfindungsgemäßen Verfahrens.

It is understood that the features mentioned above and those to be explained below can be used not only in the combinations specified, but also in other combinations or on their own, without leaving the scope of the present invention.

• 1 shows a flowchart for an embodiment of the method according to the invention.

In 1 A flowchart 10 for an embodiment of the method according to the invention is shown. First, before driving operation, preferably before the vehicle is first put into operation, the first large visual language model 22 and the second large visual language model 32 are trained with their respective image-word pairs. For this purpose, for the first large visual language model 22, a word corresponding to a given image-word pair is taken from a catalog of traffic scenarios 21, and for the second large visual language model 32, a word corresponding to a given image-word pair is taken from a general dictionary 31. For each traffic scenario, corresponding driving scenarios are available, which can be used, for example, to train an autonomous driving system or an autonomous driver assistance system to handle the respective traffic scenario. During a journey, a vehicle camera focused on the area in front of the vehicle continuously captures an image 11 of the surroundings. In an exemplary traffic scenario presented here, this image shows a cow 14 and a person 15 on the left side of the road (from the observer's perspective). The image 11 is transmitted to an evaluation unit located in the vehicle. This unit analyzes the image 11 using both the first large visual language model 22 and the second large visual language model 32. The evaluation unit assigns a similarity measure 23, 33 to each word associated with a corresponding image identified in the image 11. The value for the similarity measure 23, 33 is determined by the respective large visual language model 22, 32 according to the degree of similarity between the identified image and the corresponding image of the trained image-word pair. In the exemplary environment image 11 shown, the first large visual language model 22 recognizes the image of a country road 24, a person 25 and a lane 26 and assigns the respective value for the similarity measure 23 to the associated word (country road 24, person 25, lane 26). 1 represented by a respective bar height, the words found (country road 24, person 25, lane 26) are assigned to a first analysis group. Similarly, in the environment image 11, the second large visual language model 32 recognizes the image of a cow 34, a man 35, and a road 36, with the image of the cow 34 receiving a comparatively high similarity value. Here, too, the found words (cow 34, man 35, road 36) are assigned to a second analysis group. In the next step, similar words in both analysis groups are made synonymous. For this purpose, a distance measure between the respective words arranged in a word space is considered. For example, the distance measure between country road 24 and road 36 is so small in light of a traffic scenario description that both terms are considered synonymous. Words can be treated as synonyms, for example, the word "Straße 36" (Street 36) is deleted, leaving only the word "Landstraße 24" (Country Road 24). Such processes can be automated using a lexical-semantic word network by specifying an upper limit for the distance measure below which words are treated as synonyms. The upper limit is chosen, for example, according to an average distance measure within word families. In an assignment step of the embodiment of the method according to the invention, those words from the remaining words of both analysis groups that are contained in the catalog of traffic scenarios are assigned to a traffic scenario group 41 (42, 43) (which is already intrinsically the case for all words of the first analysis group). On the other hand, those words of the second analysis group that are not contained in the catalog of traffic scenarios form a difference group. In the illustrated case, "Landstraße 24", "Person 25", "Fahrlane 26", and "Mann 35" are found in traffic scenario group 41, while "Rind 34" forms the difference group. Then the difference group or information contained therein relating to the identified image-word pair, here Rind 34, is transmitted to a cloud server 50 together with the image from the vehicle camera in which the image-word pair was identified as a marginal case 45.

List of reference symbols

10

Flowchart

11

Surround images from vehicle camera

14

Beef

15

person

21

Entities as words from scenario catalog

22

Trained large image-language model

23

Similarity measure for first analysis group

24

person

25

country road

26

road

31

Entities as words from a general dictionary

32

Trained large image-language model

33

Similarity measure for second analysis group

34

Beef

35

People

36

Street

40

Assignment step

41

Traffic Scenarios Group

42

Supply

43

Supply

45

transmission

50

Cloud server

QUOTES CONTAINED IN THE DESCRIPTION

This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions.

Cited patent literature

• CN 117073701 A [0004]
• WO 2022/187063 A1 [0005]
• US 2023/0154213 A1 [0006]

Claims (10)

Method for edge case detection in vehicle camera images, wherein a first large visual language model (22) is trained with image-word pairs, each word being taken from a catalog of traffic scenarios (21), wherein a second large visual language model (32) is trained with image-word pairs, each word being taken from a general dictionary (31), wherein during a vehicle journey, ambient images (11) are continuously recorded by at least one vehicle camera directed towards the vehicle's forecourt and transmitted to an evaluation unit arranged in the vehicle, wherein the evaluation unit analyzes each ambient image (11) according to the first large visual language model (22) and assigns at least one image-word pair identified therein to a first analysis group, wherein the evaluation unit analyzes the respective ambient image (11) according to the second large visual language model (32) and assigns at least one image-word pair identified therein to a second analysis group is assigned, in which at least one image-word pair not included in the catalog of traffic scenarios (21) of the second analysis group (32) is assigned to a difference group, and in which respective analysis data and a respective environment image (11) to which at least one image-word pair of the difference group is transmitted as a marginal case scenario to a cloud server (50) (45), thereby reducing the number of environment images (11) to be transmitted to marginal cases by pre-evaluation in the vehicle. Procedure according to Claim 1 , in which each word is assigned a respective coordinate in a language-related phase space, in which a distance measure is formed between individual coordinates, and in which two words are made synonymous if a predetermined upper limit is not exceeded by the distance measure formed for them. Method according to one of the preceding claims, wherein the edge case scenario transmitted by the vehicle is forwarded by the cloud server to an instance for the development of a corresponding driving function. Procedure according to Claim 3 , in which, after the development of a driving function adapted to the edge case scenario, the catalog of traffic scenarios is extended to include the edge case scenario, and in which the catalog of traffic scenarios (21) with corresponding driving functions assigned to the respective traffic scenarios is transmitted to the vehicle. Method according to one of the preceding claims, wherein the vehicle is controlled by an autonomous driving system or a driver assistance system in accordance with the catalogue of traffic scenarios (21) and respective corresponding driving functions. System comprising a cloud server and a vehicle, comprising at least one vehicle camera facing the area in front of the vehicle, an evaluation unit with a computer processor and a storage medium, a first and second major visual language model (22, 32) stored on the storage medium and executable on the computer processor, and means for radio communication, wherein the first major visual language model (22) is trained with picture-word pairs, each word being taken from a catalog of traffic scenarios (21), wherein the second major visual language model (32) is trained with picture-word pairs, each word being taken from a general dictionary (31), wherein the evaluation unit is configured to continuously: • receive environmental images (11) from the at least one vehicle camera while the vehicle is driving, • analyze each environmental image (11) according to the first major visual language model (22) and assign at least one picture-word pair identified therein to a first analysis group to assign, • to analyze the respective environment image (11) according to the second large visual language model (32) and to assign at least one image-word pair identified therein to a second analysis group, • to assign at least one image-word pair not included in the catalog of traffic scenarios to the second analysis group of a difference group, and • to transmit the respective analysis data and the respective environment image (11) to the at least one image-word pair of the difference group as a marginal case scenario via radio communication to the cloud server (50) (45). System according Claim 6 , wherein each word is assigned a respective coordinate in a language-related phase space, wherein a distance measure is formed between individual coordinates, and wherein the evaluation unit is configured to treat two words whose distance measure is less than a specified upper limit as synonymous. system according to one of the Claim 6 or 7 , where the cloud server is configured to handle the edge case scenario transmitted by the vehicle to forward the information to an authority for the development of a corresponding driving function. System according Claim 8 , wherein the cloud server is configured to transmit to the vehicle a driving function developed by the instance for the edge case scenario and the catalog of traffic scenarios (21) extended by the edge case scenario with corresponding driving functions assigned to the respective traffic scenarios. system according to one of the Claims 6 until 9 , comprising an autonomous driving system or a driver assistance system, wherein the autonomous driving system or the driver assistance system is configured to perform appropriate driving functions in accordance with the catalogue of traffic scenarios (21).