DE102023211591B4 - Method and assistance system for behavior-dependent warning of a driver of hazards and correspondingly equipped motor vehicle - Google Patents

Abstract

Method for warning a driver (8) of a motor vehicle (4) of dangers in its surroundings, wherein during the travel of the motor vehicle (4)
- the surroundings of the motor vehicle (4) are continuously monitored for objects (5, 6, 7) located there,
- the driver’s behaviour (8) is continuously recorded,
- based on the recorded behaviour of the driver (8), a driving manoeuvre (9) likely intended by the driver is determined,
- those objects (5, 6, 7) detected in the surroundings of the motor vehicle (4) which are relevant for this driving manoeuvre (9) are determined,
- a warning (21) is issued only on these specific relevant objects (7, 19), and
a) driver behavior data indicating the behavior of the driver (8) of the motor vehicle (4) and corresponding driving maneuver data indicating driving maneuvers (9) actually performed by the driver (8) according to recorded driver behavior data are provided as training data for a machine learning device by an assistance system (13) for the motor vehicle (4) set up to carry out the method, and an output of the machine learning device trained therewith is used to determine or indicate the respectively expected intended driving maneuver (9),
and/or
b) the assistance system (13) for the motor vehicle (4) set up to carry out the method records routes travelled by the driver (8) over several journeys of the motor vehicle (4) as part of the driver behaviour data and a resulting movement profile of the driver (8) is taken into account to determine the respective likely intended driving manoeuvre (9).

Description

The present invention relates to a method and an assistance system for warning a driver of a motor vehicle of hazards. The invention further relates to a correspondingly configured motor vehicle.

Dangerous situations and accidents repeatedly occur in road traffic when, for example, drivers fail to see other road users in their surroundings. Although mirrors are usually provided in motor vehicles to help the driver get an overview of their surroundings, experience has shown that this is not always sufficient, for example in situations with a large number of other road users in the vicinity and/or high relative speeds between the vehicle and other road users. For example, a quick glance in the mirror may not always correctly identify all of the road users visible there, and a brief glance at the mirror may not reveal relative speeds. There is therefore a need for further improvements in this area.

One approach is described by the DE 10 2009 002 277 A1 A driver assistance system for warning the driver of a vehicle about obstacles in the vehicle's surroundings. An evaluation unit generates a reference value based on sensor data and compares it with a subsequent instantaneous value for obstacle detection, also generated based on sensor data. A warning is then issued if the instantaneous value deviates from the reference value. The reference value is also generated before the vehicle starts moving if a specific driver behavior is detected.

The DE 10 2011 012 793 A1 Describes a driver assistance system for displaying and/or autonomously or semi-autonomously controlling a collision-avoiding or collision-mitigating driving maneuver of a motor vehicle. This system determines an evasive trajectory based on the current surrounding situation and a trajectory prediction for the motor vehicle and for a detected object in the surrounding area, taking driver characteristics into account. The driving maneuver based on the determined evasive trajectory is then displayed and/or controlled.

The DE 11 2019 004 597 T5 describes an information processing device comprising a data processing unit that determines a level of alertness corresponding to the level of consciousness of a driver of a vehicle. The data processing unit obtains observation information about the driver and determines the driver's level of alertness based on the obtained observation information.

The DE 11 2014 001 436 T5 Describes a method for controlling vehicle systems in a motor vehicle. The method includes obtaining information from a first vehicle system, determining a level of drowsiness, detecting a hazard, modifying control of the first vehicle system based on at least the level of drowsiness, and selecting a second vehicle system. The second vehicle system differs from the first vehicle system. The method further includes modifying control of the second vehicle system based on at least the level of drowsiness.

The DE 10 2012 002 149 B3 Describes a method for visualizing the surroundings of a motor vehicle, in which the surroundings are recorded using a camera and the camera image is displayed on a screen. Furthermore, different colored frames are used for vehicles with different relative speeds, with the color of the frame indicating whether the detected vehicle is one with a critical relative speed or not. This is intended to support the driver by displaying processed objects in the vehicle's surroundings.

As a further approach, the DE 20 2006 010 960 U1 A video surveillance system for use in motor vehicles. Several cameras are mounted around a vehicle to record the surroundings. A digital image processing unit, which contains an image display unit and a unit for highlighting moving objects, serves as an intermediary between the cameras and a display unit within the vehicle. The image display unit combines the raw images from the individual cameras into a single, flat image for display on the display unit. The unit for highlighting moving objects places them in a frame that tracks movement within the cameras' detection range. This is intended to increase security.

The object of the present invention is to enable improved safety in traffic.

This problem is solved by the subject matter of the independent claims. Further possible Various embodiments of the invention are disclosed in the dependent claims, the description, and the figures. Features, advantages, and possible embodiments presented in the description for one of the subject matters of the independent claims are to be regarded, at least analogously, as features, advantages, and possible embodiments of the respective subject matters of the other independent claims, as well as any possible combination of the subject matters of the independent claims, optionally in conjunction with one or more of the dependent claims.

The method according to the invention serves to warn a driver of a motor vehicle of hazards in their surroundings. The method can be carried out automatically while the motor vehicle is moving, for example, using an appropriately configured assistance system of the motor vehicle. In the method according to the invention, the surroundings of the motor vehicle are continuously monitored for objects located there. Such objects can be, in particular, other road users or static obstacles, such as bollards, fences, walls, lampposts, or the like. Thus, the respective surroundings or surrounding traffic events can be observed using a corresponding environmental sensor system of the motor vehicle, such as at least one camera and/or a radar device and/or a lidar device. Based on corresponding sensor data, objects in the surroundings can be automatically detected, for example, using corresponding image or data processing, in particular based on machine learning. Likewise, as part of the monitoring of the surroundings, car-to-car data from the surroundings or from other road users or vehicles in the surroundings of the motor vehicle can be collected, and other vehicles or road users in the surroundings can be detected based on this data. Such Car2Car data can, for example, indicate the types, positions, speeds, directions of travel, and/or planned trajectories of other vehicles or road users. For example, an approaching vehicle can communicate via a Car2Car connection that it is approaching the motor vehicle at a certain speed and/or that it is likely to enter its vicinity. This makes it possible to identify relevant objects, if necessary, even if the motor vehicle itself does not have its own environmental sensors or if these are impaired. Likewise, the position of the motor vehicle can, for example, be continuously recorded or tracked. Based on this, map data, particularly high-resolution or highly detailed map data, can then be evaluated. For example, static objects in the respective surroundings of the motor vehicle can be identified based on the map data. Likewise, a comparison or plausibility check can be carried out between the sensor data or objects detected therein and the map data.

Furthermore, the method according to the invention continuously records and monitors the driver's behavior. For this purpose, the driver can be observed, for example, using an interior camera of the motor vehicle. Gestures, gaze directions, gaze changes, head positions, hand positions, foot positions, operating actions, verbal utterances, and/or the like can be recorded as part of the driver's behavior. Likewise, the driver's mental and/or physical state can be recorded and/or monitored. For example, a state of alertness or fatigue can be determined, and/or pupil movements, for example in terms of extent, pattern, frequency, and speed, can be recorded.

Furthermore, in the method according to the invention, a driving maneuver that is likely intended by the driver, i.e., a driving maneuver that is likely planned or imminent, is determined or predicted on the basis of the detected behavior of the driver. For this purpose, for example, an appropriately trained machine learning device can be used that is trained to predict the driver's likely subsequent or intended driving maneuvers on the basis of driver behavior data that indicate, depict, or characterize the driver's behavior. Likewise, for example, predetermined criteria and/or a predetermined assignment table can be evaluated in order to assign detected behavior or behaviors of the driver to a driving maneuver that is likely intended by the driver. If, for example, a glance over the shoulder to the left and a brief movement of the steering wheel to the left to steer the vehicle to the left edge of the currently occupied lane (in the direction of travel) is detected as driver behavior in right-hand traffic, this can be used to infer a likely intended lane change to the adjacent lane on the left. For example, a driver's more than just passing glance at a traffic sign indicating an upcoming exit and a subsequent or associated reduction in speed and the positioning of a hand within reach of a control element for a direction indicator can be used to infer an impending driving maneuver to leave the current road via the upcoming exit. Likewise, a variety of other behaviors or behavior patterns of the driver indicate a variety of other driving maneuvers.

Likewise, map and/or environmental data for the respective current surroundings and those ahead in the direction of travel of the motor vehicle can be taken into account or evaluated. Based on this, it can be determined, for example, which driving maneuvers of the motor vehicle are actually possible, sensible, or plausible, or whether the determined, likely intended driving maneuver is possible, sensible, or plausible in the respective environment. For example, it can be determined that the motor vehicle is driving in a right-hand traffic system in the right of two lanes intended for the same direction of travel, and that the left lane ends immediately ahead or merges with the right lane. In such a situation, it can then be determined or decided, for example, that the driver is unlikely to plan an overtaking maneuver. Corresponding glances by the driver in the left side mirror, which might otherwise be typical of an impending overtaking maneuver, can then be interpreted differently, or the likely intended driving maneuver initially identified based on these data can then be rejected based on the map and/or environmental data. As part of the map or environmental data, for example, lane layouts, speed limits, overtaking bans, positions of restricted areas and/or the like can be recorded and taken into account or evaluated. Likewise, the speed of one or more other road users or motor vehicles in the vicinity of the motor vehicle, in particular in or only in the area ahead of the motor vehicle in the direction of travel, relative to the motor vehicle and/or the operating state of at least one driver assistance system of the motor vehicle set up for the longitudinal guidance or longitudinal control of the motor vehicle can be determined and taken into account in order to determine or verify the plausibility of the likely intended driving maneuver. Based on this, it can be determined, for example, that the motor vehicle is currently being controlled with active adaptive cruise control in order to follow a vehicle in front at the same speed. In this case, for example, a manual overtaking maneuver can be rejected as implausible, or it can be provided that, in this case, detection of a likely intended driving maneuver, such as an overtaking maneuver that requires or includes overriding the active automated cruise control, is determined only based on additional or different behaviors than the likely intended one. This can apply, for example, in comparison to a situation - for example, an otherwise identical situation - in which the speed of the motor vehicle is not adapted to the speed of one or more other vehicles in the surrounding area, particularly those ahead, or is not automatically controlled.

In a first variant of the present invention, driver behavior data indicating the behavior of the driver of the motor vehicle and corresponding driving maneuver data indicating driving maneuvers actually performed by the driver based on recorded driver behavior data are provided as training data for a machine learning device by an assistance system for the motor vehicle configured to carry out the method, and an output of the machine learning device trained therewith is used to determine or indicate the respectively anticipated intended driving maneuver. In principle, the use of the machine learning device proposed here for processing the driver behavior data can enable particularly accurate and reliable determination of the respectively anticipated intended driving maneuver. Furthermore, the present invention can thus be implemented comparatively simply and with little effort, for example, compared to a manual implementation of a driver behavior detection system and a decision algorithm for determining the respectively anticipated intended driving maneuver therefrom.

Additionally or alternatively, in a second variant of the present invention, the assistance system for the motor vehicle configured to execute the method records routes traveled by the driver over several journeys of the motor vehicle as part of the driver behavior data, and a resulting driver movement profile is then taken into account to determine the likely intended driving maneuver. This enables a particularly accurate and reliable determination of the actually intended driving maneuver.

Furthermore, the method according to the invention determines those objects detected in the respective current surroundings of the motor vehicle which are actually or presumably relevant for the determined driving maneuver presumably intended by the driver. Relevant objects can be objects which must be observed or taken into account for the safe execution of the driving maneuver, or which are or can become actually or potentially dangerous when performing the driving maneuver, or which represent an obstacle or with which there is a risk of collision when performing the driving maneuver, or the like. Objects which are Objects that need to be observed or taken into account can, for example, be objects that are or would be located, at least temporarily, within a specified maximum distance from the motor vehicle when the driving maneuver is carried out. To determine the relevant objects, for example, movement data of the motor vehicle and the detected objects can be evaluated or extrapolated, or movements or relative movements of the motor vehicle and the detected objects can be modeled or simulated, or the like.

Furthermore, in the method according to the invention – at least if at least one relevant object has been detected – a warning is issued only for the specific relevant objects. This allows the driver's attention to be directed or focused precisely on these objects. For this purpose, for example, only the relevant objects can be highlighted or marked in a display device of the motor vehicle compared to their surroundings and/or other detected objects that were not classified as relevant for the expected intended driving maneuver. The type or design of the warning can be adapted to the previously recorded or determined mental and/or physiological state of the driver. For example, the warning can be selected or optimized to particularly effectively attract the driver's attention and/or avoid startling the driver. For this purpose, for example, a corresponding specification, such as in the form of a table or a characteristic map, can be taken into account, or a predefined model or a suitably trained machine learning device can be used to determine the optimal type and/or design of the warning to be used in each case. Likewise, the driver's current line of sight can be taken into account. This means that the warning or part of it can, for example, be output only or at least also on one or more of the display devices of the motor vehicle that are currently in the driver's field of vision, i.e., for example, at least or only on the central interior mirror or a side mirror or an instrument cluster or a display in the center console or a HUD. If the driver uses AR, VR, or XR glasses or a corresponding headset, at least one virtual spatial area can be specified for outputting the marking or warning. The output or display of the warning can also follow the driver's field of vision. If the driver therefore turns their gaze from a first display device or a first line of vision to a second display device or in a second line of vision, the warning can be output accordingly first, in particular only, on the first display device or in the first line of vision and then, in particular only, on the second display device or in the second line of vision. This can help to ensure that the driver sees the warning message and avoid distractions to the driver caused by changes in the display on a display that is only in the driver's peripheral field of vision.

The present invention can enable the driver to recognize relevant objects or potential hazards particularly quickly and easily, and with particularly low cognitive load. Likewise, the warning can enable the driver to detect the presence and relevance of objects that, without the warning, they might not yet be able to correctly recognize or identify, or assess in terms of their position and/or speed relative to the motor vehicle. This allows the driver to react promptly and specifically to avoid dangerous situations. The driver can be relieved of the burden of having to completely assess the relevance of various recognizable objects themselves. Furthermore, the driver does not have to keep their gaze fixed on the mirror, to the side, or to the rear for extended periods in order to reliably detect the relative movements of other road users. This allows the driver, for example, to monitor the traffic ahead in the direction of travel or the wider surroundings of the motor vehicle more reliably, i.e., with shorter pauses. Thus, the present invention can contribute to increasing safety and relieving the driver's workload, thus correspondingly increasing comfort. This can be achieved, in particular, without the driver having to fundamentally change their behavior compared to using conventional motor vehicles. For example, the driver does not have to consciously behave in a specific, predetermined manner, nor does he have to explicitly indicate or communicate his intended driving maneuvers. This makes the present invention particularly simple and reliable to use.

In particular, it can be provided that the warning is only issued upon detection of the impending driving maneuver, i.e., the corresponding driver intention, and at most until the driving maneuver has been successfully completed or until it has been determined that the driving maneuver will probably not be carried out. In other words, for example, not all detected objects or all detected objects within a specified distance from the motor vehicle and/or at a specified relative speed are permanently displayed. ity relationship to the motor vehicle is highlighted or marked. Likewise, if, for example, the driver accidentally or routinely glances in the mirror or at the surroundings without further indications of a likely intended driving maneuver, the detected objects may not be highlighted or marked accordingly, and no other warning will be issued. This can prevent the driver from being distracted or overwhelmed by a large number of markings, highlights, or warnings. This can lead to, or contribute to, the driver recognizing a warning actually issued according to the present method more quickly, more reliably, and more consciously, and attaching greater importance to it. This effect can also improve road safety compared to other approaches.

Likewise, the driver's behavior or reaction to an issued warning can be automatically recorded. In particular, it can also be recorded or determined whether the driver has carried out the identified, presumably intended, driving maneuver, or which driving maneuver the driver actually carries out. It can also be determined whether a critical situation, a near-accident, a subsequent evasive maneuver, emergency braking, or the like occurs, or whether, for example, all specified speed limits and safety distances were observed. Based on this, it can then be automatically determined, in particular using machine learning, in which situations a warning is appropriate and/or which type, design, and/or output position of warnings is appropriate. This can then be taken into account for future issuance of warnings or determine future issuance of warnings.

The present invention also relates to an assistance system for a motor vehicle. The assistance system according to the invention has an input interface for acquiring environmental data that depicts or characterizes a respective environment or indicates objects located therein, and driver behavior data that indicates, for example, depicts or characterizes, the behavior of a driver of the motor vehicle. Furthermore, the assistance system has a data processing device for processing the acquired data and for determining a respective likely intended driving maneuver and relevant objects in the environment based on the acquired data, and for generating a respective corresponding control signal for generating a warning only for precisely the relevant objects. For this purpose, the data processing device can, for example, comprise a processing device, such as a microchip, microprocessor, microcontroller, or the like, and a computer-readable data memory coupled thereto. Likewise, the data processing device can be, comprise, or implement a machine learning device, such as an artificial neural network. The assistance system according to the invention also has an output interface for outputting the control signals. The input interface and the output interface can be different from one another, combined with one another, or integrated, for example, as a single bidirectional interface. The input interface and the output interface can each be implemented entirely or partially in hardware and/or software. The assistance system according to the invention is configured to execute the method according to the invention, in particular automatically. For this purpose, a corresponding operating or computer program can be used, for example, which codes or implements the method steps, measures, or sequences or corresponding control instructions described in connection with the method according to the invention or also subsequently in connection with the assistance system according to the invention and/or motor vehicle. This operating or computer program can, for example, be stored in the aforementioned data memory and be executable by means of the aforementioned processor device in order to execute the corresponding method or to effect its execution. The assistance system according to the invention can, in particular, be the assistance system mentioned in connection with the method according to the invention. For example, the assistance system according to the invention can be configured as a control unit for a motor vehicle or as a part or module of a more comprehensive vehicle system.

In a first variant of the present invention, the assistance system is configured to provide the driver behavior data for the driver and corresponding driving maneuver data, which indicate driving maneuvers actually performed by the driver according to recorded driver behavior data or driver behaviors, as training data for a machine learning device. Furthermore, the assistance system is configured to use an output of the corresponding machine learning device trained therewith to determine or indicate the respective likely intended driving maneuver. The assistance system can itself comprise the machine learning device. Likewise, the machine learning device can This can be a device that is different from the assistance system itself, in particular one external to the vehicle, for example an external server device, such as a cloud server or backend or a central data center or the like. The assistance system can then be configured to send the driver behavior data and the driving maneuver data to the external machine learning device – for example, via a wireless data connection, such as a Wi-Fi or mobile network connection.

The assistance system can be configured to send driver behavior data recorded during the journey to the external machine learning device and to collect data sent by the device in response, indicating a likely intended driving maneuver of the driver, determined or predicted by the machine learning device. Based on this, the assistance system can then determine the relevant objects in the environment.

The use of a local machine learning device, i.e., one provided as part of the assistance system or the motor vehicle, can enable particularly fast and low-latency processing of driver behavior data for applying or implementing the method according to the invention, without relying on an existing data connection. Furthermore, the machine learning device can then easily be individually trained for each driver, i.e., it can learn or have learned the individual driver-specific behavior patterns or behaviors of each driver with particular precision.

The use of an external, particularly vehicle-external, machine learning facility to process driver behavior data can, however, more easily enable the use of more or more powerful computing resources and/or the use of data from a large number of drivers and thus a particularly robust and reliable determination or prediction of the intended driving maneuver in each case.

In principle, the proposed use of the machine learning device for processing driver behavior data can enable a particularly accurate and reliable determination of the likely intended driving maneuver. Furthermore, the present invention can be implemented comparatively simply and with little effort, for example, compared to a manual implementation of driver behavior detection and a decision algorithm for determining the likely intended driving maneuver from it.

Additionally or alternatively, in a second variant of the present invention, the assistance system is configured to record routes traveled by the driver over multiple journeys of the motor vehicle as part of the driver behavior data and to use or consider a resulting long-term movement pattern or movement profile of the driver, learned in particular by means of machine learning, to determine the respective likely intended driving maneuver. For example, a driving plan or timetable, i.e., a driver's routine, can be determined here. From this, a likely upcoming driving maneuver, such as a lane change or turning, or the like, can then be derived or predicted, even when navigation guidance is not active. This is based on the realization that many journeys are regularly repeated routine journeys and/or the driver's journeys or behaviors can follow a specific recurring plan or pattern. For example, it can be determined that the driver is currently on a journey or is following a route that they regularly repeat. Not only position or movement data, but also time data, such as the current time of day and/or the current day of the week, or similar, can be taken into account. For example, it can be detected that the driver is currently on their morning commute from their home to their regular place of work. Based on previous such journeys, it can then be known or learned which driving maneuvers the driver typically performs on this stretch or route. For example, it can be determined or learned that the driver only performs overtaking maneuvers at certain points on a regularly traveled stretch or route. Based on this, a determination, prediction, or plausibility check of the currently anticipated or intended driving maneuver can be carried out.

The second variant of the present invention can be combined with the other variant or other embodiments. For example, the determined or learned movement profile can be used as one of several inputs for determining the likely intended driving maneuver and/or, for example, for validating or verifying the plausibility of the likely intended driving maneuver. For example, both data on the movement profile or a resulting likely intended driving maneuver as well as the The driver's current or actual body movements and/or operating actions during the current journey, or a resulting likely intended driving maneuver, can be combined or compared. This enables a particularly precise and reliable determination of the actually intended driving maneuver.

In one possible embodiment of the present invention, the assistance system is configured to capture data, as part of the driver behavior data, from which momentary movements and/or operating actions of the driver during the current journey can be derived. In other words, movements and/or operating actions captured during the current journey, in particular during the current journey only within a sliding time window of a predetermined length extending from the current point in time into the past, can be used to determine the likely intended driving maneuver(s). Corresponding momentary movements of the driver can be or include, for example, head movements, glances to the side or over the shoulder or into a rearview mirror, positions or changes in position of hands and/or feet, a turning or straightening of the driver's torso or upper body, and/or the like. Such data can be or include, for example, camera data from an interior camera or driver observation camera of the motor vehicle and/or data or signals from operating elements or a user interface or a control unit of the motor vehicle coupled thereto. The embodiment of the present invention proposed here can achieve or ensure that, in every situation, at least particularly relevant driver behavior data is used or taken into account to determine the respective likely intended driving maneuver. For example, such momentary movements or operator actions by the driver can indicate an intended driving maneuver that contradicts the driver's long-term average behavior, and/or unusual behaviors or behavior patterns of the driver can be associated with a likely intended driving maneuver in a longer-term comparison.

In a further possible embodiment of the present invention, the assistance system is configured to use the environmental data to predict the expected future movement behavior of detected objects in the surroundings of the motor vehicle relative to the latter and to identify the relevant objects based on the respective prediction. The future movement behavior can be predicted, for example, using a predefined movement or traffic model and/or using a suitably trained machine learning device. For this purpose, positions, movement states, directions of movement, speeds, accelerations, activities, or states of turn indicators and/or brake lights and/or the like of other vehicles derived from or specified in the environmental data, as well as, for example, hand signals from cyclists or the like, can be taken into account. In particular, the future movement behavior of the detected objects can be predicted via a simpler extrapolation of the current movement state or beyond. For example, expected driving maneuvers, such as lane changes or turning maneuvers of other vehicles identified as objects in the surroundings, can be predicted and taken into account to identify the relevant objects. This enables particularly precise and reliable determination of the objects relevant to the driver's intended driving maneuver.

If there are several objects in the vicinity of the motor vehicle, their expected future movement behavior can be predicted jointly or in combination, for example by means of a corresponding group or swarm simulation. Likewise, the expected future movement behavior of one or more of the objects detected in the vicinity of the motor vehicle can be predicted individually or in subgroups. For example, the objects can be prioritized according to their relative speed to the motor vehicle and/or the lane they are traveling in. For example, objects that are moving at a relative speed to the motor vehicle that exceeds a predetermined threshold and/or are moving in a lane targeted by the driver or motor vehicle in accordance with the anticipated intended driving maneuver can be examined or classified as a priority. The expected future movement behavior of such objects can be predicted individually. Since this can be less computationally intensive, the driver can be warned particularly quickly or very early about such potentially relevant objects. Likewise, such objects, if they are classified or identified as relevant for the intended driving maneuver, can be highlighted over other relevant objects, i.e., marked or displayed differently, particularly more prominently. Thus, different objects can be highlighted depending on their relevance or - at least potential - danger can be marked or highlighted differently.

In a further possible embodiment of the present invention, the assistance system is configured to generate the warning message by controlling a mirror display of the motor vehicle to display a respective marking on the relevant objects visible in the mirror display as a warning message. A mirror display in the present sense can be, for example, a mirror configured as a display, i.e., as an actively controllable display device, in particular an interior rearview mirror, or a display or display device for a digital side mirror. In such a display for a digital side mirror, a camera image or video stream from an environmental observation camera of the motor vehicle arranged instead of a conventional side mirror can be displayed. Here, just as when the marking or warning message is output or displayed by means of another display device, a frame, a circle, a highlighted contour, or the like for a relevant object can be displayed on the respective mirror display as a marking or warning message.

Likewise, the direction of travel and/or the speed or relative speed of the respective object relative to the motor vehicle can also be displayed or indicated in addition to or as part of the marking. A relevant object can, for example, be outlined or highlighted in color, with the criticality or relevance, i.e., the degree of relevance of the respective object, being indicated in color. For example, a particularly relevant object, which, for example, represents a direct collision risk, can be marked red, while a relevant but less critical or less dangerous object, with which there is only a lower probability of collision, can be marked yellow or orange.

Additionally or alternatively, the assistance system can be configured to briefly display or color the entire image displayed in the respective mirror display - or generally in the respective display device - in a predetermined signal color when at least one relevant object is detected and/or to frame the entire displayed image with a frame, in particular in a predetermined signal color. Likewise, instead of a single color, a predetermined color sequence can be used for marking and/or displaying, coloring, or framing the entire image. The display of the marking of relevant objects in a mirror display proposed here, which can be used by the driver anyway to observe the surroundings of the motor vehicle, can enable the driver to recognize the relevant objects particularly easily, quickly, and with little distraction.

In a further possible embodiment of the present invention, the assistance system is configured to automatically increase or magnify the intensity or conspicuity of the warning compared to a respective initial display if or as soon as a predefined criterion is met. This can be the case, in particular, if no corresponding change in the driver's behavior or the likely intended driving maneuver is detected within a predefined period of time following the warning displayed in the initial display. The intensity or conspicuity of the warning can also be increased in several steps or stages according to a predefined escalation cascade. For example, to increase the intensity or conspicuity, the brightness and/or color and/or size and/or line type or dashed lines and/or a temporal display pattern of the warning can be adjusted. The latter can mean, for example, that the warning is initially displayed or displayed constantly or continuously over time and then flashing to increase the intensity or conspicuity. The proposed embodiment of the present invention can increase the likelihood that the driver will perceive the warning and react accordingly. This can contribute to further improving traffic safety using the present invention.

In general, the type and nature of the marking or warning, for example with regard to color and/or line thickness and/or line pattern and/or text, can be individually adjusted or designed based on at least one predefined property of the respective object. This can, for example, be predefined or learned automatically, for example based on an effectiveness that is subsequently recorded or evaluated - automatically and/or by the driver. The marking or warning for an object can also be dynamically adapted, i.e. changed, with changing relevance or with a changing value of at least one predefined property. For example, the conspicuity or intensity of the marking or warning can be automatically increased as the distance between the object and the motor vehicle decreases and/or as the approach speed between the object and the motor vehicle increases.

In a further possible embodiment of the present invention, the assistance system is configured to automatically control or activate at least one further device, for example of the motor vehicle or the assistance system, to output a further warning that addresses a different sense of the driver, if or as soon as a predefined criterion is met. This can be the case, in particular, if, following the initial warning, no corresponding change in the driver's behavior or the likely intended driving maneuver is detected within a predefined period of time. The output of one or more such further warnings proposed here can, for example, occur according to a predefined escalation cascade. If the original or initial warning is, for example, an optical or visual indication, for example on the mirror display mentioned elsewhere, then an audio indication or a haptic indication can be output as a further warning that addresses a different sense, i.e., not the driver's sense of sight. An audio cue can, for example, be or include a signal tone or a voice output indicating at least one detected relevant object or a corresponding hazard, or the like. A haptic cue can, for example, be a mechanical impulse or a vibration. Such a haptic cue can be generated by means of a corresponding actuator on or in a component of the motor vehicle that is touched by the driver or is in contact with the driver. Such a component can, for example, be a steering wheel or a seat or an armrest of the motor vehicle. The haptic cue can be generated at a location or on a side of the respective component or the driver that corresponds to the position of the detected relevant object relative to the motor vehicle.

As a criterion for issuing the further warning, the reaching or exceeding of a predetermined threshold value for the critical quality or relevance or danger or a collision probability when executing the anticipated intended driving maneuver can also be specified, used or evaluated. In an exemplary application of the embodiment of the present invention proposed here, the assistance system can, for example, cause a vibration of the steering wheel and/or the driver's seat of the motor vehicle and/or the output of a signal tone or a voice output. Likewise, the assistance system can control a ventilation system of the motor vehicle to issue the further warning, i.e., to draw the driver's attention to the current situation. In this way, the driver can, for example, be exposed to a blast of air, in particular from a direction corresponding to the position or direction of approach of the respective detected relevant object relative to the motor vehicle. By addressing various senses of the driver as proposed here, in particular in an escalating manner, i.e. by using various devices to issue warnings, the probability can be increased that the driver will recognise the actual or potential danger before actually carrying out the intended driving manoeuvre.

The present invention also relates to a motor vehicle having an environmental sensor system for detecting objects in the surroundings of the motor vehicle, a detection device, in particular a driver observation device, for detecting the behavior of the driver of the motor vehicle, i.e., corresponding driver behavior data, the assistance system according to the invention, and at least one output device coupled thereto for outputting warnings to the driver. The motor vehicle according to the invention can, in particular, be the motor vehicle mentioned in connection with the method according to the invention and/or in connection with the assistance system according to the invention, or correspond thereto. The motor vehicle according to the invention can therefore be configured for, in particular, automatically executing the method according to the invention.

The motor vehicle according to the invention and/or the assistance system according to the invention can be configured to automatically deactivate the assistance system or the described method, i.e., the corresponding function, in an efficiency or energy-saving mode. This can be applied, for example, in a battery-electric vehicle (BEV).

Further features of the invention can be derived from the following description of the figures and from the drawings. The features and combinations of features mentioned above in the description, as well as the features and combinations of features shown below in the description of the figures and/or in the figures alone, can be used not only in the respective combinations specified, but also in other combinations or on their own, without departing from the scope of the invention.

• 1 eine schematische Übersichtsdarstellung einer Verkehrssituation zum Veranschaulichen eines Verfahrens zum Warnen eines Fahrers eines Kraftfahrzeugs vor Gefahren; und
• 2 eine ausschnittweise schematische Darstellung eines Innenraums des Kraftfahrzeugs zum Veranschaulichen einer Möglichkeit zum Warnen des Fahrers.

The drawing shows:

• 1 a schematic overview of a traffic situation to illustrate a method for warning a driver of a motor vehicle of hazards; and
• 2 a partial schematic representation of an interior of the motor vehicle to illustrate a possibility for warning the driver.

Identical or functionally equivalent elements are provided with the same reference numerals in the figures.

1 shows a partial schematic overview of a traffic scene or a surrounding area with a section of a road that has a right lane 1, a center lane 2, and a left lane 3. These three lanes are intended for the same direction of travel. In the situation depicted here, a motor vehicle 4 and a rear-following vehicle 5 are traveling in the right lane 1.

The rear-following vehicle 5 can therefore be another vehicle that is following behind the motor vehicle 4. In the center lane 2, a center-following vehicle 6 is traveling, which, viewed in the direction of travel, is also traveling behind the motor vehicle 4. In the left lane 3, a left-following vehicle 7 is traveling in the same direction. For example, the left-following vehicle 7 is closer to the position of the motor vehicle 4, viewed in the direction of travel, but without having yet reached its position or overtaken it. Likewise, the left-following vehicle 7 could, for example, be at least substantially at the same height, i.e., at the same position in the direction of travel, as the motor vehicle 4.

The motor vehicle 4 is driven or controlled by a driver 8 in manual driving mode. For example, the driver 8 may currently be planning or intending a driving maneuver 9, indicated schematically here, to change from the right lane 1 to the center lane 2.

Current and future vehicles increasingly feature displays, i.e., display devices, for displaying the respective surroundings. For example, the motor vehicle 4 here has several mirror displays 10. Content, i.e., images and/or data, which were recorded or captured, for example, by an environmental sensor system 11 of the motor vehicle 4, can be shown on such displays or on the mirror displays 10. In the present case, however, it is not intended, or not intended only, to display such content, i.e., camera images or video data recorded, for example, by a camera used as part of the environmental sensor system 11, unprocessed on the mirror displays 10. Rather, the driver's 8 attention is intended to be directed, depending on the situation, to specific content or parts of content, or corresponding objects in the surroundings of the motor vehicle 4, which are important to him in the respective situation, i.e., for example, relevant for the safe execution of the respective intended driving maneuver 9.

For this purpose, the motor vehicle 4 also has a detection device 12 for detecting the behavior of the driver 8, i.e., corresponding driver behavior data, and an assistance system 13. The assistance system 13 can be connected, for example, via an interface 14 to an on-board electrical system of the motor vehicle 4, indicated schematically here. The assistance system 13 can thereby detect data, such as, for example, environmental data provided by the environmental sensor system 11 and driver behavior data provided by the detection device 12. To process this data, the assistance system 13 has, for example, a processor 15 and a computer-readable data memory 16. The assistance system 13 can, for example, control the mirror displays 10 or an additional output device 17 of the motor vehicle 4 via the interface 14. The additional output device 17 can, for example, be or include a loudspeaker and/or a mechanical or haptic actuator and/or a ventilation system or the like.

In this case, the environmental sensors 11 or the assistance system 13 can be used to observe the traffic to the rear or to the side of the motor vehicle 4 as seen from the perspective of the vehicle, i.e., to detect objects located there, in particular the rear-following vehicle 5, the middle-following vehicle 6, and the left-following vehicle 7. The environmental sensors 11 or the assistance system 13 can, for example, using artificial intelligence or a suitably trained machine learning device, determine what type of objects are involved and how they are currently moving, behaving, or will likely behave in the future. Furthermore, the assistance system 13 can recognize or determine the behavior or a behavior profile of the driver 8 during the current journey or in the respective current situation and/or over a longer period of time, i.e., in particular, over several journeys. The assistance system 13, in particular a correspondingly trained machine learning device provided therein, can combine the available data, i.e. in particular the driver behavior data and the respective current environmental data, and on the basis of the data in the respective situation, determine the driving maneuver 9 likely intended by the driver 8 and specifically relevant objects.

In the situation depicted here, it can be automatically detected, for example, that the driver 8 likely intends to steer the motor vehicle 4 into the center lane 2 according to the driving maneuver 9 indicated here. This likely intended driving maneuver 9 to change to the center lane 2 can be detected or predicted, for example, by the fact that the driver 8 has already steered the motor vehicle 4 to the left edge of the right lane 1 in the direction of travel and, for example, has already directed his gaze, in particular repeatedly, to the mirror display 10 acting as the left side mirror or, for example, has already positioned his hand to more easily reach a lever or control element for activating the left turn indicator of the motor vehicle 4. For example, the assistance system 13 or its machine learning facility may have learned from similar previous behaviors or behavior patterns of the driver 8 that this combination of behaviors typically precedes a lane change to the left.

Furthermore, it can be detected, for example, that the left-following vehicle 7 has activated its turn signal in the direction of the central lane 2 and that the central following vehicle 6 has activated its brake lights. The latter can be detected, for example, by a correspondingly red-illuminated ground area in the rear area of the central following vehicle 6 and/or by corresponding reflections or the like. From this, it can be concluded that the left-following vehicle 7 is likely to change into the central lane 2 and that the central following vehicle 6 will brake to enable this without a collision. The rear-following vehicle 5, on the other hand, is likely to continue driving in the right-hand lane 1, since it shows no signs of deviating behavior and there is insufficient space, for example, to change into the central lane 2.

Overall, the assistance system 13 can thus determine that, in the situation depicted here, the rear-following vehicle 5 and the center-following vehicle 6 are unlikely to interfere with the motor vehicle 4 during the execution of the driving maneuver 9, but that the left-following vehicle 7 represents or could pose a potential collision risk. Accordingly, the assistance system 13 can classify the rear-following vehicle 5 and the center-following vehicle 6 as secondary objects 18 that are not relevant for the safe execution of the driving maneuver 9. The left-following vehicle 7, on the other hand, can be classified as a hazardous object 19 that is relevant for the driving maneuver 9.

Since at least one relevant hazardous object 19 has been detected, the assistance system 13 can point this out or warn the driver 8.

To illustrate this, 2 A partial schematic representation of an interior of the motor vehicle 4. Visible here, in particular, is a mirror display 10, which has a central interior rearview mirror area and, to the sides thereof, side mirror areas 20. Several objects in the vicinity of the motor vehicle 4 are visible therein, which, as described, may each have been classified as secondary objects 18 or as hazardous objects 19.

In this case, the detected relevant hazardous object 19 is marked by a warning sign 21 at the request of the assistance system 13. The warning sign 21 not only identifies the relevant hazardous object 19 as such, but also indicates its direction of movement or its approach to the motor vehicle 4, for example, by a corresponding arrow and, if applicable, a corresponding coloring or color coding of the warning sign 21.

Furthermore, the assistance system 13 can, for example, activate or control the additional output device 17 - simultaneously with the output of the warning 21 or with a delay - in order to output a further indication to the driver 8 regarding the hazardous object 19 or the detected hazardous situation. For example, a sequence of corresponding warnings 21 or the warning 21 and one or more further warnings can be initiated or output according to a detected danger level or upon a detected increase in the danger level in the respective current situation. A corresponding indication can be individually dependent on the respective situation and/or the driver 8 or their behavior and/or preferences or learned reactions and/or corresponding user settings. The reactions of the assistance system 13 can therefore, for example, be based on experience learned from observing the driver 8 or adapted or carried out as specified in a user profile of the driver 8. Thus, in different situations and/or for different drivers 8, for example, an optical or visual warning 21 can first be issued and, for example if the driver 8 does not react to it, an acoustic and/or haptic further indication can be issued or, for example, both the visual or optical warning 21 and at least one other further indication can be issued initially.

If the warning or further information is an acoustic warning, its output and/or design can depend on the properties of the relevant object about which the driver is to be warned. Such properties can include, for example, the position and/or speed of the object relative to the motor vehicle and/or the direction in which the object is approaching the motor vehicle and/or the type of object. For example, an acoustic warning can be generated or output in such a way that a corresponding sound source appears to the driver to be located at the position of the relevant object and/or to be moving in accordance with the actual movement of the relevant object. 3D or surround effects or functions or binaural 3D audio synthesis can be used for this purpose. Likewise, a sound sequence or sound order of the acoustic warning or further information can be adapted or adjusted accordingly depending on the properties of the respective relevant object.

Overall, the examples described show how digital side mirrors and the content marked therein can be used to increase safety in traffic.

List of reference symbols

1

right lane

2

central lane

3

left lane

4

motor vehicle

5

Rear-following vehicle

6

Middle follower vehicle

7

Left-following vehicle

8

driver

9

Driving maneuvers

10

Mirror display

11

Environmental sensors

12

Recording device

13

Assistance system

14

interface

15

processor

16

Data storage

17

Additional output device

18

secondary object

19

Danger object

20

Side mirror areas

21

Warning

Claims (9)

Method for warning a driver (8) of a motor vehicle (4) of hazards in its surroundings, wherein, while the motor vehicle (4) is traveling, automatically - the surroundings of the motor vehicle (4) are continuously monitored for objects (5, 6, 7) located there, - the behavior of the driver (8) is continuously recorded, - a driving maneuver (9) likely intended by the driver is determined based on the recorded behavior of the driver (8), - those objects (5, 6, 7) detected in the surroundings of the motor vehicle (4) that are relevant to this driving maneuver (9) are determined, - a warning (21) is issued for only these specific relevant objects (7, 19), and a) an assistance system (13) for the motor vehicle (4) configured to execute the method receives driver behavior data indicating the behavior of the driver (8) of the motor vehicle (4), and corresponding driving maneuver data that were actually performed by the driver based on the recorded driver behavior data (8) specify the driving maneuvers (9) performed, are provided as training data for a machine learning device, and an output of the machine learning device trained thereby is used to determine or indicate the respective likely intended driving maneuver (9), and/or b) the assistance system (13) for the motor vehicle (4) configured to execute the method records routes traveled by the driver (8) over several journeys of the motor vehicle (4) as part of the driver behavior data, and a resulting movement profile of the driver (8) is taken into account to determine the respective likely intended driving maneuver (9). Assistance system (13) for a motor vehicle (4), comprising an input interface (14) for detecting environmental data and driver behavior data indicating a behavior of a driver (8) of the motor vehicle (4), a data processing device (15, 16) for processing the detected data to determine a likely intended driving maneuver (9) and relevant objects (7, 19) in the environment and to generate a control signal to generate a warning (21) only for precisely these relevant objects (7, 19) and an output interface (14) for outputting the control signal, wherein the assistance ency system (13) for carrying out the method according to Claim 1 is set up. Assistance system (13) according to Claim 2 , characterized in that the assistance system (13) is designed to record, as part of the driver behavior data, data from which current movements and/or operating actions of the driver (8) during the current journey emerge. Assistance system (13) according to one of the Claims 2 until 3 , characterized in that in variant b) the movement profile of the driver (8) is learned by means of machine learning. Assistance system (13) according to one of the Claims 2 until 4 , characterized in that the assistance system (13) is designed to use the environmental data to predict an expected future movement behavior of recognized objects (5, 6, 7) in the environment of the motor vehicle (4) relative to the latter and to determine the relevant objects (7, 19) based on the respective prediction. Assistance system (13) according to one of the Claims 2 until 5 , characterized in that the assistance system (13) is designed to control a mirror display (10) of the motor vehicle (4) to generate the warning (21) in order to display a respective marking on the relevant objects (7, 19) visible in the mirror display. Assistance system (13) according to one of the Claims 2 until 6 , characterized in that the assistance system (13) is designed to automatically increase an intensity of the warning (21) compared to a respective initial display as soon as a predetermined criterion is met, in particular if no corresponding change in the behavior of the driver (8) or the likely intended driving maneuver (9) is detected in response to the warning (21). Assistance system (13) according to one of the Claims 2 until 7 , characterized in that the assistance system (13) is designed to automatically control at least one further device for outputting a further warning which addresses a different sense of the driver (8) as soon as a predetermined criterion is met, in particular when no corresponding change in the behavior of the driver (8) or the likely intended driving maneuver (9) is detected in response to the warning (21). Motor vehicle (4), comprising an environmental sensor system (11) for detecting objects (5, 6, 7) in the surroundings of the motor vehicle (4), a detection device (12) for detecting a behavior of the driver (8) of the motor vehicle (4), an assistance system (13) according to one of the Claims 2 until 8 and an output device (10, 17) coupled thereto for outputting warning messages (21) to the driver (8).