DE102024207836A1 - METHOD FOR OPERATING A (PARTIALLY) AUTONOMOUSLY OPERATING CONTROL UNIT OF A VEHICLE, VEHICLE CONTROL SYSTEM AND VEHICLE - Google Patents

Abstract

The invention relates to a method (1) for operating an autonomously or semi-autonomously operating control unit (190) of a vehicle (2). The method (1) comprises the following steps: assigning (S1) permission to control the vehicle (2) to a passenger (400) of the vehicle (2); detecting (S2) a voice input from the passenger (400); recognizing (S3) a driving maneuver command based on the voice input from the passenger (400); generating and outputting (S4) an audio signal corresponding to the recognized driving maneuver command; detecting (S5) a signal output; determining (S6), based on the detected signal output, whether the audio signal was successfully output; and determining (S7) whether confirmation was given by the passenger (400). and when it has been determined that the audio signal has been reliably output and confirmation has been received from the passenger (400), the driving maneuver command is transmitted (S8) to the autonomously or semi-autonomously operating control unit (190) of the vehicle (2). Furthermore, the invention relates to a corresponding vehicle control system (3), vehicle (2), and storage medium.

Description

Field of invention

The invention relates to autonomous or semi-autonomous driving of a vehicle. In particular, the invention relates to a method for operating an autonomously or semi-autonomously operating control unit of a vehicle. Furthermore, the invention relates to a vehicle control system and a vehicle with such a vehicle control system.

background

Voice control systems in vehicles allow a driver to control a variety of vehicle functions. To authenticate the driver, there are numerous methods available that allow for personnel verification of safety-critical operations or instructions.

These user control options are reserved for the driver. However, if the driver is tired, distracted, or wishes to focus their attention elsewhere, the passenger should also be able to control driving instructions via voice input. Driving safety must still be guaranteed in this regard.

Summary

It is therefore an object of the invention to provide a method for a passenger to operate an autonomously or semi-autonomously functioning control unit of a vehicle, which in particular ensures driving safety. Furthermore, it is an object of the invention to provide a corresponding vehicle control system and a vehicle with such a vehicle control system.

One aspect of the invention relates to a method for operating an autonomously or semi-autonomously functioning control unit of a vehicle. This method can be executed by a vehicle control system.

The vehicle can be a road-bound vehicle, such as a passenger car, a truck, or a special-purpose vehicle. The vehicle can also be, for example, a boat or an aircraft. In the latter two cases, the words "driver" and "passenger" should be replaced by "captain" and "staff captain" or "pilot" and "co-pilot," respectively.

The autonomously or semi-autonomously operating control unit of the vehicle can in particular be understood as a control unit within the meaning of the SAE standard J3016, whereby "autonomously or semi-autonomously" specifically refers to levels 3, 4 and 5.

According to the procedure, authorization to control the vehicle is granted to a passenger. A passenger is defined here as a person who is in the vehicle and, due to their position within the vehicle, is able to control it by voice commands. In a passenger car, this can be determined, for example, by the fact that the person is sitting in the front passenger seat.

Furthermore, voice input from the passenger is recorded. Passenger identification can be achieved, for example, via a secure driver and passenger authentication unit, which may be based on iris recognition using an interior camera and/or continuous biometric voice recognition. The passenger's voice input can be captured contextually from an ongoing conversation or triggered by a detection signal, such as the voice input of a code word (e.g., "Hello, car") or the activation of a corresponding voice input button.

Based on the passenger's voice input, a driving maneuver command is then recognized. This recognition of the driving maneuver command can be achieved, for example, via speech recognition, syntax analysis, and/or natural language processing, preferably in conjunction with a list of possible driving behavior instructions.

When a driving maneuver command is recognized, an audio signal corresponding to that command is generated and output. This output is typically via loudspeakers, particularly those positioned near the passenger's head, for example, in or on the passenger seat's headrest. The audio signal corresponding to the recognized driving maneuver command can be retrieved from a database where each possible driving maneuver command is assigned a corresponding audio signal. Alternatively, the database can contain text snippets for the corresponding audio signal, and/or the audio signal can be generated using speech synthesis. The audio signal corresponding to the driving maneuver command can, in particular, include a brief description of the maneuver. This should include a statement that the driving maneuver order has been recognized and a request for confirmation.

While the audio signal corresponding to the recognized driving maneuver command is being emitted, a signal output is recorded, for example, by microphones. Based on this recorded signal output, it is then determined whether the audio signal was reliably emitted. For example, it can be checked whether the audio signal corresponding to the driving maneuver command is contained within the recorded signal output. If the audio signal corresponding to the driving maneuver command is drowned out by strong ambient noise, such as road noise or conversations, it can be determined that the audio signal was not reliably emitted.

The system then determines whether the passenger has confirmed the command. Only when it has been established that the audio signal was successfully transmitted and that the passenger has confirmed the command is the driving maneuver command forwarded to the vehicle's autonomous or semi-autonomous control unit.

This ensures that the driving maneuver command reflects the passenger's intention and that no incorrect driving maneuver command is executed.

In some embodiments, granting permission to control the vehicle to the passenger involves detecting a signal from a control button operated by the driver of the vehicle for the transfer of control. This control button may be a dedicated button used solely for granting permission to control the vehicle to the passenger, or it may be part of an input system that allows the driver, via a menu, to grant or transfer control of the vehicle to the passenger.

Alternatively or additionally, the authorization to control the vehicle can be granted to the passenger via voice input. For this purpose, the driver's voice input is recorded. Based on this voice input, a control transfer command is recognized. The recognition of this control transfer command can be analogous to the recognition of the driving maneuver command described above.

A further audio signal corresponding to the detected control transfer command is then generated and output. This can also be done analogously to the generation and output of the audio signal described above. The output of the further audio signal can preferably be via loudspeakers located near the driver's head, for example in or on a driver's seat headrest, and/or via loudspeakers located near the passenger's head, for example in or on the passenger's seat headrest.

Furthermore, another signal output is detected, and based on this detected signal output, it is determined whether the additional audio signal was successfully output. This is done analogously to the signal output detection described above and the determination, based on the detected signal output, of whether the audio signal was successfully output.

It is then determined whether further confirmation has been given by the driver and/or the passenger, and only if it has been determined that the further audio signal was definitely given and that confirmation has been given by the driver and/or passenger, is the permission to control the vehicle assigned to the passenger.

The additional audio signal to the driver and the driver's confirmation serve to ensure that the driver wishes to transfer control of the vehicle to the passenger. The additional audio signal to the passenger and the passenger's confirmation serve to ensure that the passenger also agrees to assume control of the vehicle.

In some embodiments, determining whether the passenger has given confirmation and/or determining whether the driver and/or passenger have given further confirmation involves detecting a further signal from a confirmation button operable by the passenger or driver. This confirmation button may be a dedicated button used solely for confirming the audio signals, or it may be part of an input system that allows the passenger or driver to confirm the audio signals via a menu.

Alternatively or additionally, determining whether the passenger has given confirmation and/or whether the driver and/or passenger have given further confirmation is done via a further voice input from the passenger or driver. In particular, the further voice input of the passenger or driver is recorded, and a confirmation command is recognized based on this further voice input.

In some embodiments, the method further includes relinquishing control of the vehicle from the passenger. This relinquishment of control can be carried out, in particular, by the driver and can be accomplished in various ways: for example, control can be relinquished by pressing the control button. Alternatively or additionally, control can be relinquished by a corresponding voice command. Again, alternatively or additionally, control can be relinquished by a manual vehicle operation, for example, when the driver operates the steering wheel and/or accelerator and/or brake pedals. In this way, the driver can regain control of the vehicle at any time.

In some embodiments, capturing the passenger's voice input, the signal output, the driver's voice input, further signal output, and/or further voice input from the passenger or driver involves capturing sound signals using a microphone array and evaluating the sound signals using a sound processing unit. The microphone array can comprise multiple microphones, which are positioned in the vehicle to optimally capture the driver's voice, driver's speakers, passenger's voice, and/or passenger's speakers. The captured sound signals are then processed in the sound processing unit. The sound processing unit can be configured to separate individual sound signals. For this purpose, the signals from the individual microphones in the microphone array can be combined (e.g., delayed and/or filtered) to amplify the driver's voice, the driver's speaker output, the passenger's voice, and/or the passenger's speaker output, while attenuating other sound sources inside and outside the vehicle. The processed signal is then passed on accordingly.

In some embodiments, the audio signal corresponding to the recognized driving maneuver command and/or the additional audio signal corresponding to the recognized control transfer command is generated with a characteristic modulation. Determining whether the audio signal and/or the additional audio signal has been reliably output is then based on this characteristic modulation. The characteristic modulation can be, in particular, imperceptible to humans, i.e., in a frequency range that is neither perceptible nor disturbing to the human ear at the moment of output. Specifically, the characteristic modulation can be in the near-ultrasound range, for example, in a frequency range between 25 and 30 kHz. The characteristic modulation can, for example, correspond to a predefined or randomly generated signal sequence that is added to the audible audio signal or the additional audio signal. If the characteristic modulation is then detected in the signal output or the additional signal output, it can be concluded that the audio signal or the additional audio signal has been reliably output. It should be noted that if the characteristic modulation occurs in a frequency range imperceptible to the human ear, the loudspeakers and microphones must be designed for playback or recording in this frequency range.

In some embodiments, the method further includes generating and outputting a graphic signal corresponding to the recognized driving maneuver command. This signal can, for example, be output to a visual display unit visible to the passenger. The graphic signal can, for example, include the driving maneuver command in written form in abbreviated form and/or a graphic animation of the driving maneuver command, for example, incorporating environmental information.

In some embodiments, the system also determines whether the passenger is attentive. The driving maneuver command is then only transmitted to the autonomous or semi-autonomous control unit if it has been determined that the passenger is attentive. This further enhances driving safety. Attentiveness can be determined, for example, by analyzing microphone signals, such as whether the voice sounds alert or sleepy. Alternatively or additionally, data from an interior camera can be used to determine attentiveness, analyzing, for example, the passenger's position and facial features. Alternatively or additionally, sensors (e.g., touch or proximity sensors) in the passenger headrest can provide information about the passenger's head position and thus their level of attentiveness.

In some embodiments, the method further includes an evaluation of the driving maneuver command. This evaluation can be performed, in particular, by the vehicle's autonomous or semi-autonomous control unit. The driving maneuver command can be assessed for feasibility, for example, using image signals from an external camera and/or signals from other sensors such as radar or lidar.

A further audio signal corresponding to the assessment of the driving maneuver command is then generated and output. Additionally, a graphic signal corresponding to the assessment of the driving maneuver command can also be output. In particular, the further audio signal can include information on whether the driving maneuver command is feasible or not, in order to prepare the driver and/or passenger for whether the driving maneuver command will be carried out or not.

To verify the successful output of the additional audio signal, a further signal output is detected, and based on this detection, it is determined whether the additional audio signal was successfully output. Details are provided analogously to the description above.

If the driving maneuver command has been assessed as feasible and it has been determined that the additional audio signal has been reliably output, the driving maneuver command will be executed.

Another aspect of the invention relates to a vehicle control system. This vehicle control system can be a dedicated system for controlling an autonomously or semi-autonomously operating control unit of a vehicle, but it can also be part of a vehicle system. The vehicle control system comprises a processing unit, an interface to at least one loudspeaker, and an interface to a microphone array, and is configured to carry out the method according to the preceding description. Advantages, further embodiments, and details will become apparent from the preceding description.

A further aspect of the invention relates to a vehicle comprising a vehicle control system as described above. The vehicle can be a road-bound vehicle, such as a passenger car, a truck, or a special-purpose vehicle. The vehicle can also be, for example, a boat or an aircraft. Advantages, further embodiments, and details will become apparent from the preceding description.

A further aspect of the invention relates to a non-volatile, computer-readable storage medium containing a program which, when executed on a computing unit of a vehicle control system as described above, instructs the vehicle control system to execute the method described above. Advantages, further embodiments, and details will become apparent from the preceding description.

1. 1) Der Fahrer muss die Eingabe von mündlich zu erteilenden Fahrverhaltensanweisungen dem Beifahrer erlauben oder diesem jederzeit wieder entziehen können.
2. 2) Der Beifahrer muss die Erlaubnis zur Eingabe von Fahrverhaltensanweisungen eindeutig und zweifelsfrei bestätigen.
3. 3) Der Fahrer muss jederzeit dem Beifahrer die Erlaubnis zur Erteilung von Fahrverhaltensanweisungen durch eine manuelle Fahroperation und/oder per Sprachanweisung entziehen können.
4. 4) Die Fahrverhaltensanweisung des Beifahrers muss eindeutig erkannt werden und darf nicht mit einer anderen möglichen Fahrverhaltensanweisung verwechselt werden.
5. 5) Das Fahrzeugsystem muss sicherstellen, dass eine Fahrverhaltensanweisung dem Fahrzeuginsassen korrekt zugeordnet wird, der sie gegeben hat (Beifahrer oder Fahrer).
6. 6) Das Fahrzeugsystem muss sicherstellen, dass eine Fahrverhaltensanweisung von einem Fahrzeuginsassen nur dann berücksichtigt wird, wenn dieser dafür vorher autorisiert worden ist.
7. 7) Wenn Beifahrer und Fahrer gleichzeitig eine mündliche Fahrverhaltensanweisung geben, sollten beide Fahrverhaltensanweisungen erkannt und bewertet werden.
8. 8) Bei gleichzeitigen mündlichen Fahrverhaltensanweisungen darf nur eine vom Fahrzeugsystem als gültig akzeptiert und zur Bestätigung ausgegeben werden.
9. 9) Eine Bestätigung der Fahrverhaltensanweisung darf nur von dem anweisenden Sprecher entgegengenommen werden.
10. 10) Eine Bestätigung kann manuell oder optional durch eine sichere Spracherkennung eines Schlüsselwortes oder Satzes des Anweisenden erfolgen.
11. 11) Die teilautonome oder autonome Steuereinheit des Fahrzeugs soll sicherstellen, dass der autorisierte Beifahrer die aktuelle Fahrsituation vor und während seiner Spracheingabe visuell erfassen kann.
12. 12) Optional kann es erforderlich sein, zusätzlich die Aufmerksamkeit des Beifahrers vor Ausführung einer Fahrverhaltensanweisung zu überprüfen, z.B. ob dieser nicht müde, abgelenkt oder alkoholisiert ist.
13. 13) Die akustische Umgebung (z.B. unter Berücksichtigung von Störgeräuschen) muss eine eindeutige und einwandfreie Kommunikation zwischen Fahrer und Beifahrer und dem Spracherkennungssystem des Fahrzeugs erlauben.
14. 14) Unter Berücksichtigung aller dem autonomen Fahrzeugsteuerungssystem bekannten Informationen und möglichen Risiken muss eine eindeutig erkannte und bestätigte Fahrverhaltensanweisung vom autonomen Fahrzeugsteuerungssystem auf ihre sichere Durchführbarkeit beurteilt werden und im Falle der nicht sicheren Durchführbarkeit abgelehnt werden.
15. 15) Dabei muss im Falle der Fahrverhaltensanweisung durch den Beifahrer die reduzierte oder fehlende manuelle Beherrschbarkeit des Fahrzeugs durch den Beifahrer berücksichtigt werden, was die möglichen Fahrverhaltensanweisungen für den Beifahrer einschränken kann.
16. 16) Im Falle der sicheren Durchführbarkeit muss die Fahrverhaltensanweisung sicher ausgeführt werden können, andernfalls muss die Ausführung verweigert werden.
17. 17) Es muss eine Rückmeldung an die Fahrzeuginsassen über die Durchführbarkeit bzw. Undurchführbarkeit der Fahrverhaltensanweisung erfolgen.

The above-described method, vehicle control system, vehicle and storage medium solve, among other things, the following problems with safety-critical or safety-relevant voice input to the autonomously or semi-autonomously operating control unit of the vehicle:

1. 1) The driver must be able to allow the passenger to enter verbal driving instructions or to withdraw them at any time.
2. 2) The passenger must clearly and unambiguously confirm permission to input driving instructions.
3. 3) The driver must be able to withdraw the passenger's permission to give driving instructions at any time by means of a manual driving operation and/or by voice command.
4. 4) The passenger's driving instruction must be clearly identifiable and must not be confused with any other possible driving instruction.
5. 5) The vehicle system must ensure that a driving instruction is correctly assigned to the vehicle occupant who gave it (passenger or driver).
6. 6) The vehicle system must ensure that a driving instruction from a vehicle occupant is only taken into account if the occupant has been previously authorized to do so.
7. 7) If the passenger and driver give verbal driving instructions simultaneously, both driving instructions should be recognized and evaluated.
8. 8) If oral driving instructions are given simultaneously, only one may be accepted as valid by the vehicle system and issued for confirmation.
9. 9) Confirmation of the driving instruction may only be accepted by the instructing speaker.
10. 10) Confirmation can be given manually or optionally by secure speech recognition of a keyword or phrase from the instructor.
11. 11) The semi-autonomous or autonomous control unit of the vehicle shall ensure that the authorized passenger can visually perceive the current driving situation before and during his voice input.
12. 12) Optionally, it may be necessary to additionally check the attention of the passenger before carrying out a driving instruction, e.g. whether he is tired, distracted or under the influence of alcohol.
13. 13) The acoustic environment (e.g. taking into account background noise) must allow clear and unambiguous communication between the driver and passenger and the vehicle's voice recognition system.
14. 14) Taking into account all information and potential risks known to the autonomous vehicle control system, a clearly identified and confirmed driving instruction must be assessed by the autonomous vehicle control system for its safe feasibility and rejected if it is not safe.
15. 15) In the case of driving instructions from the passenger, the reduced or non-existent manual control of the vehicle by the passenger must be taken into account, which may limit the possible driving instructions for the passenger.
16. 16) If the driving instruction can be carried out safely, it must be possible to execute it safely; otherwise, execution must be refused.
17. 17) Feedback must be given to the vehicle occupants regarding the feasibility or imfeasibility of the driving instructions.

For further clarification, the invention is described with reference to embodiments illustrated in the figures. These embodiments are to be understood as examples only, and not as limitations.

Brief description of the characters

• 1 ein Ablaufdiagramm für ein Verfahren zum Bedienen einer Steuereinheit eines Fahrzeugs;
• 2 eine schematische Darstellung eines Fahrzeugs mit einem Fahrzeugkontrollsystem;
• 3 eine detaillierte schematische Darstellung eines Fahrzeugkontrollsystems; und
• 4 ein Ablaufdiagramm für ein Computerprogramm.

This shows:

• 1 a flowchart for a procedure for operating a vehicle control unit;
• 2 a schematic representation of a vehicle with a vehicle control system;
• 3 a detailed schematic representation of a vehicle control system; and
• 4 A flowchart for a computer program.

Detailed description of embodiments

In the figures, identical reference symbols denote either identical elements or elements with equivalent functions. Elements that have already been described are not necessarily described again in subsequent figures.

1 shows a flowchart for procedure 1 for operating an autonomously or semi-autonomously operating control unit of a vehicle.

The vehicle can be a road-bound vehicle, such as a passenger car, a truck, or a special-purpose vehicle. The vehicle can also be, for example, a boat or an aircraft. In the latter two cases, the words "driver" and "passenger" should be replaced by "captain" and "staff captain" or "pilot" and "co-pilot," respectively.

The autonomously or semi-autonomously operating control unit of the vehicle can in particular be understood as a control unit within the meaning of the SAE standard J3016, whereby "autonomously or semi-autonomously" specifically refers to levels 3, 4 and 5.

Procedure 1 involves assigning S1 permission to control the vehicle to a passenger. A passenger is defined here as a person who is in the vehicle and, due to their position within the vehicle, is able to control it by voice commands. In a passenger car, this can be determined, for example, by the fact that the person is sitting in the front passenger seat.

Assigning S1 the permission to control the vehicle to the passenger may include detecting a signal from an operating button that can be operated by a driver of the vehicle for the transfer of control of the vehicle.

Alternatively, or additionally, assigning S1 permission to control the vehicle to the co-driver may include the following: detecting a voice input from the driver; recognizing a control transfer command based on the driver's voice input; generating and outputting a further audio signal corresponding to the recognized control transfer command; detecting a further signal output; determining, based on the detected further signal output, whether the further audio signal was safely output; determining whether further confirmation was given by the driver and/or co-driver; and, if it has been determined that the further audio signal was safely output Once the authorization has been given and confirmation has been received from the driver and/or passenger, permission to control the vehicle is assigned to the passenger.

Furthermore, procedure 1 includes capturing S2 a voice input from the passenger. Passenger recognition can be achieved, for example, via a secure driver and passenger authentication unit, which may be based on iris recognition using an interior camera and/or continuously operating biometric voice recognition. The passenger's voice input can be captured contextually from an ongoing conversation or can be triggered by a detection signal, such as the voice input of a codeword (e.g., "Hello, car") or the activation of a corresponding voice input button.

Based on the passenger's voice input, a driving maneuver command recognition (S3) is performed. This recognition (S3) of the driving maneuver command can be achieved, for example, via speech recognition, syntax analysis, and/or natural language processing, preferably in conjunction with a list of possible driving behavior instructions.

Furthermore, the procedure includes generating and outputting S4 an audio signal corresponding to the recognized driving maneuver command. Output is primarily via loudspeakers, particularly loudspeakers located near the passenger's head, for example, in or on a headrest of the passenger seat. The audio signal corresponding to the recognized driving maneuver command can be retrieved from a database in which a corresponding audio signal is assigned to each possible driving maneuver command. Alternatively, the database can contain text modules for the corresponding audio signal, and/or the audio signal corresponding to the driving maneuver command can be generated via speech synthesis. The audio signal corresponding to the driving maneuver command can include, in particular, a brief description of the driving maneuver command, a statement that this driving maneuver command has been recognized, and a request for confirmation.

While the audio signal corresponding to the recognized driving maneuver command is being emitted, a signal output is detected (S5). Based on this detected signal output, a determination (S6) is then made as to whether the audio signal was reliably emitted. For this purpose, it can be checked, for example, whether the audio signal corresponding to the driving maneuver command is contained within the detected signal output. If the audio signal corresponding to the driving maneuver command is drowned out by strong ambient noise, such as road noise or conversations, it can be determined that the audio signal was not reliably emitted.

For this purpose, the audio signal corresponding to the recognized driving maneuver command can be generated with a characteristic modulation, and the determination S6 of whether the audio signal was safely output can be carried out based on this characteristic modulation.

Furthermore, a graphical signal corresponding to the recognized driving maneuver command can be generated and output.

Furthermore, the procedure includes determining S7 whether confirmation has been given by the passenger.

Determining whether the passenger has given confirmation (S7) and/or determining whether the driver and/or passenger has given further confirmation may include detecting a further signal from a confirmation button operable by the passenger or driver.

Alternatively, or additionally, determining whether confirmation has been given by the passenger and/or determining whether further confirmation has been given by the driver and/or passenger may include: capturing further voice input from the passenger or driver; and recognizing a confirmation command based on the further voice input from the passenger or driver.

Once it has been determined that the audio signal has been safely output and confirmation has been received from the passenger, the driving maneuver command S8 is forwarded to the autonomously or semi-autonomously operating control unit of the vehicle.

This ensures that the driving maneuver command reflects the passenger's intention and that no incorrect driving maneuver command is executed.

When capturing the passenger's voice input, capturing the signal output, capturing the driver's voice input, capturing further signal output, and/or capturing further voice input from the passenger or driver, sound signals can be captured using a microphone field and evaluated using a sound evaluation unit.

In another embodiment, not shown here, method 1 can additionally This includes taking control of the vehicle away from the passenger.

In a further embodiment, not shown here, the method 1 can additionally include determining whether the passenger is attentive, whereby the transmission of the driving maneuver command to the autonomously or semi-autonomously operating control unit only takes place if it has been determined that the passenger is attentive.

In a further embodiment, not shown here, the method 1 may additionally include: evaluating the driving maneuver command; generating and outputting a further audio signal corresponding to the evaluation of the driving maneuver command; detecting a further signal output; determining, based on the detected further signal output, whether the further audio signal was reliably output; and if the driving maneuver command was evaluated as feasible and it was determined that the further audio signal was reliably output, executing the driving maneuver command.

2 Figure 1 shows a schematic representation of a vehicle 2 with a vehicle control system 3. The vehicle control system 3 comprises a computing unit 4 and interfaces (not shown here) to at least one loudspeaker 103, 403 and to a microphone array 5, comprising a plurality of microphones 303. The vehicle control system 3 is specifically designed to execute the method 1.

3 shows a detailed schematic representation of a vehicle control system as well as parts of a vehicle interior.

A driver's seat 101 is suitable for a person, in particular a driver 100, who can control the vehicle from this seat via user controls, for example lateral and longitudinal user controls such as a steering wheel 111 or foot pedals 112 for braking and accelerating.

A passenger seat 401 is also suitable for a person, in particular a passenger 400, who can oversee a traffic situation from this passenger seat 401, for example through a windshield.

Near the driver's seat 101, an easily accessible confirmation button 110 is located for the driver 100. This button allows the driver 100 to transfer vehicle control to the passenger 400 or reactivate vehicle control. The activation button 110 is located, for example, on the steering wheel 111. A signal B from the activation button 110 can be transmitted to a secure dialogue control unit 150.

The driver's seat 101 may also include a driver's headrest 102 which has a device that detects the driver's 100 sitting upright and/or leaning his head against the driver's headrest 102, for example by means of a switch or a proximity sensor.

The front passenger seat 401 can also include a front passenger headrest 402, which likewise has a device that detects whether the front passenger 400 is sitting upright and/or leaning their head against the front passenger headrest 402, for example via a switch or a proximity sensor. If the front passenger 400's head is leaning against the front passenger headrest 402, this can be transmitted as signal K to the safe dialogue control 150.

The above-described determination of the head's position against the headrest serves to recognize that the driver (100) or the passenger (400) has their head in a position in which they can observe the traffic situation.

Alternatively or in addition to head position detection by the headrests 102, 402, an interior camera 120 can be provided for monitoring the driver 100 and front passenger 400, wherein the data from the interior camera 120 are processed to determine the position of the head of the driver 100 or front passenger 400, and thus to recognize whether the driver 100 or front passenger 400 has their head in a position in which they can observe the traffic situation.

Furthermore, one or more driver loudspeakers 103 are arranged on or near the head of the driver 100, for example in the driver's headrest 102. Likewise, one or more passenger loudspeakers 403 are arranged on or near the head of the passenger 400, for example in the passenger's headrest 402.

Furthermore, a microphone array consisting of several microphones 303 is arranged in the vehicle. These microphones 303 are preferably arranged such that they can optimally capture the voice of the driver 100, output from the driver's loudspeaker 103, the voice of the passenger 400, and output from the passenger's loudspeaker 403.

The signals captured by the microphones 303 can be forwarded to a sound evaluation unit 302 and processed there. In particular, the signals from the individual microphones can be 303 are combined with each other to form a combined signal Ma, for example by delaying or filtering, such that the voice of the driver 100 and the outputs of the driver's loudspeaker 103 or the voice of the passenger 400 and the outputs of the passenger's loudspeaker 403 are amplified and other sound sources inside and outside the vehicle are dampened.

The combined signal Ma can then be forwarded, for example, to a secure audio output 130, to a speech recognition 140, to an attention detection 160 and/or to a driver and passenger authentication unit 170.

The secure audio output 130 generates an audio signal Sa, which can be reliably identified, for example, by means of a characteristic modulation from the microphone signal Ma that is not perceptible to humans, so that it can be ensured that the output audio signal Sa is a speech reproduction of a digital input signal, e.g. a driving maneuver command Fa.

The speech recognition system 140 recognizes instructions from the driver or passenger in the combined microphone signal.

The safe dialogue control 150 concludes from the recognized voice instruction Ft of the driver or passenger, if a clear assignment is possible from a list Lf of possible driving behavior instructions specified by an autonomous vehicle control unit 190, that a specific driving maneuver command Fa.

The specific driving maneuver command Fa is passed on to the safe audio output 130, which uses a database to output a modulated audio signal Sa defined for the driving maneuver command Fa to the driver 100 and/or passenger 400 and reports the success of the audio output back to the safe dialogue control 150 via a signal E.

The driving maneuver command Fa is optionally transmitted to a visual output unit 180 visible to the passenger 400 and/or driver 100, which graphically displays the driving maneuver command Fa in an animated manner, taking into account the environmental information contained in the driving maneuver command (e.g., overtaking a truck in front).

The secure driver and passenger authentication unit 170 can, for example, be implemented based on iris recognition using the interior camera 120 and continuous biometric voice recognition. If authentication fails (e.g., of the driver's or passenger's voice), a signal Na can be forwarded to the secure dialogue control unit 150.

The attention detection system 160 can, for example, assess the attention of the driver or passenger based on the combined microphone signals Ma and interior camera image data Kd and, for example, communicate this to the safe dialogue control system 150 via a signal Sh if fatigue is detected, which then prevents the output and execution of a driving behavior instruction from the driver or passenger.

If no fatigue is present, the safe dialogue control 150 can issue a confirmed driving maneuver command Af to the autonomously or semi-autonomously operating vehicle control unit 190. The autonomously or semi-autonomously operating vehicle control unit 190 then evaluates the confirmed driving maneuver command Af for feasibility, for example, using image signals Sk from an external camera 191 and/or signals S from other sensors 192 such as radar or lidar.

A feasibility result Ef is then passed to the safe dialogue control 150, which forwards the result Ef to the safe audio output 130 and/or the visual output unit 180 for output to the passenger or driver.

If the confirmed driving maneuver command Af is feasible, it is executed by issuing control signals Ak using actuators 200 of the vehicle.

The secure dialogue control 150 is designed to allow the driver to designate the passenger 400 as responsible for the vehicle's driving behavior via voice input 140 and/or haptic input 110, for example, with the voice command "Hand over the steering wheel to the passenger." To do this, the secure dialogue control 140 analyzes the voice input to determine the most probable instruction from a list Lf of possible instructions using a suitable method, such as syntax analysis or natural language processing. If the passenger authorization is correctly recognized using this method, the passenger 400 is prompted to confirm it via a secure audio output 130, for example, with the voice output "Passenger XY, please confirm the transfer of control of the vehicle behavior!" The passenger 400 can do this securely either haptically using a passenger activation button 410 or via voice input with authentication 170 and the input of a key phrase that is stored in the which does not occur in other language areas, do (e.g. “Confirm, Ha-Four-Te-Two-Nine”).

If the voice input cannot be clearly assigned to a possible driving instruction, or if confirmation is not given or not clearly given by the passenger 400, the safe dialogue control 150 can repeat the driving dialogue.

The safe dialogue control 150 prevents the execution of a command by the passenger 400 if the driver 100 makes a haptic input 110 or a manual vehicle operation using the steering wheel 111 or the pedals 112, so that the driver 100 can revoke the passenger 400's permission to issue driving instructions at any time. This can optionally also occur if the driver makes a voice input initiated by a key phrase.

The driving instruction of the passenger 400 is always reliably recognized because it is clearly assigned by the safe dialogue control 150 to a possible driving instruction of the autonomous vehicle control unit 190, this is made audible to the passenger 400 by means of a safe voice output and is then only executed upon confirmation by the passenger 400.

The driving behavior instruction is only accepted by the passenger 400 from the safe dialogue control 150 if the passenger can correctly perceive the driving situation, which is ensured by the proximity sensor signals or switches in the passenger headrest 402 and/or by the interior camera 120 with image recognition of the passenger 400's head.

Optionally, the passenger's driving instruction 400 can be refused if the attention detection 160 or the interior camera 120 detects fatigue, absence or unfitness to drive (e.g. due to slurred speech, slowed speech input and/or closed eyes or strongly dilated pupils).

Each driving maneuver command Af confirmed by the safe dialogue control 150 is assessed for its safe feasibility as input to the at least partially autonomous vehicle control unit, based on the sensors 191, 192 and the driving, traffic, and environmental situation generated from these signals. The command is approved if its execution is deemed sufficiently safe. This decision Ef is communicated to the safe dialogue control 150 along with a possible reason for rejection. The control unit then either confirms or rejects the execution via a safe audio output 130 and, optionally, a visual output to the driver 100 and passenger 400.

In the event of loud ambient noise, the safe audio output 130, for example, detects excessive ambient noise based on the ratio of the combined power of the microphone signals and the audio signals output by the loudspeakers to the total power of the microphone signals over a period relevant for communication (e.g., 50 ms). It then terminates the safe voice output and notifies the safe dialogue control 150 of this termination via signal E. The safe dialogue control 150 then safely terminates the driving dialogue, preventing any driving instructions from being executed under unsafe acoustic communication conditions.

The safe dialogue control 150 always communicates with a confirmed driving maneuver command Af to the at least semi-autonomously operating vehicle control unit 190 whether it is an instruction from the passenger 400 or the driver 100, so that it can reject these instructions from the passenger 400 due to lack of controllability of the vehicle by the passenger 400 or delayed controllability of the inactive driver 100.

4 shows a flowchart for a computer program that implements safe dialog control.

At step 00, the safe driver dialogue is aborted and an abort message Ef is sent to the safe audio output 130 and optionally to the visual output unit 180.

Step 1 checks whether a fully recognized speech input (Ft) has been received. If not, step 1 is repeated; if so, the process continues with step 2.

In step 02, the speech input Ft is analyzed for the most probable instruction from a list Lf of possible instructions using a suitable method, e.g., syntax analysis or natural language processing. The selected instruction Af is then output to the safe audio output 130.

Step 3 checks whether attention to attention control has been positively confirmed. If not, the process returns to step 00; if so, it proceeds to step 4.

In step 04, positive attention is noted for the recognized driving instruction Af.

Step 5 checks whether passenger or driver authentication is present. If not, the process returns to step 00; if so, it proceeds to step 06.

In step 06, a positive authentication is recorded for the recognized driving instruction Af, either for the driver 100 or the passenger 400.

Step 7 checks whether a list of possible driving instructions Lf from the at least partially autonomous vehicle control unit 190 can be read. If not, the process returns to step 00; if yes, it continues with step 08.

In step 08, the possible driving instructions Lf are read in.

Step 9 checks whether the signal K from the proximity sensor of the passenger headrest 402 indicates that the passenger's head 400 is near the passenger headrest 402. If not, the process returns to step 00; if yes, it proceeds to step 10.

In step 10, a positive overview of the passenger 400 is noted for the recognized driving instruction Af.

Step 11 checks whether signal P indicates that the driver has activated the vehicle pedals 112. If so, the process returns to step 00; if not, it continues with step 12.

In step 12, no manual operation of the accelerator pedals is recorded for the recognized driving instruction Af.

Step 13 checks whether signal L indicates that the driver has actuated the handlebar 111. If yes, the process returns to step 00; if no, it continues to step 14.

In step 14, for the recognized driving instruction Af, no steering input from the driver is recorded, and the system waits for the end of the safe acoustic voice output Fa that corresponds to the recognized driving instruction Af.

Step 15 checks whether signal E of the safe voice output indicates the correct acoustic output Fa for the driving instruction Af. If not, the system returns to step 00; if yes, it proceeds to step 16.

In step 16, the safe voice output for the recognized driving instruction Af is recorded and confirmation from the passenger is awaited by pressing button B4 or a confirming voice input Ft.

Step 17 checks whether a confirmation corresponding to step 16 is received within a time period T (e.g., 2 seconds). If not, the process returns to step 00; if so, it continues with step 18.

In step 18, the confirmation of the passenger is noted for the recognized driving instruction Af.

Step 19 checks whether all positively recorded signals from steps 4, 6, 10, 12, 14, 16, and 18 are present. If not, the process returns to step 1; if so, it continues with step 20.

In step 20, the driving behavior instruction Af, which was reliably detected by the passenger 400, is output to the at least partially autonomous vehicle control unit 190.

Reference symbol list

1

Procedure for operating a control unit

2

vehicle

3

Vehicle control system

4

computing unit

5

Microphone field

100

driver

101

driver's seat

102

Driver's headrest

103

Driver speaker

110

Actuating button

111

steering wheel

112

Foot pedals

120

Interior camera

130

secure audio output

140

Speech recognition

150

secure dialogue control

160

Attention recognition

170

Driver and passenger authentication unit

180

visual output unit

190

autonomous vehicle control unit

191

Outdoor camera

192

additional sensors

200

Actuators

302

Sound evaluation unit

303

microphone

400

passenger

401

passenger seat

402

passenger headrest

403

passenger speaker

410

Passenger control button

Af

confirmed driving maneuver order

Ak

Control signals

B

Signal from the activation button

E

signal

Ef

Decision on implementation

fa

Driving maneuver order

Ft

Instructions from the driver or passenger

K

Passenger headrest signal

Kd

Interior camera image data

Lf

List of possible driving behavior instructions

Ma

combined signal

N/a

Non-authentication signal

S

Signals from other sensors

Sat

audio signal

Sh

Fatigue signal

Sk

Image signals from the outdoor camera

S1

Assigning a permit

S2

Capturing a voice input

S3

Recognizing a driving maneuver command

S4

Generating and outputting an audio signal

S5

Capturing a signal output

S6

Determining a safe audio signal output

S7

Determining a confirmation

S8

Passing on the driving maneuver order

Claims (14)

A method (1) for operating an autonomously or semi-autonomously operating control unit (190) of a vehicle (2), comprising: Assigning (S1) permission to control the vehicle (2) to a passenger (400) of the vehicle (2); Capturing (S2) a voice input from the passenger (400); Recognizing (S3) a driving maneuver command based on the voice input from the passenger (400); Generating and outputting (S4) an audio signal corresponding to the recognized driving maneuver command; Capturing (S5) a signal output; Determining (S6), based on the captured signal output, whether the audio signal was safely output; Determining (S7) whether confirmation was given by the passenger (400); and when it has been determined that the audio signal has been safely output and confirmation has been received from the passenger (400), the driving maneuver command is passed (S8) to the autonomous or semi-autonomous control unit (190) of the vehicle (2). The procedure (1) according to Claim 1 , wherein the assignment (S1) of permission to control the vehicle (2) to the passenger (400) includes: detecting a signal from an operating button (110) operable by a driver (100) of the vehicle (2) for the transfer of control of the vehicle (2). The procedure (1) according to Claim 1 or 2 , wherein the assignment (S1) of permission to control the vehicle (2) to the co-driver (400) comprises: detecting a voice input from the driver (100); recognizing a transfer of control command based on the voice input from the driver (100); generating and outputting a further audio signal corresponding to the recognized transfer of control command; detecting a further signal output; determining, based on the detected further signal output, whether the further audio signal was safely output; determining whether further acknowledgment has been given by the driver (100) and/or co-driver (400); and, if it has been determined that the further audio signal was safely output and acknowledgment has been given by the driver (100) and/or co-driver (400), assigning permission to control the vehicle (2) to the co-driver (400). The procedure (1) according to one of the Claims 1 until 3 , wherein determining (S7) whether confirmation has been given by the passenger (400) and/or determining whether further confirmation has been given by the driver (100) and/or passenger (400) includes: detecting a further signal from a confirmation button operable by the passenger (400) or driver (100). The procedure (1) according to one of the Claims 1 until 4 , where determining (S7) whether confirmation has been given by the passenger (400) and/or the Determining whether further confirmation has been given by the driver (100) and/or passenger (400) includes: capturing further voice input from the passenger (400) or driver (100); and recognizing a confirmation command based on the further voice input from the passenger (400) or driver (100). The procedure (1) according to one of the Claims 1 until 5 , furthermore comprehensively: taking control of the vehicle (2) away from the passenger (400). The procedure (1) according to one of the Claims 1 until 6 , wherein the acquisition (S2) of the passenger's voice input (400), the acquisition (S5) of the signal output, the acquisition of the driver's voice input (100), the acquisition of further signal output, and/or the acquisition of further voice input from the passenger (400) or driver (100) includes: acquisition of sound signals by means of a microphone field (5) and evaluation of the sound signals by means of a sound evaluation unit. The procedure (1) according to one of the Claims 1 until 7 , wherein the audio signal corresponding to the recognized driving maneuver command and/or the further audio signal corresponding to the recognized control transfer command is generated with a characteristic modulation and the determination (S6) of whether the audio signal was safely output and/or the determination of whether the further audio signal was safely output is based on this characteristic modulation. The procedure (1) according to one of the Claims 1 until 8 , furthermore comprehensive: generating and outputting a graphic signal corresponding to the recognized driving maneuver command. The procedure (1) according to one of the Claims 1 until 9 , furthermore comprehensively: Determine whether the passenger (400) is attentive, whereby the transmission (S8) of the driving maneuver command to the autonomously or semi-autonomously operating control unit (190) only takes place if it has been determined that the passenger (400) is attentive. The procedure (1) according to one of the Claims 1 until 10 , furthermore, encompassing: evaluating the driving maneuver command; generating and outputting a further audio signal corresponding to the evaluation of the driving maneuver command; detecting a further signal output; determining, based on the detected further signal output, whether the further audio signal was reliably output; and if the driving maneuver command was deemed feasible and it was determined that the further audio signal was reliably output, executing the driving maneuver command. A vehicle control system (3) comprising a computing unit (4); an interface to at least one loudspeaker (103; 403); and an interface to a microphone array (5), wherein the vehicle control system (3) is used to perform the method (1) according to one of the Claims 1 until 11 is trained. A vehicle (2) comprising the vehicle control system (3) according to Claim 12 . Non-volatile, computer-readable storage medium containing a program stored therein, which, when placed on a computing unit (4) of a vehicle control system (3) according to Claim 12 is executed, instructs the vehicle control system (3) to execute the procedure (1) according to one of the Claims 1 until 11 to execute.