CN101405177A - Interactive operating device and method for operating same - Google Patents

Abstract

The invention relates to an interactive operating device (100) having a display device (35; 43; 59; 72; 102; 150; 170; 180) and to a method for operating the interactive operating device (100). The method comprises the following steps: displaying graphical information on the display device (35; 43; 59; 72; 102; 150; 170; 180); receiving sensor information; activating a manipulation movement, which is spatially determined relative to a display area of a manipulation element (33; 61; 151; 161; 169; 181; 186) on the display device (35; 43; 59; 72; 102; 150; 170; 180) if it is detected by means of the sensor information that the body part (36) of the user is located in the activation area, wherein the manipulation movement corresponds to the display area; wherein the received sensor information comprises user information which is analyzed for the purpose of detecting a manipulation intention for the at least one manipulation element (33; 61; 151; 161-; and the information displayed on the display device (35; 43; 59; 72; 102; 150; 170; 180) is adjusted in dependence on the detected actuation intention in order to optimally display the at least one actuation element (33; 61; 151; 161-169; 181-186) for the activation of the actuation action assigned to the actuation element (33; 61; 151; 161-169; 181-186). The operating device (100) is preferably designed as a component of a motor vehicle console in order to be able to carry out the operating method.

Description

Interactive manipulator and method for operating the same

technical field

The invention relates to a method for operating an interactive operating device with a display device according to the preamble of claim 1 and an operating device with a display device according to the preamble of claim 16 .

Background technique

Vehicle functions in the cockpit of the vehicle are often operated via so-called interactive controls. The interactive operating device includes a display device on which one or more operating elements are displayed in graphical form. Individual vehicle functions are assigned to the operating elements. The user moves a body part, for example a finger of a hand, towards the graphic representation of the operating element, thereby activating or triggering an action, which is referred to below as an operating action. Does the body part have to touch a sensor unit (such as a touch-sensitive film) arranged in front of the display device or integrated in the display device or is it sufficient to activate the associated operating element only at a certain distance? The operating action for the operating element depends on the structure of the sensor unit of the operating device.

On some interactive manipulation devices it is necessary to touch to trigger the manipulation action, such interactive manipulation devices are referred to as touch screen-controllers. An interactive actuation device that can be activated without contact includes, for example, sensors that detect high-frequency signals transmitted through the human body. In this case it is necessary to arrange the high-frequency transmitter close to the body or in contact with the body. In automobiles, for example, such high-frequency transmitters can be integrated in vehicle seats. A method and a device for transmitting information by means of high-frequency signals transmitted through the human body are described, for example, in the publication WO 2004/078536.

The body part must be in a specific area to activate the maneuvering action, this area is called the activation area. On a touchscreen operator, the activation area is a two-dimensional area on the surface of the touchscreen operator. The activation surface usually corresponds to the display surface of the operating element to be actuated. On operating devices that can be operated without contact, the activation region is usually a three-dimensional region that lies immediately above or directly adjacent to the surface of the display device. The projection of the activation region onto the display device along a plane perpendicular to the surface of the display device usually coincides with the display area of the associated operating element.

The display of information on the display device of such an interactive manipulation device is not based on the resolution ability of the human eye in the prior art, but mainly depends on the tapping reliability and maneuvering accuracy of the user's fingers .

The operating elements displayed on the display device are also referred to as virtual operating elements, and these operating elements include menus, buttons, and the like. However, they can also have the shape of graphical objects, for example in navigation maps designed interactively. Usually such a virtual manipulation element is also called a window component (Widget). Said operating elements are now usually shown on interactive operating devices with a relatively large size, so that they can be activated comfortably and easily by the user with a finger. As a result, the display area of the operating elements usually covers a considerable part of the display area of the display device, which could otherwise be used for displaying additional information. The display area often obscures the "behind" graphic information. If the representation of the operating elements on the display device is reduced, the position for information display can be obtained, but the operating comfort is also reduced because of the difficulty of "tapping" the operating elements with a finger Significantly increased.

It is known from the prior art to zoom the scale of the operating element as a response to a first operating action. On such actuating devices, at least two actuating movements are required to activate and carry out the desired actuating movement.

Laid-open document US 5,579,037 discloses a display device with an electromagnetic digitizer tablet, with which Japanese characters displayed on the display device can be entered. The digitizing tray includes a recording pen connected to the digitizing tray through wires. If the stylus is placed at a first distance from the display device, the area displayed around the upper position of the stylus is shown enlarged. As long as the distance between the stylus and the display device is smaller than the first distance and greater than the second distance smaller than the first distance, the enlarged area is parallel to the distance between the stylus and the display device. Position changes match. If it is below the second distance, the magnified area is no longer dependent on the movement parallel to the display surface, so that multiple magnified displays can be entered by touching the display device at a suitable position One of the characters in the corresponding enlarged character is displayed at the appropriate position. Such an operating device, which can only be operated with a stylus, is not suitable for many fields of application, for example as an interactive operating device in the cockpit of a motor vehicle. Furthermore, it is very difficult to move the stylus at a predetermined distance in parallel in front of the display device in a moving vehicle to obtain the correct cutout shown in the enlarged view.

From DE 10 2004 045 885 A1, a control element preferably configured as a turn-press-pull button for a motor vehicle is disclosed, which is especially controlled by pressing on the control element, touching the control element, Rotation of the element and/or pulling out of the operating element to operate the function of the vehicle, wherein the operating element is equipped with a lighting device for illuminating the operating element, for controlling the access of the operator of the operating element An identified proximity sensor and a lighting control device for adjusting the lighting of the actuating element as a function of the output signal of the proximity sensor.

It is therefore desirable to seek an interactive operating device which, on the one hand, correspondingly enables an optimal display of information and, at the same time, offers a high degree of operating comfort due to the easy activation of the operating elements.

Contents of the invention

It is therefore the technical task of the present invention to provide an operating device and a method for operating an interactive operating device, with which not only an optimal information transmission adapted to the human eye can be achieved, but also At the same time, the operating elements can be activated comfortably.

According to the invention, this object is achieved by a method with the features of claim 1 and a device with the features of claim 16 . The basis of the invention is to adapt the information displayed on the display device in dependence on the specific situation. In this case, the display can be designed optimally for the visual perception of the displayed information. However, if an actuation intention by the user is detected, the displayed information is changed so that the actuation element can be optimally activated correspondingly depending on the detected actuation intention for the actuation element.

According to the invention, a method is proposed for operating an interactive operating device, in particular in a motor vehicle, wherein the operating device has a display device and information with operating elements that is displayed and/or can be displayed on the display device , the method includes the following steps:

- detecting a manipulation intention for at least one of the manipulation elements displayed on the display device,

Adapting the information displayed on the display device as a function of the detected manipulation intention, so that the at least one manipulation element intended to be manipulated is displayed optimally for activation of the manipulation element.

According to a preferred embodiment of the invention, in order to detect an actuation intention for at least one of the actuation elements displayed on the display device, it is detected whether a body part of the operator is in an activation region, which is spatially Determined relative to a display area of at least one of the operating elements on the display device.

According to a further embodiment of the method according to the invention, an actuation attempt for the at least one actuation element is detected temporally before the activation of the actuation action.

In a preferred embodiment, the method for operating an interactive operating device with a display device, in particular in a motor vehicle, comprises the following steps:

- displaying graphical information on display means;

- Receive sensor information;

- activation of a manipulation action, if by means of the sensor information it is detected that a body part of the user is in an activation area, which is spatially determined with respect to the display area of the manipulation element on the display device, to which the manipulation action is assigned the manipulation element;

which stipulates that,

- the received sensor information includes user information, which is analyzed in order to convey a manipulation intention for the at least one manipulation element temporally prior to activation of the manipulation action; and

- Adapting the information displayed on the display device as a function of the detected manipulation intention, so that the at least one manipulation element is displayed optimally for activating a manipulation action assigned to the manipulation element with a body part.

When replacing a perceptually optimized visual layout with a manipulation-optimized tactile layout, the operating elements are displayed, for example, gradually or enlarged. When an actuation intention is detected, for example, a smaller displayed actuation element is shown enlarged. This achieves the effect that although a high information density can be displayed on the display device, it can be comfortably operated at any time as long as it is required. User information refers to sensor information that describes the user's behavior or portrays the user himself. The user information is detected by means of a sensor unit and is a component of the sensor information, which is evaluated here to detect an actuation intention.

In order to be able to reliably detect manipulation intentions, in a preferred embodiment of the invention, the sensor information includes information about physical movements of the user. Physical actions are for example movements of the user's arms or hands or fingers. Another body movement is, for example, the orientation of the eyes for capturing information displayed on the display device.

Therefore, in a particularly preferred embodiment, the user information includes information about the viewing direction of the user. This means that the user's gaze towards the display device is often a prerequisite for actuating an operating element.

For example, the viewing direction of the user can be detected by means of a camera sensor. Therefore, according to a preferred embodiment of the invention, the user information is detected at least partially by means of a camera system. With such a camera system, movements of body parts can generally also be reliably detected.

According to a further preferred embodiment of the invention, the user information is detected at least partially by means of an ultrasonic sensor system. Ultrasonic sensor systems are suitable for particularly precise and reliable determination of the position of body parts. Movements towards the interactive operator can thus be reliably detected with the ultrasonic sensor system even at relatively large distances from the interactive operator. Precise position, extension and speed measurements can likewise be carried out in the vicinity of the actuating device. Ultrasonic sensor systems may be used as an alternative or in conjunction with other sensor systems.

In a particularly preferred embodiment of the invention, the direction of movement of the body part is detected with the aid of user information. This direction of movement can be used individually or e.g. collectively together with other information, in particular user information, preferred viewing directions, to find from a group of operating elements one or more operating elements that the user would like to operate as the next operating element manipulation elements. Therefore, in a refinement of the invention, during the detection of an actuation intention, an actuation possibility for this actuation element and possibly other actuation possibilities for other actuation elements are detected and the displayed information is adjusted so that The at least one operating element or the other operating elements are optimally displayed for activation of the respectively associated operating action or other operating actions depending on their operating possibilities. A kind of "spotlight manipulation" can thus be realized. Depending on the distance of the body parts, their direction and/or speed of movement and other user information, an area can be detected on the display device in which the operating element to be actuated is likely to be located. This detected area is called the manipulation focus area. The manipulating focus area is relatively unclear when the operator's hand is far away from the display device and the movement speed is small, and becomes clearer as the operator moves closer to the display device. Unclearness here means that there is uncertainty regarding the target position towards which the body part is moving on the display device. This target position serves as the center of the steering focus area. Unreliability manifests itself in the area of the manipulation focus area on the display device. The steering focus area is generally circular in shape, so that any ambiguity or uncertainty about the position of the target is represented on the radius of the steering focus area toward which the body part is positioned on the display device. positional movement. The greater the unreliability (the less sharp the manipulation focus area is), the larger the manipulation focus area is. The smaller the unreliability with respect to the target position (the sharper the steering focus area), the smaller the steering focus area. It is preferably provided here that the operating element is located on the display device according to how much of its surface is located in the region of the operating focus area at a predetermined zoom scale, how sharp the operating focus area is and/or the The operating elements are scaled according to how far they are from the position (center) of the operating focus area or otherwise optimized for simple activatability, for example by means of animation, expansion of a menu bar, or the like.

In a particularly preferred embodiment, the manipulation intention is detected repeatedly and/or continuously and the displayed information is adapted accordingly. It is thus possible, for example, to continuously detect the actuation focus area, and to zoom in and out on the display device the operating element located in the actuation focus area and to adapt it to the movement of the user.

In one embodiment of the invention, the transparency, the distance to adjacent other operating elements and/or the animation of one or more operating elements are changed during the adjustment. In this case, a list including a plurality of manipulation elements can be suitably animated and simulated to facilitate the user to select one or more list elements. In particular, long lists can be scrolled or browsed as a function of the detected manipulation intention.

It is also preferred to detect the distance between the body part and the display image of the operating element by means of the user information and to scale the operating element as a function of this distance. If, for example, a contactless sensor is used to activate the operating element, then the approach of the user's hand can be detected by the sensor, for example approximately 30 cm in front of the display device. The closer the hand or finger is to the display device, the better it may be to predict which operating element should be activated. A particularly optimized display of this operating element on the display device can then easily be achieved. Also, adjust the displayed information accordingly.

As a supplement to the information about the user, in a preferred embodiment the information about the driving state is analyzed in order to optimize the setting of the actuating elements. For example, driving on an uneven road surface can have an effect on the reliability of the user's actuation of the maneuvering of the actuation element. In one such case, the actuation elements are preferably displayed on a larger scale than in the driving situation on a smooth and smooth road. This ensures better maneuverability and increased comfort. At the same time, the largest possible display of information is ensured depending on the situation. The information is therefore preferably adapted to a display optimized for visual information transmission, for example all operating elements are faded out or "shown" with 100% transparency, as long as no manipulation intention is detected. This is advantageous in particular on interactive controls, on which only a very limited selection of information can be displayed because, for example, the display area is small.

In a preferred embodiment of the invention, when detecting the manipulation intention, gestures performed by the body parts are detected and evaluated, wherein the displayed information is adjusted according to a control function assigned to the respectively recognized gesture. Make adjustments. This makes it possible, for example, to concretize the manipulation intention when the number of potentially manipulable manipulation elements is particularly large. A static gesture can, for example, transmit a stop signal which prevents further adjustments of the operating element and enables a final selection without further scaling adjustments. It is likewise possible to facilitate viewing of a large number of operating elements shown in small graphics, one of which is assigned the focus and is shown enlarged for the purpose of the operating. The adjustment function can thus bring about any suitable changes in the displayed information.

The so-called static postures, dynamic postures, and combined postures proved to be suitable postures. According to one embodiment of the invention, the gesture therefore includes at least one static gesture which is detected by means of a predefined body part gesture. According to a further specific embodiment, the gesture includes at least one dynamic gesture which is detected by means of a predefined trajectory followed by the body part. For trajectories, on the one hand the shape can be analyzed in three-dimensional space. In addition, however, it is also possible to evaluate velocities at which the body part passes successively through the individual segments in three-dimensional space. Finally, in a further embodiment, the gesture comprises at least one composite gesture, which is detected by means of a predefined static gesture and/or a mutual transition of the static gesture via a predefined trajectory posture. Various embodiments of the invention can therefore consider only static gestures, only dynamic gestures or only composite gestures or any desired combination of these gestures. The gesture is detected by at least one sensor unit which can determine the position of the body part in space. The statement that the gesture is detected by at least one sensor unit should not mean that the gesture must be detected by a single sensor unit in the case of a plurality of sensor units. Rather, this expression also includes the recognition of the gesture by the interaction of a plurality of sensor units present. Whether a single sensor unit is sufficient for detecting a gesture depends on the particular design of the at least one sensor unit and on the gesture itself.

The information is preferably adjusted in such a way that the corresponding maximum amount of information is displayed for visual perception, and nevertheless optimal operability for the manipulation intention is displayed at least one manipulation element for which the manipulation intention was detected. The influence of very different circumstances can be considered in this case.

The features of the device according to the invention have the same advantages as the corresponding features of the method according to the invention.

Description of drawings

The invention is explained in detail below with the aid of preferred exemplary embodiments with reference to the drawings. in:

FIG. 1 is a schematic flow chart for explaining an embodiment of a method for operating an interactive manipulation device;

FIG. 2 is a schematic illustration by means of which the principle of the situation-dependent zooming of an operating element is shown;

3 is a schematic diagram of a display device on which information is optimally displayed for visual perception or partly for tactile operation;

4A-4D are schematic diagrams of information displayed on a display device, the information including a manipulation element shown partially enlarged according to a detected manipulation focus area;

5A-5D are schematic diagrams of a display device on which a manipulation element is scaled depending on the size of a detected manipulation focus area;

6A-6C are schematic diagrams of a display device on which manipulation elements are shown at different scales depending on the manipulation intent;

7A-7B are schematic diagrams of a display device on which manipulation elements of a list are browsed depending on manipulation intentions;

Fig. 8 is a schematic diagram of an interactive manipulation device in a car;

Figure 9-17 is a static gesture performed by the hand;

Figures 20-27 are dynamic gestures performed by hands; and

Figures 28-33 are composite gestures performed by hands.

Detailed ways

FIG. 1 shows a block diagram depicting a method 10 for operating an interactive manipulation device. As indicated by means of method block 12 , the method includes displaying information on a display device. First of all, the layout for displaying the information is preferably selected such that it is designed for optimal visual information transmission. One such layout is referred to below as a visual layout.

The actuating device detects sensor information 12 . On the one hand, the sensor information includes information about one or more users. This information is called user information. This information includes in particular information about the user's body movements, which are detected by means of the sensor unit. The body movement then includes, for example, an arm movement, which can be detected, for example, by means of a camera-detected sensor unit and/or an ultrasound-based sensor unit. Furthermore, the user's viewing direction can be detected by means of a sensor unit provided with a camera. Looking at the display device is a particularly important physical action, since in many cases the gaze at the display device of the interactive operating device precedes the manipulation process. In addition, the detected sensor information preferably includes information about the driving situation or the environment of the vehicle. It has been shown that specific driver events (Fahrereignisse) often precede specific maneuvers.

A user's actuation intention 14 is then detected with the aid of the detected sensor information. In order to detect the manipulation intention of the user, various pieces of information contained in the sensor information, in particular user information, are analyzed. In this case, in particular the behavior of the user, that is to say his body movements, is analyzed and evaluated. If, for example, a change in the viewing direction of the display device of the interactive manipulation device is detected and a temporally associated movement of the arm or hand that is spatially closest to the interactive manipulation element is detected, Manipulative intent can then be inferred from this. Many different combinations of individual pieces of information can be produced, where the individual piece of information can be analyzed and evaluated. For example, in one specific embodiment, a targeted movement of a body part in the direction of the actuation unit can be a prerequisite for the actuation intention to be recognized.

It is checked by means of questions whether there has been a change in the intention to manipulate 16 . If this is not the case, the display of the information on the display device continues unchanged. If a change in the manipulation intention occurs, that is to say a manipulation intention is detected or no longer exists, the display on the display device is adjusted according to the manipulation intention or the no (no longer) existing manipulation intention. The displayed information 18. There is also variation when manipulative intentions are (further) reified.

If the change in the manipulation intention consists in the recognition of a previously non-existing manipulation intention of the user, the information to be displayed is changed such that the layout on the display device is optimized for tactile manipulation. In one specific embodiment, it can be provided that no control elements are graphically represented in the visual layout. Manipulation elements are now faded in the haptic layout, ie their transparency is reduced to 100%. Likewise, provision can be made to switch from a graphic display to a text display. In another specific embodiment, a small operating element is represented graphically in the visual layout, in which case the control element can be shown enlarged in the tactile layout. Furthermore, the adjustment can include a change in the structure of the operating element, for example an animation of the operating element (widget). On a display device designed as a stereoscopic three-dimensional display device for a motor vehicle, a spatial emphasis of the operating elements or a spatially advanced overlapping of the information display can be provided.

The information shown is preferably detected and adjusted in phases that can be switched continuously into one another. Firstly, the manipulation intention is detected by means of sensor information from the first sensor unit. The first sensor unit preferably comprises a sensor system providing a picture, such as a camera system. If the usual manipulation intent is recognized, switch from a visual to a tactile layout. For example, previously invisible manipulation elements are now gradually revealed.

Furthermore, in a preferred embodiment, when detecting the manipulation intention, a position and an area surrounding the position are detected on the display device, the position being signaled for the manipulation action, ie the activation of the manipulation element. area. This process is referred to as determination 20 of the manipulation focal zone.

To detect the actuation focal zone, various sensor information can be evaluated. In particular, the direction of movement and the velocity of movement of the body parts of the user, the viewing direction towards the display device and information about the driving situation are evaluated individually or collectively. As a result of the manipulation of the focus area detection, a position indicating the notified target position and a radius for the detected notified target position are obtained on the display device. measure of certainty. In other embodiments it can be provided that the actuation focus area is not circular in shape. This is particularly advantageous if the direction of actuation deviates considerably from the direction perpendicular to the plane of the display surface of the display device.

The information characterizing the manipulation focus area (position/radius and/or other area specifications) is taken into account when adjusting the information to be displayed. Preferably, the operating element is displayed zoomed as a function of the position and the distance from the areal overlapping region of the operating focus region. This means that control elements located closer to the center of the control focus zone are larger than such control elements which are further away from this position of the control focus zone. Furthermore, those control elements which have a large area of overlap with the control focus zone when the control focus zone is projected onto the display device are enlarged. 4A to 4D show examples of this. Control elements 61' which are not in the control focus zone 62 but which are closer to the control focus zone 62 than other control elements 61" can preferably be displayed somewhat larger than these other control elements 61".

The described method is the determination 22 of the actuation possibility of the individual actuation elements. These manipulation possibilities can also be determined according to another method using user information, optionally taking into account other information on situational influences, such as the driving situation, past habits of the user, etc.

A continuous adjustment of the manipulation focus area now takes place. In a further stage, the movement of the body part and possibly other properties of the body part are detected more precisely by means of the second sensor unit.

In one embodiment, the signal transmitter generating the high-frequency signal is arranged near the user's body and the receiver is arranged on or around the display device for receiving the high-frequency signal propagating through the user's body and From this, the position of the user's hand in the vicinity of the display device is determined, and in this embodiment the setting of the displayed information can be fine-tuned particularly well. At a distance of approximately 30 cm from the display device, body parts of the user can be detected with such a sensor system. If several users are within range of the interactive operating device in the vehicle, for example the driver and the co-driver of the vehicle, these users can be distinguished by means of different high-frequency signals, which The transmission into the body of the user is via various signal transmitters, which are integrated, for example, in the driver's seat and the front passenger's seat. As a result, the displayed information can additionally be scaled or generally adjusted to suit the user's role (pilot/copilot). For example, it is expedient to display fewer and therefore larger operating elements on the display device if the driver is to activate the operating elements on the display device while the vehicle is driving at high speed. A co-driver who does not have to concentrate on driving the vehicle can, for example, operate smaller control elements, for which a greater number of control elements can be displayed on the display device. For the fine-tuning, the distance between the user's body part and the display device is preferably detected when the manipulation intention is detected. Furthermore, the stretching of the body part is additionally detected. This extension is also taken into account when scaling and/or designing the operating element. For a user with large, wide fingers, the operating elements have to be displayed on a somewhat larger area than for a user with small, narrow fingers. Thus, in another stage, finer adjustments can be made. More than two stages can be set. The second sensor unit may also be an ultrasound-based sensor unit.

Figure 2 schematically illustrates the transition from a visual layout to a tactile layout. In the upper region of FIG. 2 , two possible configurations of the information display diagrams 31 , 32 in a so-called visual layout are shown. The operating elements are not shown in the left-hand illustration 31 . In the illustration 32 on the right, a small operating element 33 is provided. The largest area 34 of the display device 35 is provided for displaying information. If a body part 36 , here a hand, approaches the display device 35 as shown in the center of FIG. 2 , the displayed information changes so that the operating element 33 ′ is shown enlarged, including additional text information. (A, B, C, D) and changes in its transparency at least with respect to the top left display 31 . The tactile layout is optimized such that the operating elements 33' can be optimally touched by a finger in order to be able to activate and trigger operating actions associated with the operating elements 33'.

FIG. 3 shows another example for switching from a visual layout 41 to a tactile layout 42 . Points of particular interest (points of interest—POIs) 44 - 47 of a map view 48 of the navigation system are schematically shown on the display device 43 . If the finger 49 approaches the display device 43 with a movement pointing downward towards the right edge 50 of the display device, then the point of interest is determined in terms of its size depending on the distance from the lower end of the right edge 50. zoom. For better maneuverability some of the points 46, 47 are slightly shifted relative to their original positions. In the left section 51 of the map view 48 the layout is also always adapted for visual information transfer.

4A to 4D show different display diagrams 55-58 of information on the display device 59 . Apart from the finger 60 , which is to correspondingly actuate one of the operating elements 61 , a so-called operating focus area 62 is drawn in a circle on the display device 59 . The operating element 61 is scaled as a function of its position relative to the position 63 (center) of the operating focus area 62 and its areal overlap with the operating focus area 62 . The overall size of the zoom may depend on driving conditions. If the vehicle is traveling at high speeds, for example, on uneven roads, the actuation elements have to be larger than when traveling at low speeds on a smooth road. The actuation focus field 62 is only present virtually and is generally not displayed on the display device 59 . The actuation focus area 62 is shown here for illustrative purposes only.

5A to 5D show how the manipulation focus area 70 changes when a finger 71 approaches the display device 72 and how this manipulation focus area 70 affects the scaling of the manipulation element in a preferred embodiment. Views 73 of display device 72 shown in FIGS. 5A to 5D each show a navigation map with points of interest 81 - 86 , which are operating elements. In FIG. 5A a tactile layout has been selected in which the points of interest 61 - 65 lying in the manipulation focus area 70 are displayed larger than the point of interest 86 lying outside the manipulation focus area 70 . View 73 according to FIG. 5A corresponds to the representation in which the manipulating hand is still further away from display device 72 . Therefore, the fingers are not yet shown in Fig. 5A. As the finger 71 approaches, the radius 74 of the actuation focus area 70 decreases, so that the actuation focus area 70 is shown smaller in FIG. 5B . The operating elements 82 , 83 located in the vicinity of the position 75 (center) of the operating focus area 70 are maximized. However, the point of interest 86 is enlarged somewhat relative to its appearance in FIG. 5A. As the finger 71 approaches further, the direction of movement of the finger 71 is evaluated, so that the position 75 of the actuation focus area 70 is shifted slightly. Thus, the point of interest 82 is enlarged in FIG. 5C than the point 83 which is now farther away from the center position 75 of the steering focus area 70 . FIG. 5D shows the situation in which finger 71 is located in an activation area that is determined relative to the display image of the operating element. In the case of an interactive operating device designed as a touch screen, the activation area is usually the area of the graphic display. On operating devices with sensors that operate in a non-contact manner, the activation area can be a spatial area that is preferably fixed at a small distance in front of or directly adjacent to the graphic representation of the operating element. Figure to confirm. If the body part, here the finger 71 , is in the activation region, the manipulation action assigned to the manipulation element is activated.

FIG. 6A shows a schematic illustration of a display device 150 on which operating elements 151 are arranged, which are divided into two groups 152 , 153 . The associated operating elements 151 are arranged as a list in a first group 152 . Arranged in the second group are operating elements 151 as number fields. In FIG. 6A , the operating elements 151 of the two groups 152 , 153 occupy only a small surface area on the display device 150 . Most of the surface can be used to display other information (not shown).

If finger 155 approaches display device 150 , operating element 151 shown in one of two groups 152 , 153 is enlarged. If an actuation attempt is detected for one of the actuation elements 151 of the first group 152 designed as a list, the list is shown enlarged, as shown in FIG. 6B . If the finger 155 comes closer to the second group 153 of operating elements 151 designed as a number field, these operating elements are shown enlarged, as shown in FIG. 6C .

In the embodiment described in connection with FIGS. 5A to 5D the center of the manipulation element is preferably not moved on the display device, whereas in the embodiment described in connection with FIGS. 6A-6C the position of the manipulation element is clearly Change.

It will be explained with the aid of FIGS. 7A and 7B how the detected manipulation intentions can advantageously be used to select manipulation elements from a list.

FIG. 7A shows a horizontal list 160 of operating elements 161 - 169 on a display device 170 . If a manipulation attempt is detected, a focus is assigned to a manipulation element 165 of the manipulation elements of list 160 . In this case, the focus is assigned to the operating element 165 , which is denoted by the number "5". This fact is made clear by the way the operating element 165 is shown enlarged. Typically, the focus is assigned to the control element of list 160 which is shown in the center of display device 170 . Alternatively, the focal point can also be assigned to an operating element shown on the edge.

The focal point can be "moved" by the user moving his finger 171 to the edge of the display device 170 . In FIG. 7A , the finger 171 is at the right edge 172 of the display device 170 . In one specific embodiment, the focal point can thus be shifted to the right.

Alternatively, the focal point preferably remains in the center of the display device 170 and the operating elements 161 - 169 are moved relative to the focal point. That is to say, the control panel is shifted to the left, as indicated by arrow 173 . This means that the focus is assigned as the next control panel to the control panel 166 with the numeral “6”, wherein the control panel 166 is shown enlarged here in the center of the display device 170 .

In both implementations, the focus scrolls or browses through the list of manipulation elements.

This process ends when the finger 171 is moved over the center of said display device 170 . The manipulation element to which the focus is assigned can then be manipulated in order to trigger the associated manipulation action.

Scrolling or browsing speed may vary depending on the position of the finger 171 . That is, the farther the finger is from the center, the faster the scrolling/browsing will be.

The browsing/scrolling direction can be the same as or opposite to the direction of finger movement. Furthermore, it can be specified in the list that the list is considered to be infinite. That means, the first list element is reconnected to the last list element.

FIG. 7B shows a display device 180 on which a list of operating elements 181 - 186 is shown. If finger 187 approaches display device 180 , a focus is assigned to one of control elements 181 - 186 . The manipulation element 183 to which the focus is assigned is shown enlarged for optimal manipulation. A finger 187 is located at the lower edge 188 of the display device 180 . As a result, operating elements 181 - 186 are moved “through focus” upwards, as indicated by arrow 189 . If the finger 187 moves to the center of the display device 180, the browsing process is stopped. The manipulation element that is in focus can be manipulated optimally. If finger 187 is moved into a position between the focal point and upper edge 190 of display device 180 , operating elements 181 - 186 are moved downwards. This allows browsing in both directions.

The focal point does not have to remain fixed at one position on the display device. Rather, the focus can be moved together with the operating element to which the focus is assigned. If this control element is further away from the predetermined target position of the focus than the next control element of the list, the focus jumps to this next control element.

Further embodiments can provide that a plurality of operating elements are shown enlarged. For example, you can set a main focus and two sub-focus. The operating element assigned to the secondary focus is enlarged, for example, but displayed slightly smaller than the operating element assigned to the main focus.

In one embodiment, it can be provided that an operating element is actuated if the user's body part, such as a finger, remains unchanged for a predetermined dwell time without changing the displayed information. The highest possibility of manipulation (for example the manipulation element in fixed focus or in the center of the manipulation focus area) is detected. This also applies in particular if the finger is not in the actual activation area of the operating element. As a result, a somewhat manipulation-free operating procedure of the manipulation element can be achieved.

FIG. 8 schematically shows an embodiment of an interactive operating device 100 in a vehicle 101 . Operating device 100 includes a display device 102 on which information can be displayed in a visual and tactile layout. A receiving sensor 104 is integrated in the operating device 100 , which can receive high-frequency signals transmitted through the user's body 106 , 108 without contact, the high-frequency signal being arranged on the body 106 , 108 nearby signal transmitters 110, 112 input into the body. The signal transmitters 110 , 112 are connected to a signal transmitter unit 113 , which in turn is connected to a vehicle bus system 126 . The signal transmitter unit 113 can also be regarded as a sensor system or sensor unit together with the receiving sensor 104 which is already a sensor unit in the narrow sense. In addition, the interactive manipulation device 100 includes a control unit 120 , and the control unit 120 includes a display module 122 . The display module 122 prepares information to be displayed on the display device 102 . In particular, display module 122 adapts information to a visual or tactile layout. In addition, interactive operating device 100 includes a receiving unit 124 , which receives information from a sensor unit, which can be designed, for example, as camera system 130 or as ultrasonic sensor system 132 , for example via vehicle bus system 126 . Furthermore, receiving unit 124 receives information about the driving state of motor vehicle 101 via vehicle bus system 126 . Using the received sensor information including information about the driving state, an actuation intention of user 106 or 108 is detected by evaluation module 128 . Depending on the detected manipulation intention, the display module 122 changes the information displayed on the display device 102 . Evaluation module 128 is preferably designed in such a way that an actuation focus area is detected and an actuation possibility is detected for the individual actuation elements.

Furthermore, the control unit includes an activation module 134 which triggers or activates an actuation action when a body part, such as a finger, is located in an activation region defined relative to the display of the actuation element on the display device 102 . In the process, signals can be transmitted via the vehicle bus, which signals influence other control units of vehicle 101 .

The control unit 120 and the respective included modules 122 , 128 , 134 can be embodied not only in hardware but also in software.

The display device can be designed as a projection display in which displayed information is projected on a surface. In one such case, the distance or the movement of the body part relative to the surface or the gaze on the surface is important.

In sensor units that transmit high-frequency signals through the human body, frequencies in the range of approximately 80 kHz to 150 kHz have proven particularly suitable. However, the sensor unit can also be operated at frequencies outside this stated frequency range.

In addition to or as an alternative to sensor units that detect body parts due to the transmission of high-frequency signals through the human body, other sensor units that function in a non-contact manner can be used, such as ultrasound-based sensor units or Use those sensor units that use optical methods. Such a sensor unit can be designed, for example, according to the following principles. Transmitting—LEDs (Light Emitting Diodes) emit a rectangular, amplitude-modulated light signal in the visible or infrared wavelength range. The light signal reflected on the target is detected by a photodiode. The same rectangular amplitude-modulated reference light signal with a phase shift of 180° is sent by the compensation LED to the photodiode via a constant optical path. The compensation LED is adjusted by means of a control signal via a control loop such that the received reflected light signal of the transmission LED and the received reference light signal of the compensation LED cancel out at the photodiode And an equal signal is detected. The change in the adjustment signal is a measure for the pitch of the target. A sensor unit designed according to this principle is largely independent of temperature fluctuations and brightness fluctuations.

Preferably, at least one sensor unit or a plurality of sensor units are designed in such a way that they can detect, if necessary by interacting with each other, the area extension of the body part. Gestures performed with body parts such as the hand can thus be detected and interpreted as manipulation intentions. Those poses that depend only on the poses of body parts, especially the hands, are called fixed or static poses. The hand 140 held flat in front of the display device as shown in FIG. 9 can be interpreted, for example, as a stop signal which stops the browsing process or prevents the display of the displayed information for a predetermined time interval. any adjustments. FIGS. 10 and 11 show by way of example further simple static gestures performed with the left hand 140 in each case. If a plurality of component parts of the hand can be disassembled separately, different fixed gestures can also be detected and used, as shown by way of example in FIGS. 12 to 17 .

Dynamic gestures are detected if the movement of the body part in a specific spatial region is compared with a predetermined trajectory. An example for a trajectory in a two-dimensional plane is exemplarily shown in FIG. 18 , and an example for a trajectory in a three-dimensional space is exemplarily shown in FIG. 19 . Examples of gestures for dynamics are shown in FIGS. 20 to 27 . The position according to FIG. 20 includes an upward movement of the hand 140 extended horizontally, and the position according to FIG. 21 correspondingly includes a downward movement of the hand 140 . In FIGS. 22 to 25, gestures are performed by movements of the open hand vertically and flatly away from the display ( FIG. 22 ) or by movements towards the display ( FIG. 23 ) and by movements to the left. Execute (FIG. 24) or by moving to the right (FIG. 25). The approach velocity and/or the approach velocity of the body part can likewise be interpreted and applied accordingly. 26 and 27 show gestures performed by rotating the hand 140 about its longitudinal axis in the counterclockwise or clockwise direction of the outstretched hand. The gesture can accordingly also include two opposite directions of movement. During the analysis of the trajectory, the velocity at which the movement is carried out can also be taken into account in addition to the passed spatial points of the defined movement.

If the various components of the hand can be decomposed separately and/or their relative velocities can be detected, it is also possible to analyze complex postures or gestures and movement processes such as the spreading and closing of the fingers of a flat hand or the fisting of the fingers (Fig. 28) and the opening of the fist (FIG. 29) are analyzed and accounted for accordingly. FIGS. 30 and 31 illustrate further integrated gestures in which the execution of the corresponding gesture includes a closing of the fingers 142 ( FIG. 30 ) and an extension of the fingers 142 ( FIG. 31 ). FIG. 32 shows a composite posture in which the center of gravity of hand 140 executes a movement to the right in addition to the closing of fingers 142 . In FIG. 33 , the combined gesture shown by way of example consists of a clenching of the originally vertically oriented flat outstretched hand 140 and a simultaneous rotation of the hand by approximately 90° to the right.

The poses shown are exemplary poses only. Different manipulation intentions can be assigned to the individual gestures, which lead to a corresponding adjustment of the displayed information.

Preferably, the interactive operating device can be operated in a learning mode and thus be able to learn specific gestures with individual characteristics performed by different persons. For example, the driver may be asked to perform a certain gesture. With the aid of measurement data detected during execution, individual features of the gesture can be learned. Learning is preferably done on a human basis. It is even possible to identify persons by means of gesture features or by means of a user or driver identification program coded, for example, in a vehicle key.

Some embodiments are designed in such a way that the driver's posture is distinguished from that of the copilot. If the interactive control device is arranged on the passenger panel of a car, the driver's gesture can be distinguished from that of the co-pilot according to whether the gesture is performed with the right hand or with the left hand, where it is assumed that the driver and The co-pilots respectively use the hand closest to the co-dashboard to operate. Some sensor units are able to differentiate between the driver and the passenger, for example by means of the transmission of different high-frequency signals through the bodies of the driver and passenger, which are used by the sensor unit to detect the position of body parts , on such a sensor unit it is possible to distinguish the posture of the copilot and the driver by means of information from the sensor unit. If a distinction can be made, different maneuvering intentions or different display or adjustment functions for the driver and co-pilot can be assigned to the same gesture.

Claims (33)

1. Method for operating an interactive operating device (100), especially in a motor vehicle (101), said interactive operating device (100) having a display device (35; 43; 59; 72; 102; 150; 170; 180 ) and displayed and/or displayable on the display device (35; 43; 59; 72; 102; 150; 170; 180), including manipulation elements (33; 61; 151; 161-169; 181-186 ), the method includes the following steps: a) detecting for at least one of the manipulation elements (33; 61; 151; 161-169; 181-186) displayed on the display device (35; 43; 59; 72; 102; 150; 170; 180) manipulative intent, b) adapting the information displayed on said display device (35; 43; 59; 72; 102; 150; 170; 180) in dependence on the detected manipulation intention so as to optimally provide said at least one manipulation element Activation of (33;61;151;161-169;181-186) reveals said at least one manipulation element (33;61;151;161-169;181-186). 2. The method according to claim 1, characterized in that for detecting the control element (33; 61; 151; 161-169; 181-186), the manipulation intention of at least one of the manipulation elements detects whether the user's body part (36) is within an activation area spatially relative to said manipulation element (33 ; 61; 151; 161-169; 181-186) at least one manipulation element is determined in a display area on the display device (35; 43; 59; 72; 102; 150; 170; 180). 3. The method according to claim 1 or 2, characterized in that the manipulation of the at least one manipulation element (33; 61; 151; 161-169; 181-186) is detected temporally before the manipulation action is activated intention. 4. The method as claimed in claim 1, characterized in that the manipulation intention is detected repeatedly or continuously and the displayed information is adapted accordingly. 5. The method according to any one of claims 2 to 4, characterized in that relevant information is used for detecting the manipulation intention, said relevant information comprising information about body movements, in particular about movements of body parts (36) information about the direction and/or about the viewing direction of the user. 6. The method according to claim 5, characterized in that the information is used to detect the relationship between the body part (36) and the operating element (33; 61; 151; 161-169; 181-186). The spacing of the graph is displayed and the manipulation elements are scaled (33; 61; 151; 161-169; 181-186) depending on this spacing. 7. The method as claimed in claim 1, characterized in that the user information is detected at least partially by means of a camera system (130) and/or an ultrasonic sensor system. 8. The method according to any one of the preceding claims, characterized in that relevant information is used for detecting the maneuvering intention, said relevant information including environmental information about the driving situation, which is used when adjusting the optimally displayed information. Environmental information is taken into consideration. 9. The method as claimed in any one of the preceding claims, characterized in that when detecting the manipulation intention, the manipulation possibility of the manipulation element (33; 61; 151; 161-169; 181-186) is detected, and Further actuation possibilities of other actuation elements (33; 61; 151; 161-169; 181-186) are possibly detected and the displayed information is adjusted so that the respective associated actuation actions are optimally prepared according to their actuation possibilities Activation or activation of other manipulation actions to display. 10. Method according to any one of the preceding claims, characterized in that when adjusting the displayed information on the display device, the operating elements (33; 61; 151; 161-169; 181 - 186) transparency, size, distance from adjacent other manipulation elements (33; 61; 151; 161-169; 181-186) and/or said manipulation elements (33; 61; 151; 161-169; 181-186) or multiple manipulation elements (33; 61; 151; 161-169; 181-186). 11. Method according to any one of the preceding claims, characterized in that when detecting the manipulation intention, gestures performed by body parts are recognized and evaluated, wherein according to the corresponding recognized gesture Adjust function to adjust the information displayed. 12. The method as claimed in claim 1, characterized in that the gesture comprises at least one static gesture, which is detected by means of a predefined body gesture. 13. The method as claimed in claim 1, characterized in that the gesture comprises at least one dynamic gesture which is detected by means of a predefined trajectory followed by the body part. 14. The method as claimed in claim 1, characterized in that the gestures comprise at least one integrated gesture by means of a predefined static gesture and/or a predefined The inter-transitions of static poses of trajectories are recognized. 15. The method as claimed in claim 1, characterized in that, as long as no manipulation intention is detected, the information displayed on the display device is adjusted for a display optimized for visual information transmission. . 16. An interactive manipulation device (100), especially in a car (101), comprising a display device (35; 43; 59; 72; 102; 150; 170; 180) and a control device (120), in said display device ( 35; 43; 59; 72; 102; 150; 170; 180) can display and/or display information comprising manipulation elements (33; 61; 151; 161-169; 181-186), constructing said control device ( 120), thereby relying on the detected manipulation intention when detecting a manipulation intention for at least one of the manipulation elements displayed on the display device (35; 43; 59; 72; 102; 150; 170; 180) adapting the information displayed on said display means (35; 43; 59; 72; 102; 150; 170; 180) to optimize - 186) is activated to display at least one of said manipulation elements (33; 61; 151; 161-169; 181-186). 17. Device according to claim 16, characterized in that at least one sensor device is provided for detecting a body part (36) of the user in the activation area for detecting ; 43; 59; 72; 102; 150; 170; 180), wherein the activation area is spatially relative to said manipulation element (33; 61; 151; 161-169; 181-186) at least one operating element is defined in the display area on the display device (35; 43; 59; 72; 102; 150; 170; 180). 18. The device as claimed in claim 16 or 17, characterized in that the control device (120) is designed such that manipulation intentions can be detected repeatedly or continuously and the displayed information can be adjusted accordingly. 19. The device as claimed in claim 17 or 18, characterized in that the sensor information includes information about physical movements of the user. 20. The device as claimed in one of claims 16 to 19, characterized in that the user information is detectable and/or detectable at least partially by means of a camera system and/or an ultrasonic sensor system. 21. The device according to any one of claims 17 to 20, characterized in that the sensor information includes information about the user's viewing direction and/or about the direction of motion of the user's body part (36). information. 22. Device according to any one of claims 17 to 21, characterized in that the control device (120) is configured such that a user's body part (36) can be detected and/or detected by means of the sensor information ) in the direction of movement and/or the user's viewing direction. 23. The device according to any one of claims 16 to 22, characterized in that the control device (120) is configured such that when adjusting the optimally displayed information, information about the driving situation can be taken into account and/or taken into account. environmental information. 24. The device according to any one of claims 16 to 23, characterized in that when a manipulation intention is detected, the manipulation element (33; 61; 151; 161-169; 181 -186) and, if necessary, other manipulation possibilities of other manipulation elements (33; 61; 151; 161-169; 181-186) and adjusts the displayed information so that it is optimized according to its manipulation possibilities are displayed for the activation of the corresponding associated operating action or for the activation of other operating actions. 25. The device according to any one of claims 17 to 24, characterized in that the control device (120) is configured such that the relationship between the body part (36) and the body part (36) can be detected and/or detected by means of the sensor information. The display spacing of said manipulation elements (33; 61; 151; 161-169; 181-186), and said manipulation elements (33; 61; 151; 161-169; 181-186) can be scaled and / or to zoom. 26. Device according to any one of claims 16 to 25, characterized in that the operating elements (33; 61; 151; 161-169; 181-186) can be changed and/or changed during adjustment or Transparency, spacing and/or animation of a plurality of manipulation elements (33; 61; 151; 161-169; 181-186) to adjacent other manipulation elements (33; 61; 151; 161-169; 181-186) . 27. The device according to any one of claims 16 to 26, characterized in that as long as a manipulation intention cannot and/or is not detected, a display optimized for visual information transfer is displayed on the display device ( 35; 43; 59; 72; 102; 150; 170; 180) to adjust the information displayed on. 28. Device according to any one of claims 17 to 27, characterized in that the at least one sensor device is configured such that sensor information is detected by means of high-frequency signals transmitted through the user's body, the sensor The information may include, inter alia, the position of the body part (36). 29. The device as claimed in claim 17, characterized in that the at least one sensor device is a position-resolving, touch-sensitive device. 30. The device according to any one of claims 17 to 29, characterized in that at least one sensor device is designed in such a way that gestures performed by body parts are detected and evaluated, wherein the correspondingly recognized gestures can be assigned to One of the adjustment functions to adjust the displayed information. 31. The device as claimed in claim 30, characterized in that the gesture comprises at least one static gesture which can be detected by the at least one sensor device as a predefined body gesture. 32. The device according to claim 30 or 31, characterized in that the gesture comprises at least one dynamic gesture which can be performed by the at least one sensor device by means of a predefined trajectory traversed by the body part to identify. 33. The device according to any one of claims 30 to 32, characterized in that the gesture comprises at least one integrated gesture which can be detected by the at least one sensor device by means of a predefined static gestures and/or mutual transitions of static gestures via predefined trajectories.

Images