JP7185571B2 - Viewing direction estimation device, viewing direction estimation method, and program - Google Patents

Description

The present invention relates to technology for estimating a direction viewed by a driver.

2. Description of the Related Art Conventionally, there has been known a device for detecting an object visually recognized by a driver while driving an automobile. For example, Patent Literature 1 discloses an invention for estimating an area visually recognized by a driver based on the face direction and line of sight of the driver.

The invention described in Patent Document 1 has a problem that it becomes difficult to estimate the visual recognition area when the position of the driver's eyeball cannot be detected due to the influence of light sources such as sunlight and illumination or the influence of glasses. there is In Patent Document 1, in an invention for estimating a visual recognition area by correcting a provisional direction obtained from the face direction when a residence time in which the driver's face is looking in a predetermined direction exceeds a predetermined time, The correction value to be used is devised.

JP 2015-102498 A

When estimating the area visually recognized by the driver based on the face direction and line of sight, the detection range of the line of sight is narrower than that of the face direction due to restrictions on the installation position and number of devices that detect face direction and line of sight. . On the other hand, with respect to face orientation, which has a wide detection range, the driver should move his or her face very little when looking at the speedometer or rearview mirror, and only look at the eyes, in order to quickly respond to changes in the gaze target in order to avoid a rear-end collision. It is known from the present inventors' previous research that it moves.

In Patent Document 1 described above, the visual recognition area is estimated from the face orientation. In Patent Literature 1, basically, the visual recognition area is estimated from the face orientation based on the tendency of human motion. Although Patent Document 1 also describes that vehicle speed information is taken into consideration, the amount of face movement for visually recognizing an object does not necessarily have a linear relationship with the running speed of the vehicle. For example, if there is an object ahead that needs attention, it cannot be diverted from the front, regardless of the magnitude of the vehicle speed. Conversely, for example, when changing lanes, regardless of the running speed of the vehicle, it is necessary to reliably check the sides and rear.

SUMMARY OF THE INVENTION In view of the above background, it is an object of the present invention to provide a novel viewing direction estimating device capable of appropriately detecting an object visually recognized by a driver.

A viewing direction estimating apparatus of the present invention includes a surroundings monitoring sensor that detects moving objects around a vehicle and a communication unit that communicates with onboard equipment mounted on the vehicle, a driving scene identifying unit that identifies a driving scene based on a driving scene; a camera that captures the image of the driver; a face direction detection unit that detects the face direction of the driver from the image captured by the camera; and an image captured by the camera. A line-of-sight detection unit that detects the driver's line of sight from the and an output unit configured to output information indicative of the face orientation detected by the face orientation detection unit when the line of sight detected by the sight line detection unit is equal to or greater than a predetermined angle. and the driving scene specified by the driving scene specifying unit.

The inventors of the present invention have found that the extent to which the driver moves his/her face differs depending on the driving scene (that is, the situation of the vehicle or the surroundings of the vehicle) when the driver looks to the side. ADVANTAGE OF THE INVENTION According to this invention, when estimating a visual recognition direction from a face direction, the estimation precision of a visual recognition direction can be improved by being based on a driving scene in addition to a face direction. Also, when the line of sight is not at a predetermined angle or more, that is, when the line of sight faces in a direction close to the front, the viewing direction can be accurately estimated using the line of sight.

Here, the angle is defined by an angle obtained by swinging the line of sight in the horizontal direction (panning direction) with the front direction of the driver being 0°. For example, it may be defined as 00° right and 00° left from the front, or the right angle may be defined as a positive angle and the left angle as a negative angle. However, if the right side is positive and the left side is negative, the absolute value is used to determine whether or not the angle is "greater than or equal to a predetermined angle".

According to another aspect of the present invention, a viewing direction estimating apparatus includes: a surroundings monitoring sensor for detecting moving objects around a vehicle; A driving scene identification unit that identifies a driving scene based on device data, a camera that captures the image of the driver, a face direction detection unit that detects the face direction of the driver from the image captured by the camera, and the camera. A line-of-sight detection unit that detects the line-of-sight of the driver from the image captured by the driver, a viewing direction estimating unit that estimates the direction the driver is viewing from the driver's line of sight or face direction, and the viewing direction estimating unit. an output unit for outputting information indicating the viewing direction, wherein the viewing direction estimating unit outputs information indicating the viewing direction detected by the face orientation detecting unit when the accuracy of gaze detection by the gaze detecting unit is equal to or less than a predetermined threshold. A viewing direction is estimated based on the face orientation and the driving scene specified by the driving scene specifying unit.

As described above, the visual recognition direction is estimated from the visual line while the visual line detection is being performed, and the visual recognition direction is estimated from the face orientation when the accuracy of the visual line detection is equal to or less than a predetermined threshold value. At this time, it is possible to improve the accuracy of estimating the viewing direction based on the driving scene in addition to the face orientation.

In a driving scene in which the moving object is within the range of the predetermined angle, the viewing direction estimating unit sets the face orientation detected by the face orientation detecting unit as the viewing direction or sets the face orientation as the viewing direction. 1 is performed to determine the viewing direction, and in a driving scene in which there is no moving object within the range of the predetermined angle, a second correction is performed on the face orientation detected by the face orientation detection unit. The angle of the first corrected viewing direction may be smaller than the angle of the second corrected viewing direction.

When there is a moving object within the range of a predetermined angle, it is necessary to pay attention to that direction, so it is thought that there is a tendency not to move the face when visually recognizing the side. According to the present invention, it is possible to appropriately estimate the visual recognition direction from the face orientation based on whether or not there is a moving object within the range of a predetermined angle. It should be noted that whether or not there is a moving object can be determined by image processing the image around the vehicle captured by an external camera, which is an example of a surrounding monitoring sensor.

In a driving scene in which the vehicle is traveling straight ahead, the viewing direction estimating unit sets the face orientation detected by the face orientation detecting unit as the viewing direction, or performs a first correction on the face orientation and visually recognizes the face orientation. In a driving scene in which the vehicle is not traveling straight, a second correction is performed on the face orientation detected by the face orientation detection unit to determine the viewing direction, and the first corrected viewing direction is determined. The angle may be smaller than the second corrected viewing direction.

If the vehicle is going straight, it is believed that the driver tends to look ahead. ADVANTAGE OF THE INVENTION According to this invention, the visual recognition direction can be estimated appropriately from the face direction based on the situation whether the vehicle is going straight. Whether or not the vehicle is traveling straight can be detected based on data from a steering angle sensor, which is an example of vehicle equipment.

The viewing direction estimating unit determines whether the face direction detected by the face direction detecting unit is the viewing direction in a driving scene in which the line of sight to the road in a predetermined range in front of the vehicle is blocked and visibility is poor. On the other hand, the first correction is performed to set the viewing direction, and in a driving scene with good visibility, the face orientation detected by the face orientation detection unit is subjected to the second correction to set the viewing direction, and the first correction is performed. The angle of the corrected viewing direction may be smaller than the angle of the second corrected viewing direction.

In a driving scene with poor visibility, it is thought that drivers tend to look ahead to watch out for cyclists and pedestrians jumping out. Advantageous Effects of Invention According to the present invention, it is possible to appropriately estimate the viewing direction from the face orientation based on whether or not the visibility is good. A driving scene with poor visibility in which the line of sight to the road in a predetermined range ahead of the vehicle is blocked is, for example, a vehicle parked on the shoulder of the curving road and buildings and block walls on both sides of the road. or there is a building or block wall blocking the view to the left or right at an intersection. Visibility is also poor in foggy and rainy conditions, and in driving on unlit roads at night. Whether the visibility is good or not can be determined by processing images of the surroundings of the vehicle captured by an exterior camera, which is an example of a surroundings monitoring sensor, and determining whether or not the visibility meets criteria for good visibility.

In a driving scene in which the vehicle is moving, the viewing direction estimation unit sets the face direction detected by the face direction detection unit as the viewing direction, or performs a first correction on the face direction and visually recognizes the face direction. in a driving scene in which the vehicle is stopped, the face orientation detected by the face orientation detection unit is subjected to a second correction to obtain a viewing direction, and the viewing direction is obtained by performing the first correction. may be smaller than the second corrected viewing direction.

When the vehicle is stopped, there is no danger of the vehicle colliding with something, so there is a tendency to visually recognize by moving not only the eyes but also the face. According to the present invention, it is possible to appropriately estimate the visual recognition direction from the face orientation based on whether the vehicle is stopped or moving. Note that "stopped" includes not only the state in which the engine of the vehicle is turned off, but also the state in which the engine is turned on but the vehicle is temporarily stopped at a traffic light or a pedestrian crossing. Whether the vehicle is moving or stopped can be detected based on data from a speed sensor, which is an example of vehicle equipment.

The viewing direction estimation method of the present invention includes the steps of acquiring data from a surroundings monitoring sensor that detects moving objects around a vehicle, acquiring data from on-vehicle equipment mounted on the vehicle, and obtaining data from the surroundings monitoring sensor. a step of identifying a driving scene based on the data or the data of the in-vehicle device; a step of photographing the driver with a camera; a step of detecting the orientation of the driver from the image photographed with the camera; A step of detecting the line of sight of the driver from the image taken in , a step of estimating the viewing direction that the driver is visually recognizing from the driver's line of sight or face direction, and a step of outputting information indicating the estimated viewing direction In the step of estimating the viewing direction, when the detected line of sight is at a predetermined angle or more, the viewing direction is estimated based on the driver's face orientation and the driving scene when the face orientation is detected. to estimate

According to another aspect of the present invention, there is provided a viewing direction estimation method comprising the steps of: obtaining data from a perimeter monitoring sensor that detects moving objects in the vicinity of a vehicle; obtaining data from in-vehicle equipment mounted on the vehicle; A step of identifying a driving scene based on the data of the perimeter monitoring sensor or the data of the in-vehicle device, the step of capturing an image of the driver with a camera, and detecting the orientation of the driver from the image captured by the camera. a step of detecting the line of sight of the driver from the image captured by the camera; a step of estimating a viewing direction that the driver is viewing from the driver's line of sight or face orientation; and the step of estimating the viewing direction includes: when the accuracy of line-of-sight detection is equal to or less than a predetermined threshold, the driver's face orientation and the driving scene when the face orientation is detected; Estimate the viewing direction based on

A program according to the present invention is a program for estimating a driver's viewing direction, and comprises a step of acquiring data from a peripheral monitoring sensor that detects a moving object in the vicinity of a vehicle in a computer; a step of acquiring data from an in-vehicle device; a step of identifying a driving scene based on the data acquired from the peripheral monitoring sensor or the data acquired from the in-vehicle device; the step of acquiring an image captured by a camera; a step of detecting a driver's face direction from an image; a step of detecting a driver's line of sight from the image; and the step of estimating the viewing direction is performed when the driver's face orientation and the face orientation are detected when the detected line of sight is at a predetermined angle or more. The viewing direction is estimated based on the driving scene.

A program according to another aspect of the present invention is a program for estimating the visual recognition direction of a driver, comprising the step of: acquiring data from a peripheral monitoring sensor that detects moving objects in the vicinity of a vehicle; a step of acquiring data from an in-vehicle device installed in the vehicle; a step of identifying a driving scene based on the data of the surrounding monitoring sensor or the data of the in-vehicle device; a step of acquiring an image captured by a camera; a step of detecting a driver's face direction from an image; a step of detecting a driver's line of sight from the image; and the step of outputting information indicating the viewed viewing direction, and the step of estimating the viewing direction is executed when the driver's face orientation and the face orientation are detected when the accuracy of line-of-sight detection is equal to or less than a predetermined threshold. The viewing direction is estimated based on the driving scene.

ADVANTAGE OF THE INVENTION According to this invention, when estimating a visual recognition direction from a face direction, the estimation precision of a visual recognition direction can be improved by being based on a driving scene in addition to a face direction.

It is a figure which shows the structure of the visual recognition direction estimation apparatus of embodiment. (a) It is a figure which shows the relationship between a driving scene and correction|amendment. (b) A diagram showing an example of a driving scene correction table. FIG. 5 is a diagram showing results of measuring the relationship between the face direction and the line-of-sight direction for each driving scene. It is a figure which shows the operation|movement of the visual recognition direction estimation apparatus of 1st Embodiment. It is a figure which shows operation|movement of the visual recognition direction estimation apparatus of 2nd Embodiment.

A visual recognition direction estimation device according to an embodiment of the present invention will be described below with reference to the drawings.
(First embodiment)
FIG. 1 is a diagram showing the configuration of a viewing direction estimation device 1 according to the first embodiment. The viewing direction estimation device 1 is a device for estimating the direction viewed by the driver of the vehicle. By detecting the viewing direction with the viewing direction estimation device 1, it is possible to determine whether the driver is not looking aside, for example, whether the driver has taken appropriate safety actions when changing lanes or turning left or right. can judge.

The viewing direction estimating device 1 of the first embodiment is connected to a perimeter monitoring sensor 20 and an in-vehicle device 21 provided in the vehicle. The surroundings monitoring sensor 20 is a camera 10, a LIDAR, an ultrasonic sensor, or the like that captures the surroundings of the vehicle. The in-vehicle device 21 includes a sensor that detects running speed and acceleration, a sensor that detects the steering angle, a sensor that detects the amount of depression of the brake pedal, a GPS that detects the current position, and the like. Neither the perimeter monitoring sensor 20 nor the vehicle-mounted device 21 are limited to the devices listed here.

The viewing direction estimation device 1 of the first embodiment includes a camera 10, a face orientation detection unit 11, a line of sight detection unit 12, a communication unit 13, a driving scene identification unit 14, a viewing direction estimation unit 15, A driving scene correction table 16 and an output unit 17 are provided.

The camera 10 is a near-infrared camera, and can stably photograph the driver even if the vehicle interior becomes dark. The camera 10 is mounted at a position for photographing the driver from the front, for example, under the meter visor. A “Driver Status Monitor” camera developed by Denso Corporation may be used.

The face direction detection unit 11 has a function of detecting the face direction of the driver based on the face image of the driver captured by the camera 10 . The line-of-sight detection unit 12 has a function of detecting the driver's line of sight from the image of the driver's eyes captured by the camera 10 . The line-of-sight detection unit 12 detects the direction of the iris of the driver, adds the direction of the driver's face to the direction of the driver's line of sight, and detects the direction of the driver's line of sight. detect whether

The visual recognition direction estimation unit 15 has a function of estimating an object visually recognized by the driver based on the driver's face direction and line of sight. The viewing direction estimation unit 15 estimates the viewing target based on the driver's line of sight when the line of sight is smaller than a predetermined angle. The predetermined angle is a threshold for switching whether face direction data is used for estimating the viewing direction. Determined. In this embodiment, the predetermined angle is 40°.

The viewing direction estimator 15 estimates the viewing direction based on the face direction and the driving scene when the line of sight of the driver is at a predetermined angle or more. The viewing direction estimation device 1 estimates the viewing direction from the face orientation. Specifically, the viewing direction is estimated by estimating the angle of the face orientation as the viewing direction, or performing correction to increase the angle from the angle of the face orientation. The reason for increasing the angle from the face orientation angle is that the viewing direction tends to be larger than the face angle because people move their eyes in addition to the movement of the face when looking to the side. . Also, the viewing direction estimation unit 15 changes the estimation method depending on the driving scene.

FIG. 2(a) is a diagram showing the concept of estimating the viewing direction from the face orientation. FIG. 2A shows five examples of driving scenes. In the first example, the driving scene is determined from the viewpoints of (A) whether there is a moving object such as a vehicle ahead or a pedestrian crossing in front, and (B) whether there is a moving object. Make corrections. Specifically, when there is a moving object in front, the viewing direction is set as it is without correcting the angle of the face orientation, or correction is performed to increase the angle. Conversely, when there is no moving object in front, correction is performed to increase the angle of the face orientation compared to when there is a moving object in front.

In the second example, it is determined whether the driving scene is (A) the vehicle is traveling straight or (B) the vehicle is not traveling straight. also reduces the degree of correction. In the third example, it is determined whether the driving scene is (A) poor visibility or (B) good visibility, and the degree of correction is made smaller when the visibility is poor than when the visibility is good. Here, the visibility can be defined according to a predetermined standard. As an example, if there is an object such as a building or roadside tree that blocks the line of sight to the road in a predetermined range in front of the vehicle (for example, a range of 50 m ahead and 40 degrees left and right), it is determined that the visibility is poor. If not, it can be determined that the visibility is good. Obstruction of the line of sight to the road is not limited to objects, and visibility may be determined to be poor when visibility is obstructed by fog, rain, or the like, or when driving on a road without streetlights at night. In short, "poor visibility" means a situation in which it is difficult to notice the appearance of vehicles or people.

In the fourth example, it is determined whether the driving scene is (A) the vehicle is moving or (B) the vehicle is stopped, and the degree of correction is smaller when the vehicle is moving than when it is stopped. In the fifth example, it is determined whether the driving scene is (A) with guardrails on the road or (B) with no guardrails, and the degree of correction is smaller when there are no guardrails than when there are guardrails. do. Although five examples are shown in FIG. 2(a), driving scenes to which the viewing direction estimation device 1 can be applied are not limited to the examples shown in FIG. 2(a). For example, day and night may be used as driving scenes.

FIG. 2(a) shows some classification examples of driving scenes, but if we summarize these in a broader concept, (A) driving scenes in which the driver should pay close attention to the road ahead, and (B) relatively safe driving scenes. It can be broadly classified into driving scenes that can be thought of as driving scenes that require particular attention to the surroundings. (A) can be said to be a driving scene in which an accident is likely to occur if care is not taken. (A) In a driving scene where the driver should pay close attention to the front, the divergence between the face direction and the viewing direction is small, and the viewing direction can be estimated with a small correction. It can be said that the deviation between the face direction and the viewing direction is large, and the viewing direction is estimated with a large correction.

In each example shown in FIG. 2(a), the scene of (A) is described as "no correction or small correction", and the scene of (B) is described as "large correction". This does not mean that the same corrections are made in all five examples. FIG. 2(a) shows the concept of estimating the viewing direction from the face orientation, and explains that the scene (A) requires less correction than the scene (B). Needless to say, the specific amount of correction differs depending on each scene.

The viewing direction estimation unit 15 is connected to the driving scene correction table 16 . The driving scene correction table 16 stores an appropriate correction amount for each driving scene based on the concept shown in FIG. 2(a). This correction amount is generated based on the result of measuring the face direction angle and the line of sight angle for each driving scene.

FIG. 3 is a diagram showing the results of measuring the relationship between the face direction and the line-of-sight direction in driving scenes with and without a moving object. As shown in FIG. 3, when there is no moving object, the line-of-sight direction and face orientation are in a nearly proportional relationship. On the other hand, when there is a moving object in front of the vehicle, the driver does not move his/her face much even when looking to the side. This is because, as described above, when there is a moving object in front, the user must pay close attention to the front, so the face direction is fixed in the front.

In this embodiment, as indicated by the dotted line in FIG. Based on this, the viewing direction is estimated. Drivers tend to move their faces when looking at an angle of 40° or more to the left or right from the front, which tends to make it impossible to detect line-of-sight data. , the viewing direction is estimated from the face orientation.

Fig. 3 shows the measurement results of face orientation and line-of-sight direction with respect to the difference in driving scenes when there is a moving object and when there is no moving object. Difference in scene (example 2 in Fig. 2(a)), difference in driving scene such as good or bad visibility (example 3 in Fig. 2(a)), difference in driving scene such as whether the vehicle is moving or stopped (Fig. 2(a)) Regarding (a) example 4) and the difference in driving scenes such as the presence or absence of guardrails (example 5 in FIG. 2(a)), the relationship between the face direction and the line of sight differs depending on the driving scene. confirmed.

FIG. 2B is a diagram showing an example of the driving scene correction table 16. As shown in FIG. In this example, in a scene with a moving object, an angle obtained by adding 20° to the face orientation is estimated as the viewing direction. In a scene without moving objects, an angle obtained by adding 40° to the face orientation is estimated as the viewing direction. Here, an example is given in which correction is performed by adding a predetermined angle to the face orientation angle, but the method of correction is not limited to the method of adding a constant, and a variable corresponding to the face orientation angle is added. Alternatively, the face orientation angle may be multiplied by some variable. What kind of correction is specifically appropriate may be determined based on the relationship between the angle of the face orientation and the viewing direction based on the measurement results shown in FIG.

When the line-of-sight direction is 40° or more, the viewing direction estimating unit 15 reads correction information corresponding to the driving scene from the driving scene correction table, corrects the face direction angle according to the driving scene, and corrects the angle of the driver's face. Estimate viewing direction. The output unit 17 outputs data indicating the viewing direction estimated by the viewing direction estimation unit 15 to a driving support device such as an inattentiveness determination device.

The configuration of the viewing direction estimation device 1 according to the present embodiment has been described above. is a computer with A program having modules for realizing the respective functions of the face orientation detection unit 11, the line-of-sight detection unit 12, the driving scene identification unit 14, and the viewing direction estimation unit 15 is stored in a RAM or ROM, and the program is executed by the CPU. By executing it, the functions of the above-described visual recognition direction estimation device 1 are realized. Such programs are also included in the scope of the present invention. An example of the output unit 17 is a communication interface.

FIG. 4 is a diagram showing the operation of the viewing direction estimation device 1 according to the first embodiment. The viewing direction estimating device 1 of the first embodiment receives data from the surroundings monitoring sensor 20 and the in-vehicle device 21 (S10). In addition, the viewing direction estimation device 1 photographs the driver with the camera 10 (S12). Subsequently, based on the captured image, the viewing direction estimation device 1 detects the face direction of the driver with the face direction detection unit 11, and detects the line of sight of the driver with the line-of-sight detection unit 12. (S14).

The face direction detection unit 11 and the line of sight detection unit 12 pass the face direction data and the line of sight data to the viewing direction estimation unit 15 . Based on the line-of-sight data, the viewing direction estimation unit 15 determines whether or not the line-of-sight direction is equal to or greater than a predetermined angle (40° in this example) (S16). If it is determined that the line-of-sight direction is not greater than or equal to the predetermined angle (NO in S16), the viewing direction estimator 15 estimates the viewing direction based on the line-of-sight data.

If it is determined that the line-of-sight direction is greater than or equal to the predetermined angle (YES in S16), the driving scene estimation unit identifies the driving scene based on the data of the surroundings monitoring sensor 20 or the data of the in-vehicle device 21 (S20 ). As an example, based on the image of the surroundings outside the vehicle acquired from the camera 10 for photographing outside the vehicle, which is one of the surroundings monitoring sensors 20, it is possible to determine whether there is a moving object such as a preceding vehicle or a pedestrian in front of the vehicle. Identify driving scenes.

Next, the viewing direction estimation unit 15 reads correction information corresponding to the driving scene from the driving scene correction table 16, corrects the driver's face orientation data, and estimates the viewing direction of the driver ( S22). The visual recognition direction estimation device 1 outputs the data of the estimated visual recognition direction to other devices such as a warning device for inattentive driving (S24).

The configuration and operation of the viewing direction estimation device 1 according to the first embodiment have been described above. The viewing direction estimation apparatus 1 according to the first embodiment can accurately estimate the viewing direction using the line of sight when the viewing direction is less than a predetermined angle, that is, when the viewing angle is close to the front. When the line-of-sight direction is greater than or equal to a predetermined angle, the visual direction can be estimated appropriately based on the driving scene in addition to the face direction. .

In Prior Document 1, a method of estimating the viewing direction from information such as the angular velocity level of the face, the residence time, the face direction zone, etc. is adopted based on the tendency of human motion. The device 1 accurately estimates the viewing direction from the face orientation by a completely different approach of focusing on the driving scene.

(Second embodiment)
Next, a viewing direction estimation device according to a second embodiment will be described. The basic configuration of the viewing direction estimation apparatus of the second embodiment is the same as that of the first embodiment (see FIG. 1), but the viewing direction estimation unit 15 uses line-of-sight data to estimate the viewing direction. The processing for determining whether or not face direction data is used is different.

FIG. 5 is a diagram showing the operation of the viewing direction estimation device according to the second embodiment. The viewing direction estimating device of the second embodiment receives data from the surroundings monitoring sensor 20 and the vehicle-mounted device 21 (S10). In addition, the viewing direction estimation device photographs the driver with the camera 10 (S12). Subsequently, based on the captured image, the viewing direction estimating device detects the face direction of the driver with the face direction detection unit 11, and detects the line of sight of the driver with the line-of-sight detection unit 12 ( S14). At this time, the line-of-sight detection unit 12 also measures the line-of-sight detection accuracy. The data of the sight line detection accuracy may be a flag indicating whether or not the sight line of the driver has been detected.

The face direction detection unit 11 and the line-of-sight detection unit 12 pass the face direction data, the line-of-sight data, and the detection accuracy data to the viewing direction estimation unit 15 . The viewing direction estimating unit 15 determines whether or not the data of the sight line detection accuracy is equal to or greater than a predetermined threshold (S17). When it is determined that the line-of-sight detection accuracy is higher than the predetermined threshold (NO in S17), the viewing direction estimation unit 15 estimates the viewing direction based on the line-of-sight data (S18). If the line-of-sight detection accuracy is a flag indicating whether or not the line-of-sight has been detected, the viewing direction is estimated based on the line-of-sight data when the flag has a value indicating that the line of sight has been detected.

When it is determined that the line-of-sight detection accuracy is equal to or less than the predetermined threshold (YES in S17), the driving scene estimation device specifies the driving scene based on the data of the surroundings monitoring sensor 20 or the data of the in-vehicle device 21 (S20 ), and the viewing direction is estimated based on the face orientation data and the driving scene (S22). This processing is the same as that of the viewing direction estimation device 1 of the first embodiment. The visual recognition direction estimation device outputs the data of the estimated visual recognition direction to other devices such as a warning device for distracted driving (S24).

The configuration and operation of the viewing direction estimation device according to the second embodiment have been described above. The viewing direction estimation apparatus of the second embodiment can accurately estimate the viewing direction using the line of sight when the line of sight detection accuracy is higher than a predetermined threshold. By estimating the viewing direction based on the driving scene in addition to the face direction when the gaze detection accuracy is below a predetermined threshold, the viewing direction can be estimated appropriately regardless of the difference in facial movement due to the driving scene. .

Although the visual recognition direction estimation device of the present invention has been described in detail above with reference to the embodiments, the present invention is not limited to the above-described embodiments. In the embodiment described above, as shown in FIG. For example, the method of correction may be changed for each of three cases, such as when there is a preceding vehicle in front, when there is a pedestrian in front, and when there is no moving object in front. Of course, the method of correction may be changed by dividing into four or more cases.

Although five examples of driving scenes are introduced in FIG. 2(a), these driving scenes may overlap. For example, when there is a moving object ahead on a road with poor visibility. In such a case, it may be determined in advance based on which driving scene the viewing direction is estimated.

Claims (16)

a peripheral monitoring sensor that detects moving objects around the vehicle and a communication unit that communicates with onboard equipment mounted on the vehicle;
a driving scene identification unit that identifies a driving scene based on the data of the surroundings monitoring sensor or the data of the in-vehicle device;
a camera that captures the driver,
a face orientation detection unit that detects the face orientation of the driver from the image captured by the camera;
a line-of-sight detection unit that detects the line of sight of the driver from the image captured by the camera;
a viewing direction estimating unit that estimates the direction viewed by the driver from the driver's line of sight or face orientation;
an output unit that outputs information indicating the viewing direction estimated by the viewing direction estimation unit;
with
The viewing direction estimator,
based on the face direction detected by the face direction detection unit and the driving scene identified by the driving scene identification unit when the line of sight detected by the line of sight detection unit is at a predetermined angle or more; and, in a driving scene in which the moving object is present within a predetermined angle range, the face direction detected by the face direction detection unit is set as the viewing direction, or the face direction is set as the viewing direction. 1 is performed to determine the viewing direction, and in a driving scene in which there is no moving object within the range of the predetermined angle, a second correction is performed on the face orientation detected by the face orientation detection unit. A viewing direction estimating device in which an angle of the viewing direction subjected to the first correction is smaller than an angle of the viewing direction subjected to the second correction. a peripheral monitoring sensor that detects moving objects around the vehicle and a communication unit that communicates with onboard equipment mounted on the vehicle;
a driving scene identification unit that identifies a driving scene based on the data of the surroundings monitoring sensor or the data of the in-vehicle device;
a camera that captures the driver,
a face orientation detection unit that detects the face orientation of the driver from the image captured by the camera;
a line-of-sight detection unit that detects the line of sight of the driver from the image captured by the camera;
a viewing direction estimating unit that estimates the direction viewed by the driver from the driver's line of sight or face orientation;
an output unit that outputs information indicating the viewing direction estimated by the viewing direction estimation unit;
with
The viewing direction estimator,
When the accuracy of line-of-sight detection by the line-of-sight detection unit is equal to or less than a predetermined threshold, based on the face direction detected by the face direction detection unit and the driving scene identified by the driving scene identification unit, A viewing direction is estimated , and in a driving scene in which the moving object is within a predetermined angle range, the face orientation detected by the face orientation detection unit is set as the viewing direction, or the first is corrected to determine the viewing direction, and in a driving scene in which the moving object is not present within the range of the predetermined angle, the face orientation detected by the face orientation detection unit is subjected to a second correction for visual recognition. and the angle of the first corrected viewing direction is smaller than the angle of the second corrected viewing direction. a peripheral monitoring sensor that detects moving objects around the vehicle and a communication unit that communicates with onboard equipment mounted on the vehicle;
a driving scene identification unit that identifies a driving scene based on the data of the surroundings monitoring sensor or the data of the in-vehicle device;
a camera that captures the driver,
a face orientation detection unit that detects the face orientation of the driver from the image captured by the camera;
a line-of-sight detection unit that detects the line of sight of the driver from the image captured by the camera;
a viewing direction estimating unit that estimates the direction viewed by the driver from the driver's line of sight or face orientation;
an output unit that outputs information indicating the viewing direction estimated by the viewing direction estimation unit;
with
The viewing direction estimator,
based on the face direction detected by the face direction detection unit and the driving scene identified by the driving scene identification unit when the line of sight detected by the line of sight detection unit is at a predetermined angle or more; and, in a driving scene in which the vehicle is traveling straight, the face orientation detected by the face orientation detection unit is used as the viewing direction, or the face orientation is visually recognized after performing a first correction. In a driving scene in which the vehicle is not traveling straight, a second correction is performed on the face orientation detected by the face orientation detection unit to determine the viewing direction, and the first corrected viewing direction is determined. A viewing direction estimating device in which the angle is smaller than the second corrected viewing direction. a peripheral monitoring sensor that detects moving objects around the vehicle and a communication unit that communicates with onboard equipment mounted on the vehicle;
a driving scene identification unit that identifies a driving scene based on the data of the surroundings monitoring sensor or the data of the in-vehicle device;
a camera that captures the driver,
a face orientation detection unit that detects the face orientation of the driver from the image captured by the camera;
a line-of-sight detection unit that detects the line of sight of the driver from the image captured by the camera;
a viewing direction estimating unit that estimates the direction viewed by the driver from the driver's line of sight or face orientation;
an output unit that outputs information indicating the viewing direction estimated by the viewing direction estimation unit;
with
The viewing direction estimator,
When the accuracy of line-of-sight detection by the line-of-sight detection unit is equal to or less than a predetermined threshold, based on the face direction detected by the face direction detection unit and the driving scene identified by the driving scene identification unit, In a driving scene in which the vehicle is traveling straight, the viewing direction is estimated, and the face orientation detected by the face orientation detection unit is used as the viewing direction, or the viewing direction is determined by performing a first correction on the face orientation. In a driving scene in which the vehicle is not traveling straight, the face orientation detected by the face orientation detection unit is subjected to a second correction to determine the viewing direction, and the angle of the viewing direction subjected to the first correction is is a viewing direction estimating device whose angle is smaller than the second corrected viewing direction. a peripheral monitoring sensor that detects moving objects around the vehicle and a communication unit that communicates with onboard equipment mounted on the vehicle;
a driving scene identification unit that identifies a driving scene based on the data of the surroundings monitoring sensor or the data of the in-vehicle device;
a camera that captures the driver,
a face orientation detection unit that detects the face orientation of the driver from the image captured by the camera;
a line-of-sight detection unit that detects the line of sight of the driver from the image captured by the camera;
a viewing direction estimating unit that estimates the direction viewed by the driver from the driver's line of sight or face orientation;
an output unit that outputs information indicating the viewing direction estimated by the viewing direction estimation unit;
with
The viewing direction estimator,
based on the face direction detected by the face direction detection unit and the driving scene identified by the driving scene identification unit when the line of sight detected by the line of sight detection unit is at a predetermined angle or more; In a driving scene with poor visibility where the line of sight to the road in a predetermined range in front of the vehicle is blocked, the face direction detected by the face direction detection unit is used as the viewing direction, or On the other hand, the first correction is performed to set the viewing direction, and in a driving scene with good visibility, the face orientation detected by the face orientation detection unit is subjected to the second correction to set the viewing direction, and the first correction is performed. The viewing direction estimating device, wherein the angle of the corrected viewing direction is smaller than the angle of the second corrected viewing direction. a peripheral monitoring sensor that detects moving objects around the vehicle and a communication unit that communicates with onboard equipment mounted on the vehicle;
a driving scene identification unit that identifies a driving scene based on the data of the surroundings monitoring sensor or the data of the in-vehicle device;
a camera that captures the driver,
a face orientation detection unit that detects the face orientation of the driver from the image captured by the camera;
a line-of-sight detection unit that detects the line of sight of the driver from the image captured by the camera;
a viewing direction estimating unit that estimates the direction viewed by the driver from the driver's line of sight or face orientation;
an output unit that outputs information indicating the viewing direction estimated by the viewing direction estimation unit;
with
The viewing direction estimator,
When the accuracy of line-of-sight detection by the line-of-sight detection unit is equal to or less than a predetermined threshold, based on the face direction detected by the face direction detection unit and the driving scene identified by the driving scene identification unit, In a driving scene in which the line of sight to the road in a predetermined range in front of the vehicle is blocked, the visibility direction is estimated. in a driving scene with good visibility, the face direction detected by the face direction detection unit is subjected to a second correction to determine the viewing direction, and the first correction is performed. The viewing direction estimating device, wherein the angle of the corrected viewing direction is smaller than the angle of the viewing direction corrected by the second correction. a peripheral monitoring sensor that detects moving objects around the vehicle and a communication unit that communicates with onboard equipment mounted on the vehicle;
a driving scene identification unit that identifies a driving scene based on the data of the surroundings monitoring sensor or the data of the in-vehicle device;
a camera that captures the driver,
a face orientation detection unit that detects the face orientation of the driver from the image captured by the camera;
a line-of-sight detection unit that detects the line of sight of the driver from the image captured by the camera;
a viewing direction estimating unit that estimates the direction viewed by the driver from the driver's line of sight or face orientation;
an output unit that outputs information indicating the viewing direction estimated by the viewing direction estimation unit;
with
The viewing direction estimator,
based on the face direction detected by the face direction detection unit and the driving scene identified by the driving scene identification unit when the line of sight detected by the line of sight detection unit is at a predetermined angle or more; and, in a driving scene in which the vehicle is moving, the face direction detected by the face direction detection unit is set as the viewing direction, or the face direction is visually recognized after performing a first correction. in a driving scene in which the vehicle is stopped, the face orientation detected by the face orientation detection unit is subjected to a second correction to obtain a viewing direction, and the viewing direction is obtained by performing the first correction. is smaller than the second corrected viewing direction. a peripheral monitoring sensor that detects moving objects around the vehicle and a communication unit that communicates with onboard equipment mounted on the vehicle;
a driving scene identification unit that identifies a driving scene based on the data of the surroundings monitoring sensor or the data of the in-vehicle device;
a camera that captures the driver,
a face orientation detection unit that detects the face orientation of the driver from the image captured by the camera;
a line-of-sight detection unit that detects the line of sight of the driver from the image captured by the camera;
a viewing direction estimating unit that estimates the direction viewed by the driver from the driver's line of sight or face orientation;
an output unit that outputs information indicating the viewing direction estimated by the viewing direction estimation unit;
with
The viewing direction estimator,
When the accuracy of line-of-sight detection by the line-of-sight detection unit is equal to or less than a predetermined threshold, based on the face direction detected by the face direction detection unit and the driving scene identified by the driving scene identification unit, The viewing direction is estimated, and in a driving scene in which the vehicle is moving, the face orientation detected by the face orientation detection unit is used as the viewing direction, or the viewing direction is determined by performing a first correction on the face orientation. In a driving scene in which the vehicle is stopped, the face direction detected by the face direction detection unit is subjected to a second correction to determine the viewing direction, and the viewing direction after the first correction is obtained. A viewing direction estimating device in which the angle is smaller than the second corrected viewing direction. A program for estimating the viewing direction of a driver, comprising:
obtaining data from perimeter monitoring sensors that detect moving objects in the vicinity of the vehicle;
a step of acquiring data from an in-vehicle device mounted in a vehicle;
a step of identifying a driving scene based on the data of the surroundings monitoring sensor or the data of the in-vehicle device;
obtaining an image captured by a camera;
a step of detecting a driver's face orientation from the image;
detecting a line of sight of the driver from the image;
a step of estimating a viewing direction viewed by the driver from the line of sight or face direction of the driver;
a step of outputting information indicating the estimated viewing direction;
and
In the step of estimating the viewing direction, when the detected line of sight is at a predetermined angle or more, the viewing direction is estimated based on the driver's face orientation and the driving scene when the face orientation is detected , In a driving scene in which the moving object is within a predetermined angular range, the face orientation detected in the step of detecting the face orientation is used as the viewing direction, or the face orientation is subjected to a first correction. is the viewing direction, and in a driving scene in which the moving object is not within the range of the predetermined angle, the face orientation detected in the step of detecting the face orientation is subjected to a second correction to be the viewing direction. , the program wherein the angle of the first corrected viewing direction is smaller than the angle of the second corrected viewing direction. A program for estimating the viewing direction of a driver, comprising:
obtaining data from perimeter monitoring sensors that detect moving objects in the vicinity of the vehicle;
a step of acquiring data from an in-vehicle device mounted in a vehicle;
a step of identifying a driving scene based on the data of the surroundings monitoring sensor or the data of the in-vehicle device;
obtaining an image captured by a camera;
a step of detecting a driver's face orientation from the image;
detecting a line of sight of the driver from the image;
a step of estimating a viewing direction viewed by the driver from the line of sight or face orientation of the driver;
a step of outputting information indicating the estimated viewing direction;
and
The step of estimating the viewing direction includes estimating the viewing direction based on the driver's face orientation and the driving scene when the driver's face orientation is detected when the accuracy of line-of-sight detection is equal to or less than a predetermined threshold ; In a driving scene in which the moving object is within a predetermined angular range, the face orientation detected in the step of detecting the face orientation is used as the viewing direction, or the face orientation is subjected to a first correction. is the viewing direction, and in a driving scene in which the moving object is not within the range of the predetermined angle, the face orientation detected in the step of detecting the face orientation is subjected to a second correction to be the viewing direction. , the program wherein the angle of the first corrected viewing direction is smaller than the angle of the second corrected viewing direction. A program for estimating the viewing direction of a driver, comprising:
obtaining data from perimeter monitoring sensors that detect moving objects in the vicinity of the vehicle;
a step of acquiring data from an in-vehicle device mounted in a vehicle;
a step of identifying a driving scene based on the data of the surroundings monitoring sensor or the data of the in-vehicle device;
obtaining an image captured by a camera;
a step of detecting a driver's face orientation from the image;
detecting a line of sight of the driver from the image;
a step of estimating a viewing direction viewed by the driver from the line of sight or face direction of the driver;
a step of outputting information indicating the estimated viewing direction;
and
In the step of estimating the viewing direction, when the detected line of sight is at a predetermined angle or more, the viewing direction is estimated based on the driver's face orientation and the driving scene when the face orientation is detected, In a driving scene in which the vehicle is traveling straight, the face orientation detected in the step of detecting the face orientation is set as the viewing direction, or the face orientation is subjected to the first correction and set as the viewing direction. In a driving scene in which the vehicle is not traveling straight, the face direction detected in the step of detecting the face direction is subjected to a second correction to determine the viewing direction. A program that assumes that the angle is smaller than the second corrected viewing direction. A program for estimating the viewing direction of a driver, comprising:
obtaining data from perimeter monitoring sensors that detect moving objects in the vicinity of the vehicle;
a step of acquiring data from an in-vehicle device mounted in a vehicle;
a step of identifying a driving scene based on the data of the surroundings monitoring sensor or the data of the in-vehicle device;
obtaining an image captured by a camera;
a step of detecting a driver's face orientation from the image;
detecting a line of sight of the driver from the image;
a step of estimating a viewing direction viewed by the driver from the line of sight or face orientation of the driver;
a step of outputting information indicating the estimated viewing direction;
and
The step of estimating the viewing direction includes estimating the viewing direction based on the driver's face orientation and the driving scene when the driver's face orientation is detected when the accuracy of line-of-sight detection is equal to or less than a predetermined threshold; In a driving scene in which the vehicle is traveling straight, the face orientation detected in the step of detecting the face orientation is set as the viewing direction, or the face orientation is subjected to the first correction and set as the viewing direction. In a driving scene in which the vehicle is not traveling straight, the face direction detected in the step of detecting the face direction is subjected to a second correction to determine the viewing direction. A program that assumes that the angle is smaller than the second corrected viewing direction. A program for estimating the viewing direction of a driver, comprising:
obtaining data from perimeter monitoring sensors that detect moving objects in the vicinity of the vehicle;
a step of acquiring data from an in-vehicle device mounted in a vehicle;
a step of identifying a driving scene based on the data of the surroundings monitoring sensor or the data of the in-vehicle device;
obtaining an image captured by a camera;
a step of detecting a driver's face orientation from the image;
detecting a line of sight of the driver from the image;
a step of estimating a viewing direction viewed by the driver from the line of sight or face orientation of the driver;
a step of outputting information indicating the estimated viewing direction;
and
In the step of estimating the viewing direction, when the detected line of sight is at a predetermined angle or more, the viewing direction is estimated based on the driver's face orientation and the driving scene when the face orientation is detected, In a driving scene in which the line of sight to the road in a predetermined range in front of the vehicle is blocked and visibility is poor, the face orientation detected in the step of detecting the face orientation is set as the visual recognition direction, or the face orientation is set to the first direction. Correction is performed to determine the viewing direction, and in a driving scene with good visibility, the face direction detected in the step of detecting the face direction is subjected to a second correction to be the viewing direction, and the first correction is performed. A program for setting the angle of the viewing direction to be smaller than the viewing direction to which the second correction is performed. A program for estimating the viewing direction of a driver, comprising:
obtaining data from perimeter monitoring sensors that detect moving objects in the vicinity of the vehicle;
a step of acquiring data from an in-vehicle device mounted in a vehicle;
a step of identifying a driving scene based on the data of the surroundings monitoring sensor or the data of the in-vehicle device;
obtaining an image captured by a camera;
a step of detecting a driver's face orientation from the image;
detecting a line of sight of the driver from the image;
a step of estimating a viewing direction viewed by the driver from the line of sight or face orientation of the driver;
a step of outputting information indicating the estimated viewing direction;
and
The step of estimating the viewing direction includes estimating the viewing direction based on the driver's face orientation and the driving scene when the driver's face orientation is detected when the accuracy of line-of-sight detection is equal to or less than a predetermined threshold; In a driving scene in which the line of sight to the road in a predetermined range in front of the vehicle is blocked and visibility is poor, the face orientation detected in the step of detecting the face orientation is set as the visual recognition direction, or the face orientation is set to the first direction. Correction is performed to determine the viewing direction, and in a driving scene with good visibility, the face direction detected in the step of detecting the face direction is subjected to a second correction to be the viewing direction, and the first correction is performed. A program for setting the angle of the viewing direction to be smaller than the viewing direction to which the second correction is performed. A program for estimating the viewing direction of a driver, comprising:
obtaining data from perimeter monitoring sensors that detect moving objects in the vicinity of the vehicle;
a step of acquiring data from an in-vehicle device mounted in a vehicle;
a step of identifying a driving scene based on the data of the surroundings monitoring sensor or the data of the in-vehicle device;
obtaining an image captured by a camera;
a step of detecting a driver's face orientation from the image;
detecting a line of sight of the driver from the image;
a step of estimating a viewing direction viewed by the driver from the line of sight or face orientation of the driver;
a step of outputting information indicating the estimated viewing direction;
and
In the step of estimating the viewing direction, when the detected line of sight is at a predetermined angle or more, the viewing direction is estimated based on the driver's face orientation and the driving scene when the face orientation is detected, In a driving scene in which the vehicle is moving, the face orientation detected in the step of detecting the face orientation is set as the viewing direction, or the face orientation is subjected to the first correction to be set as the viewing direction, and the vehicle is moving. In a stopped driving scene, the face direction detected in the step of detecting the face direction is subjected to a second correction to determine the viewing direction, and the angle of the viewing direction after the first correction is A program for setting the angle to be smaller than the second corrected viewing direction. A program for estimating the viewing direction of a driver, comprising:
obtaining data from perimeter monitoring sensors that detect moving objects in the vicinity of the vehicle;
a step of acquiring data from an in-vehicle device mounted in a vehicle;
a step of identifying a driving scene based on the data of the surroundings monitoring sensor or the data of the in-vehicle device;
obtaining an image captured by a camera;
a step of detecting a driver's face orientation from the image;
detecting a line of sight of the driver from the image;
a step of estimating a viewing direction viewed by the driver from the line of sight or face orientation of the driver;
a step of outputting information indicating the estimated viewing direction;
and
The step of estimating the viewing direction includes estimating the viewing direction based on the driver's face orientation and the driving scene when the driver's face orientation is detected when the accuracy of line-of-sight detection is equal to or less than a predetermined threshold; In a driving scene in which the vehicle is moving, the face orientation detected in the step of detecting the face orientation is set as the viewing direction, or the face orientation is subjected to the first correction to be set as the viewing direction, and the vehicle is moving. In a stopped driving scene, the face direction detected in the step of detecting the face direction is subjected to a second correction to determine the viewing direction, and the angle of the viewing direction after the first correction is A program for setting the angle to be smaller than the second corrected viewing direction.

Images