JP6432601B2 - Signal detection device and signal detection method - Google Patents

Description

  The present invention relates to a traffic signal detection apparatus and a traffic signal detection method.

  Conventionally, a technique has been proposed for accurately detecting a traffic light even when the contrast is reduced by sunshine (see Patent Document 1). In Patent Literature 1, a traffic signal is determined from the camera image using the luminance value of the lighting area of the traffic light and the degree of separation between the extinguishing area and the casing area existing around the traffic light.

JP2013-242686A

  At night, signal lights that are lit are brighter than their surroundings. For this reason, when the exposure of the camera is adjusted so that the luminance of the signal lamp that is lit is not saturated, there is a case where it is impossible to image the signal lamp that is turned off and the casing of the traffic signal that are present around the camera.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide a traffic signal detection device and a traffic signal that can accurately detect a traffic signal by imaging the entire traffic signal including signal lights that are turned on and off. It is to provide a detection method.

  A traffic light detection apparatus according to an aspect of the present invention uses an imaging unit mounted on a vehicle to capture an image of the surroundings of the vehicle to obtain an image, and the luminance is synchronized with the AC power cycle from the image. Detect changing signal lights. The superimposed images are generated by superimposing the images at different times so that the positions between the images of the signal lights detected from the images at different times acquired by the imaging unit are maintained. Recognize including traffic lights.

FIG. 1 is a block diagram illustrating an overall configuration of a traffic light detection apparatus according to an embodiment. FIG. 2 shows the difference in the correlation value depending on whether the phase of the reference signal is synchronized or asynchronous, FIG. 2A shows the state where the phase of the reference signal is synchronized with the phase of power, and FIG. This shows a state where the phase of the signal is inverted from the phase of the power. FIG. 3 shows signal lights (DA1 to DA3) detected by the signal light detector 22, FIG. 3 (a) shows the signal lamp DA1 detected when the vehicle is stopped, and FIG. The signal lamp DA2 detected when the vehicle is moving and the vehicle body is vibrating is shown, and FIG. 3C shows the signal lamp DA3 detected after adjusting the position of the image. FIG. 4 shows the signal lights (DA1 to DA3) detected by the signal light detection unit 22, FIG. 4 (a) shows the signal lamp DA1 detected when the vehicle is stopped, and FIG. The signal lamp DA2 detected when the vehicle moves and the vehicle body vibrates is shown. FIG. 4C shows the signal lamp DA3 detected after adjusting the position of the image. FIG. 5 is a graph for explaining details of the image selection unit 23 that selects an image in which the luminance of the signal lamp is equal to or less than the threshold value. FIG. 6 is a flowchart showing an example of a traffic light detection method using the traffic light detection apparatus shown in FIG. FIG. 7 is a flowchart showing an example of signal recognition processing by the signal recognition unit 26. FIG. 8 is a diagram for explaining the scale change of the camera image, and FIGS. 8A and 8B show two signal lights SG1 and SG2 detected in two successive information processing cycles. FIG. 8C shows a camera image after the scale of FIG. 8B has been changed. FIG. 9 is a diagram for explaining the scale change of the camera image and the rotation of the camera image. FIGS. 9A and 9B show two signal lights detected in two consecutive information processing cycles. It is a camera image which shows SG1 and SG2, and FIG.9 (c) shows the camera image after changing the scale of FIG.9 (b), and rotating it. FIG. 10 is a flowchart illustrating an example of a traffic light detection method according to the second embodiment. FIG. 11 shows the traffic signal areas (CT1 to CT3) cut out from the camera image, and FIG. 10 (a) shows the size in which five signal lights can be arranged in the vertical direction (y direction) and the horizontal direction (x direction). FIG. 11 (b) shows a traffic light area CT2 having a size capable of arranging five signal lights in the vertical direction (y direction), and FIG. 11 (c) shows a horizontal direction (x direction). Shows a traffic signal area CT3 having a size capable of arranging five signal lights.

(First embodiment)
Embodiments will be described with reference to the drawings. In the description of the drawings, the same portions are denoted by the same reference numerals, and description thereof is omitted.

  With reference to FIG. 1, the whole structure of the traffic signal detection apparatus concerning 1st Embodiment is demonstrated. The traffic signal detection device is mounted on a vehicle, and includes an imaging unit 11 that captures an image of the surroundings of the vehicle and acquires an image, and a traffic signal detection unit 12 that detects a traffic signal from the images acquired by the imaging unit 11.

  The imaging unit 11 is a camera including a solid-state imaging device, for example, a CCD or a CMOS, and acquires a color image that can be image-processed. The imaging unit 11 repeatedly captures the front of the vehicle at predetermined time intervals and acquires a plurality of continuous images (frames). Imaging is performed a plurality of times during one AC cycle of power supplied to the traffic light.

  The traffic light detection unit 12 receives an image (hereinafter referred to as “camera image”) acquired by the imaging unit 11 and detects a traffic signal from the camera image. Information of the detected traffic signal is transferred to another processing arithmetic device (vehicle CPU 13) mounted on the vehicle including a controller for realizing automatic driving of the vehicle, for example. The traffic signal detection unit 12 includes, for example, a microcontroller including a CPU, a memory 25, and an input / output unit, and configures a plurality of information processing units included in the traffic signal detection device by executing a computer program installed in advance. The traffic signal detection unit 12 repeatedly executes a series of information processing cycles (including synchronous detection processing) for detecting a traffic signal from a camera image for each of a plurality of continuous camera images (frames). The traffic light detection unit 12 may also be used as an ECU used for other control related to the vehicle.

  The memory 25 stores a plurality of camera images (frames) 28 imaged by the imaging unit 11 at the same time. For example, a plurality of camera images 28 captured during 1 to 3 AC cycles of power supplied to the traffic light are stored simultaneously. The synchronous detection process is performed for each of the plurality of camera images 28.

  The plurality of information processing units configured by the traffic signal detection unit 12 include a phase detection unit 29, a signal lamp detection unit 22, an image superimposition unit 21, a traffic signal recognition unit 26, and a position change calculation unit 16.

  The phase detection unit 29 extracts pixels whose luminance changes in synchronization with the AC cycle of the power supplied to the traffic light from a preset space region in the camera image 28, and the vehicle including the traffic light from the temporal change in luminance. Detect phase information of the surrounding power system.

  The signal lamp detection unit 22 detects a signal lamp whose luminance changes in synchronization with the AC power cycle from the camera image using the phase information of the power system. That is, the signal lamp detection unit 22 detects a signal lamp that is lit among a plurality of signal lamps included in the traffic light.

  The image superimposing unit 21 generates a superimposed image by superimposing the camera images at different times so that the positions of the signal lights detected from the images at different times by the signal lamp detecting unit 22 are maintained.

  The traffic signal recognition unit 26 recognizes the entire traffic signal including the signal lamp detected by the signal lamp detection unit 22 using the superimposed image. That is, the traffic light recognition unit 26 recognizes not only a signal light that is being lit but also an adjacent signal light that is being extinguished and a frame of the traffic signal from the superimposed image.

  The position change calculation unit 16 calculates the amount of change in position between images at different times of the signal lamp detected by the signal lamp detection unit 22 at different times. Specifically, the position change calculation unit 16 calculates the amount of change in the position of the signal lamp between images at different times from the change in the center of gravity position of the signal lamp or the increase or decrease in the area of the signal lamp. The image superimposing unit 21 performs alignment between the superimposed camera images at different times based on the amount of change in the position between the images of the signal lights at different times. An example using a change in the barycentric position of a signal lamp will be described later with reference to FIG. 3, and an example using an increase / decrease in the area of the signal lamp will be described later with reference to FIG.

  The signal lamp detection unit 22 includes a synchronized image generation unit 15 (synchronized pixel extraction unit) and a signal lamp determination unit 18. The synchronized image generating unit 15 uses the phase information of the power system detected by the phase detecting unit 29, and from the camera image 28, the synchronized pixel whose luminance changes in synchronization with the AC cycle of the power supplied to the traffic light. To extract. And the synchronous image which consists of the extracted synchronous pixel is produced | generated. Specifically, the synchronized image generation unit 15 includes a reference signal generation unit 17, a multiplication unit 30, and a low-pass filter (LPF) 20.

  The reference signal generation unit 17 uses the phase information of the power system (commercial power supply) to generate a reference signal synchronized with the phase of power supplied to the traffic light. The multiplier 30 multiplies the luminance signal of each pixel of the camera image (frame) 28 read from the memory 25 and the reference signal. The multiplication unit 30 performs the multiplication process described above for each of the plurality of camera images stored simultaneously in the memory 25. The LPF 20 reduces a frequency component higher than a predetermined cut-off frequency from the multiplication result obtained by the multiplication unit 30, extracts only a low-frequency component, and outputs a synchronized image composed of synchronized pixels. The phase matching of the reference signal will be described later with reference to FIG.

  The power supplied to the traffic light is AC power obtained by full-wave rectifying the power of the commercial power supply. The luminance of a signal lamp that is turned on when supplied with power from a commercial power supply changes in the same cycle as the cycle of full-wave rectified AC power (for example, 100 Hz). Therefore, it is possible to detect a signal lamp that is turned on when power is supplied from a commercial power source by extracting from the camera image 28 synchronous pixels whose luminance changes in synchronization with the AC cycle of the power supplied to the traffic light. it can.

  The signal lamp determination unit 18 detects a signal lamp that is lit from the synchronized image. Specifically, it is determined whether or not a signal lamp exists at the position of the synchronization pixel based on the detection stability and hue of the synchronization pixel. When a remote traffic signal is imaged, the position between images of the measurement target (traffic signal) varies due to vehicle vibration during traveling. Therefore, the signal lamp determination unit 18 determines whether or not synchronized pixels are extracted at the same position continuously for a predetermined determination period. Further, the signal light determination unit 18 determines whether or not the hue of the synchronized pixel is similar to the hue of the signal color. The signal light determination unit 18 detects the presence of a signal light at the position of the synchronization pixel when the synchronization pixel is extracted continuously at the same image position for a predetermined determination period and the hue of the synchronization pixel is similar to the hue of the signal color. Judge that.

  The image superimposing unit 21 includes an image selecting unit 23, an image integrating unit 24, and an image adjusting unit 19. The image selection unit 23 selects camera images at two or more different times at which the luminance of the signal lamp is equal to or less than a threshold value from the camera images at different times. The image selection unit 23 can select camera images at two or more different times when the luminance of the signal lamp is equal to or lower than the threshold value by using the phase of the reference signal synchronized with the AC power cycle. The image integration unit 24 generates a superimposed image by superimposing two or more selected camera images at different times. Details of the image selection unit 23 will be described later with reference to FIG.

  The image adjustment unit 19 adjusts the positional relationship between the camera images at different times acquired by the imaging unit 11 so that the position between the images of the signal lamps detected by the signal lamp determination unit 18 at different times is maintained. The image adjustment unit 19 uses the amount of change in the position of the signal lamp between the images at different times calculated by the position change calculation unit 16, thereby maintaining the position between the images of the signal lamp detected respectively from the images at different times. As described above, the positional relationship between camera images at different times can be adjusted. Therefore, the image accumulating unit 24 superimposes the camera image after adjusting the positional relationship, so that the alignment accuracy of the traffic light in the superimposed image is improved.

  With reference to FIG. 2A and FIG. 2B, the phase matching of the reference signal will be described. FIG. 2A shows a state in which the phase of the reference signal matches the phase of the power supplied to the traffic light. In this state, by multiplying 1) the luminance signal and 2) the reference signal of each pixel, 3) the signal after multiplication, that is, the luminance of the synchronous pixel and the average value of the luminance of the synchronous pixel (correlation value G1) , The largest value.

  On the other hand, FIG. 2B shows a state in which the phase of the reference signal is inverted from the phase of the power supplied to the traffic light. In this state, by multiplying each pixel by 1) the luminance signal and 2) the reference signal, 3) the signal after multiplication, that is, the luminance of the synchronous pixel and the average value of the luminance of the synchronous pixel (correlation value G2) , The smallest value.

  As the distance from the vehicle to the traffic signal increases, the luminance of the signal lamp detected by the imaging unit 11 decreases and the change width of the luminance also decreases. Therefore, a high correlation value (G1) can be obtained by bringing the phase of the reference signal closer to the phase of the luminance change of the signal lamp, that is, the phase of the electric power supplied to the traffic light, and thus far away traffic lights can be detected with high accuracy. be able to.

  3A to 3C, an example of the image adjustment unit 19 that adjusts the positional relationship between the camera images at different times based on the amount of change in the center of gravity between the images at different times of the signal lamp. Will be explained.

  The position change calculation unit 16 detects the position of the center of gravity of the signal lamp detected by the signal lamp determination unit 18 on the camera image. The position change calculation unit 16 detects the position of the center of gravity of each signal lamp detected from the images at different times, and calculates the amount of change in the position of the center of the signal lamp between the images at different times.

  The image adjustment unit 19 determines that the position of the center of gravity has changed when the amount of change in the position of the center of gravity of the signal lamp between images at different times exceeds a predetermined reference value. Then, the image adjusting unit 19 shifts the position of the camera image at a later time in the direction opposite to the moving direction of the barycentric position of the signal lamp by twice the moving distance of the barycentric position. The camera image after the position shift is stored in the memory 25 again.

  FIG. 3 shows the signal lights (DA1 to DA3) detected by the signal lamp determination unit 18, and (a) shows the signal lamp DA1 detected when the vehicle is stopped. (B) shows the signal lamp DA2 detected when the vehicle is moving and the vehicle body is vibrating. (C) shows the signal lamp DA3 detected after the image adjustment unit 19 has adjusted the position of the image. The lattice frames (FL, FLm3) in FIG. 3A to FIG. 3C indicate frames composed of a plurality of pixels arranged in a matrix, and each of the lattice frames indicates a pixel of the image sensor.

  When the vehicle is stopped, the camera image does not fluctuate because there is no vibration of the vehicle body. For this reason, as shown to Fig.3 (a), the position between the images of a different time of a signal lamp does not change between the same information processing cycles. Therefore, the signal lamp determination unit 18 can detect the signal lamp DA1 with its actual size and shape. On the other hand, when the vehicle moves and the vehicle body vibrates, the camera image also changes. For example, as shown in FIG. 3B, the position of the signal lamp moves from the region ST2 to the region EN2 during the same information processing cycle. In this case, the signal lamp determination unit 18 detects an area DA2 where the area ST2 and the area EN2 overlap as a signal lamp. If there is no vibration even when the vehicle is moving, a detection result similar to that shown in FIG. Further, the signal lamp determination unit 18 detects the signal lamp DA1 in FIG. 3A and the signal lamp DA2 in FIG. 3B at different times, specifically, in different information processing cycles. More specifically, the signal lamp DA1 and the signal lamp DA2 are detected in two consecutive information processing cycles.

  In the example of FIGS. 3A and 3B, the barycentric position of the signal lamp DA2 is moved by −0.5 pixels in the x-axis direction from the barycentric position of the signal lamp DA1. Therefore, as shown in FIG. 3C, the image adjustment unit 19 shifts the position of the frame (camera image) FL at a later time by +1 pixel in the x-axis direction. As a result, the position of the camera image at a later time can be shifted by twice the movement distance of the center of gravity (= one pixel) in the direction opposite to the movement direction of the center of gravity of the signal lamp (x-axis direction). The signal lamp DA3 in the frame FLm3 after the shift is in the same position as the signal lamp DA1 in FIG. Therefore, even if there is vibration of the vehicle body, the continuity of the luminance fluctuation of the traffic light can be maintained at the pixel level, so that stable signal light detection can be performed.

  With reference to FIGS. 4A to 4C, an example of the image adjusting unit 19 that adjusts the position between camera images at different times based on the increase or decrease of the area of the signal lamp will be described.

  The position change calculation unit 16 detects the area of the signal lamp detected by the signal lamp determination unit 18. The position change calculation unit 16 detects the area of each signal lamp detected from the images at different times, and calculates the increase / decrease amount of the area of the signal lamp.

  When the increase / decrease amount of the area of the signal lamp exceeds a predetermined reference value, the image adjusting unit 19 determines that the position of the signal lamp between images at different times has changed. When the area of the signal lamp decreases, the image adjustment unit 19 shifts the position of the camera image at the later time by the width of the decreasing area in the direction in which the area of the signal lamp decreases. On the other hand, when the area of the signal lamp increases, the image adjustment unit 19 shifts the position of the camera image at a later time by the width of the increasing region in the direction opposite to the increasing direction. Thereby, the position between the camera images of the signal lamp detected by the signal lamp determination unit 18 at different times is maintained. The camera image after the position shift is stored in the memory 25 again.

  FIG. 4 shows the signal lights (DA1 to DA3) detected by the signal lamp determination unit 18 as in FIG. However, FIG.4 (c) shows the signal lamp DA3 detected after the image adjustment part 19 adjusts the positional relationship between camera images.

  When the vehicle is stopped, as shown in FIG. 4A, the position between the images of the signal lamp does not change during the same information processing cycle. Therefore, the signal lamp determination unit 18 can detect the signal lamp DA1 with its actual size and shape. On the other hand, when the vehicle moves and the vehicle body vibrates, as shown in FIG. 4B, the position of the signal lamp moves from the region ST2 to the region EN2 during the same information processing cycle. In this case, the signal lamp determination unit 18 detects an area DA2 where the area ST2 and the area EN2 overlap as a signal lamp.

  In the example of FIG. 4, the area of the signal lamp DA2 is reduced from the area of the signal lamp DA1. Specifically, the area of the signal lamp DA2 decreases by −1 pixel in the x-axis direction and by 2 pixels in the y-axis direction. Therefore, as shown in FIG. 4C, the image adjusting unit 19 shifts the position of the frame (camera image) FL at a later time by −1 pixel in the x-axis direction and by 2 pixels in the y-axis direction. . Thereby, the image adjustment part 19 can shift the position of a camera image by the width | variety of the area | region which decreases in the direction which the area of a signal lamp decreases. The signal lamp DA3 in the frame FLm3 after the shift is in the same position as the signal lamp DA1 in FIG. Therefore, even if the vehicle body vibrates, the continuity of the luminance variation of the signal lamp can be maintained at the pixel level, so that stable signal lamp detection is possible.

  Although the camera images at the later time are adjusted here, it is only necessary to be able to adjust the position between the camera images at different times, and the camera images at the previous time may be adjusted. The image may be adjusted, or a part of the camera image may be adjusted.

  With reference to FIG. 5, the detail of the image selection part 23 which selects the camera image from which the brightness | luminance of a signal lamp becomes below a threshold value is demonstrated. “BR” in FIG. 5 indicates the time change of the luminance of the signal lamp, and “LO” indicates the time change of the reference signal generated by the reference signal generation unit 17. As described above, the reference signal LO is synchronized with the phase of power supplied to the traffic light, that is, the luminance change of the signal lamp. Therefore, when the image selection unit 23 selects two or more camera images in which the luminance of the signal lamp is equal to or lower than the threshold value, the threshold value (Th ′) of the reference signal corresponding to the luminance threshold value (Th) is set. Can be sought. The image selection unit 23 selects two or more camera images captured when the reference signal LO is smaller than a threshold value (Th ′). For example, a camera image picked up when the phase of the reference signal LO is in the third quadrant or the fourth quadrant is selected. Thereby, two or more camera images in which the luminance of the signal lamp is equal to or less than the threshold value (Th) can be selected. In this way, the image selection unit 23 can select two or more camera images in which the luminance of the signal lamp is equal to or less than the threshold value (Th) using the phase of the reference signal LO synchronized with the AC power cycle. it can.

  The details of the image selection unit 23 described with reference to FIG. 5 are an example. The image selection unit 23 may select the camera image by comparing the luminance of the signal lamp with the threshold value (Th) without using the reference signal LO generated by the reference signal generation unit 17.

  Thereby, the image superimposing unit 21 can superimpose two or more camera images captured when the luminance of the signal lamp is low. Therefore, it is possible to take an image of the lighted signal light or the frame of the traffic light in a state where the luminance of the lighted signal light is not saturated. Further, by using the phase of the reference signal LO, two or more camera images in which the luminance of the signal lamp is equal to or lower than the threshold can be selected with high accuracy without determining the luminance of the signal lamp.

  In addition, as for the image superimposition part 21, it is desirable to set the number of the camera images to select and superimpose according to the brightness | luminance of the signal lamp in the light extinction in the superimposition image, and the frame of a traffic light. Specifically, the number of camera images to be superimposed may be set so that the luminance of at least one of the signal lamp being extinguished in the superimposed image and the frame of the traffic signal is equal to or higher than the lower limit value necessary for detection of the traffic signal.

  Further, instead of or in addition to the number of camera images to be superimposed, the exposure condition of the imaging unit 11 may be feedback controlled. Specifically, the image selection unit 23 sets the exposure condition of the imaging unit 11 so that the luminance of at least one of the signal lamp being turned off and the frame of the traffic light in the superimposed image is equal to or higher than a lower limit value necessary for detection of the traffic light. do it. Here, the exposure condition of the imaging unit 11 includes at least one of the exposure time (shutter speed) of the camera, the aperture value (F value) of the camera lens, and the light receiving sensitivity (ISO sensitivity) of the imaging device.

  An example of a series of information processing cycles for detecting a traffic light from a camera image, which is executed by the traffic light detection unit 12 of FIG. 1, will be described with reference to FIGS. 6 and 7. The information processing cycle shown in the flowcharts of FIGS. 6 and 7 is started at the same time as the ignition switch of the vehicle is turned on and the traffic light detection device is started, and is repeatedly executed at a predetermined cycle until the traffic light detection device is stopped.

  First, in step S01 of FIG. 6, the image adjusting unit 19 determines the positional relationship between the camera images based on the offset amount set in step S13 in the previous information processing cycle, that is, the direction and amount of shifting of the camera image. adjust.

  In step S03, the image adjustment unit 19 stores the camera image after the position adjustment in the memory 25. Proceeding to step S05, the synchronized image generating unit 15 uses the phase information of the power system, and the synchronized pixels whose luminance changes in synchronization with the AC cycle of the power supplied to the traffic light from the position-adjusted camera image. And a synchronized image composed of the extracted synchronized pixels is generated.

  In step S07, the signal lamp determination unit 18 detects a signal lamp from the synchronized image. In step S09, the position change calculation unit 16 calculates the amount of change in the position of the signal lamp between the camera images from the change in the center of gravity position of the signal lamp or the increase or decrease in the area. The position change calculator 16 calculates the amount of change between the position of the signal lamp detected from the camera image after position adjustment and the position of the signal lamp detected in another information processing cycle.

  Proceeding to step S11, the image adjusting unit 19 determines whether or not the position of the signal lamp between images at different times has changed from a predetermined value. If the position has changed from the predetermined value (YES in S11), the process proceeds to step S13 in order to set the offset amount of the camera image. If the position has not changed from the predetermined value (NO in S11), it can be determined that the position between the camera images at different times of the signal lamp has not changed, so the offset amount of the camera image is not set.

  In step S13, as shown in FIG. 3, the offset amount of the camera image is calculated and set based on the moving direction and moving amount of the center of gravity position. Based on the set offset amount, the position between the camera images is adjusted in step S01 in the next processing cycle. Of course, as shown in FIG. 4, the offset amount may be calculated based on the area increasing / decreasing direction and the width of the increasing / decreasing area.

  After step S01, steps S15 to S21 are performed in parallel. In step S15, the image selection unit 23 extracts two or more camera images captured when the reference signal LO is smaller than the threshold value (Th ′), as shown in FIG. Thereby, it is possible to extract two or more camera images in which the luminance of the signal lamp is not more than the threshold value (Th).

  In step S17, the image selection unit 23 determines whether the number of extracted camera images is equal to or greater than the number of camera images to be superimposed set by the image superimposing unit 21. If the number is equal to or greater than the number of camera images to be superimposed (YES in S17), further camera image selection is unnecessary, and the process proceeds to step S19. If it is less than the number of camera images to be superimposed (NO in S17), it is necessary to select a further camera image, and therefore the process proceeds to the next information processing cycle without starting the superimposition work.

  In step S19, the image integrating unit 24 generates a superimposed image by superimposing the extracted camera images at two or more different times. In step S 21, the generated superimposed image is output to the traffic signal recognition unit 26.

  With reference to FIG. 7, an example of signal recognition processing by the signal recognition unit 26 will be described. First, data of a superimposed image is read (S51), and a traffic signal candidate is detected from the superimposed image using local features of a luminance pattern such as a Haar-like feature (S52). Then, the circularity of the lighting area included in the candidate traffic lights is determined (S53). Specifically, the ratio of the vertical and horizontal lengths of the lighting area is obtained, and only the lighting areas whose ratio is, for example, greater than 0.7 and less than 1.3 are extracted. Thereby, the candidate of the traffic light whose lighting area is substantially circular can be extracted.

  Next, the hue of the lighting area in the HSV color space is determined (S54). Specifically, the color space of the superimposed image is converted from RGB to HSV (Hue (hue), Saturation (saturation), Value (lightness)). And the candidate of a traffic light whose hue of a lighting area is red, blue, or yellow is extracted. Finally, the extracted candidate color of the traffic light (any one of red, blue, and yellow) is output to the vehicle CPU 13 in FIG.

  As described above, according to the first embodiment, the following operational effects can be obtained.

  At night, signal lights that are lit have higher brightness than the surroundings. Therefore, by adjusting the exposure of the image pickup unit 11, an image of the entire traffic light including signal lights that are lit and extinguished can be acquired at one time. Is difficult. Specifically, when the exposure is adjusted to the signal lamp that is lit, the signal lamp and the frame of the traffic light that are not lit are insufficiently exposed, and the signal light and the frame of the traffic light that are not lit cannot be imaged. On the other hand, when the exposure is set to the frame of the signal lamp being turned off and the traffic light, the exposure to the signal lamp being turned on is exceeded, so that it is difficult to image the frame of the signal lamp being turned off and the traffic light.

  Therefore, the superimposed images are generated by superimposing the camera images at different times so that the position of the signal lamp whose luminance changes with the alternating current cycle of power is maintained. Thereby, it is possible to capture the signal light or the frame of the traffic light with high sensitivity while suppressing overexposure to the signal light that is lit. Therefore, even if the surroundings of the traffic light are dark, it is possible to detect the traffic signal with high accuracy by imaging the entire traffic light including the signal lights that are turned on and off. Note that the superimposed images are generated by superimposing the camera images at different times so that the position of the signal lamp is maintained. However, the position of the signal lamp does not have to be completely matched, for example, the circular shape of the signal lamp or the traffic light It is only necessary to maintain the position of the signal lamp within a range of positional deviation in which the frame shape can be discriminated.

  The image superimposing unit 21 generates a superimposed image by superimposing camera images at two or more different times at which the luminance of the signal lamp is equal to or less than the threshold Th among the plurality of camera images in which the signal lamp is detected. Thereby, two or more images captured when the luminance of the signal lamp is low can be superimposed. Therefore, it is possible to image the signal light or the frame of the signal light that is turned off with high sensitivity in a state where the luminance of the signal light that is turned on is not saturated.

  The image selection unit 23 selects two or more camera images in which the luminance of the signal lamp is equal to or lower than the threshold value (Th) using the phase of the reference signal (LO) synchronized with the AC power cycle. The luminance of the signal lamp changes in synchronization with a reference signal (LO) that is synchronized with the AC cycle of power. Therefore, by using the phase of the reference signal (LO), two or more camera images in which the luminance of the signal lamp is equal to or lower than the threshold value (Th) can be selected with high accuracy without determining the luminance of the signal lamp.

  The image adjustment unit 19 adjusts the positional relationship between the superimposed camera images at different times so that the position between the images of the signal lights at different times is maintained. By superimposing the image whose position has been adjusted, the positioning accuracy of the traffic light in the superimposed image is improved.

  The image superimposing unit 21 sets the exposure condition of the imaging unit 11 so that the luminance of at least one of the signal lamp being extinguished and the frame of the traffic light in the superimposed image is equal to or higher than the lower limit value necessary for detection of the traffic light. Thereby, it is possible to detect a signal lamp or a frame of a traffic light that is extinguished from the superimposed image.

  The image superimposing unit 21 sets the number of camera images to be superimposed so that the luminance of at least one of the signal lamp being turned off and the frame of the traffic light in the superimposed image is equal to or higher than the lower limit value necessary for detection of the traffic light. Thereby, it is possible to detect a signal lamp or a frame of a traffic light that is extinguished from the superimposed image.

(Second Embodiment)
In the second embodiment, an image adjusting unit 19 that corrects not only the positional deviation in the image plane (xy plane) but also the positional deviation in the normal direction (z direction) of the image plane will be described.

  The overall configuration of the traffic light detection apparatus according to the second embodiment is the same as that shown in FIG. When the signal lamp determination unit 18 detects two or more signal lamps at the same time, the position change calculation unit detects a relative position between the camera images at different times of the two or more signal lamps. Specifically, when two or more signal lights are detected in the same information processing cycle, the relative positions of the two or more signal lights are obtained.

  The image adjustment unit 19 determines whether or not the relative positions of the two or more signal lamps detected from the images at different times have changed between the camera images at different times. When the relative position between the camera images at different times of the two or more signal lights is changing, the image adjusting unit 19 is configured to perform different times so that the relative positions between the camera images at different times of the two or more signal lights are maintained. At least one of the scale and rotation between the camera images is adjusted. The camera image after adjustment of at least one of scale and rotation is stored in the memory 25.

  The scale change of the camera image and the rotation of the camera image will be described with reference to FIGS. FIGS. 8A and 8B are camera images showing two signal lamps (SG1, SG2) detected in two successive information processing cycles. FIG. ) Shows the camera image after changing the scale. FIG. 8 (a) is a camera image captured at a stable time without being affected by the vibration of the vehicle body, and FIG. 8 (b) is a diagram showing the relative position of the two traffic lights changing as the distance to the traffic lights is reduced thereafter. It is a camera image. Due to the movement of the vehicle between two successive information processing cycles, the relative positions of the two signal lights (SG1, SG2) are changed in FIGS. 8 (a) and 8 (b).

  The image adjusting unit 19 calculates and sets the scale amount so that the relative positions between the camera images at different times of the two or more signal lamps (SG1, SG2) are maintained. FIG. 8 shows a case where rotation of the camera image is unnecessary. The image adjustment unit 19 changes the scale of the camera image in FIG. 8B as shown in FIG. For example, the camera image in FIG. Of course, you may enlarge the camera image of Fig.8 (a). The relative positions of the two signal lamps (SG1, SG2) in FIG. 8 (c) are maintained in FIG. 11 (a).

  On the other hand, FIG. 9 shows an example in which the image adjustment unit 19 changes the scale of the camera image and further rotates the camera image. In FIG. 9B, since the distance to the traffic lights (SG1, SG2) is shorter than that in FIG. 9A and the vehicle body is inclined in the lateral direction, the camera image is also inclined. Therefore, as shown in FIG. 9C, the image adjustment unit 19 changes the scale of the camera image in FIG. 9B and rotates the camera image. For example, the camera image in FIG. 9B is reduced and rotated clockwise. Of course, FIG. 9A may be enlarged and rotated counterclockwise. The relative positions of the two traffic lights (SG1, SG2) in FIG. 9 (c) are maintained as in FIG. 9 (a).

  With reference to FIG. 10, an example of a series of information processing cycles executed by the traffic signal detector 12 according to the second embodiment for detecting a traffic signal from a camera image will be described. The process shown in FIG. 10 is repeatedly executed at a predetermined cycle.

  Steps S01 to S21 are the same as those in FIG. 4 and thus will not be described. Steps S23 to S27 added to FIG. 4 will be described. Proceeding to step S23 after step S11 or step S13, the position change calculation unit 16 determines whether or not the signal lamp determination unit 18 has detected two or more signal lamps in the same information processing cycle. When two or more signal lights are detected (YES in S23), the process proceeds to step S25, and the position change calculation unit 16 detects the relative positions of the two or more signal lights on the camera image. The position change calculation unit 16 detects the relative position between the camera images at different times for each of the two or more signal lights detected from the images at different times.

  Proceeding to step S27, when the relative position between the camera images at different times of the two or more signal lights has changed, the image adjustment unit 19 maintains the relative position between the images of the two or more signal lights at different times. In addition, at least one of the scale amount and the rotation amount of the camera image is calculated and set. By changing the scale of the camera image and rotating the camera image, it is possible to maintain the relative position between the images of two or more simultaneously detected signal lights at different times. If the relative positions of the two or more signal lamps at different times do not change, the scale of the camera image is not adjusted and the camera image is not rotated.

  When the scale amount and the rotation amount of the camera image are set in step S27, the image adjustment unit 19 changes the scale of the camera image and rotates the camera image in step S01 of the next information processing cycle.

  When the number of camera images superimposed to form one superimposed image increases, the displacement of the image scale and rotation due to the movement of the vehicle increases, and the alignment accuracy of the camera image decreases. Therefore, when two or more signal lights are detected at the same time, at least one of the scale and rotation of the image is adjusted so that the relative positions between the images at different times of the two or more signal lights are maintained. Thereby, not only the positional deviation in the image plane (xy plane) is corrected, but also the positional deviation in the normal direction (z direction) of the image plane can be corrected. Therefore, even if the number of images to be superimposed increases, the image alignment accuracy can be maintained.

(Third embodiment)
In the third embodiment, an example in which a minimum area (signal area) that can include the entire traffic signal is cut out from the camera image and two or more traffic signal areas are superimposed in order to reduce the image processing burden will be described. .

  As shown to Fig.11 (a), the image superimposition part 21 cuts out the traffic light area | region CT1 centering on the detected signal lamp SG from a camera image. The traffic light region CT1 has a size capable of arranging five signal lights in the vertical direction (y direction) and the horizontal direction (x direction) at the same interval as the interval between adjacent signal lights in the traffic light. The image superimposing unit 21 superimposes the traffic signal region CT1 cut out for each of the camera images to generate a superimposed image.

  By cutting out and superimposing the traffic signal area CT1 including the detected signal lamp SG from the camera image, the burden of image processing is reduced as compared with the case of superimposing the entire camera image. In general, the traffic light has three signal lights of red, blue and yellow, which are arranged in the vertical direction or the horizontal direction. Therefore, an area centered on the signal lamp and having a size that allows five signal lamps to be arranged in the vertical direction or the horizontal direction is cut out and superimposed. As a result, even in a situation where it is unclear which position of the three signal lights is lit, it is possible to cut out the minimum area that can include the entire traffic light.

  If it is known that the direction in which the three signal lights of red, blue, and yellow are arranged is the vertical direction (y direction), the image superimposing unit 21 is adjacent to each other as shown in FIG. The traffic signal area CT1 having a size that allows five signal lights to be arranged only in the vertical direction (y direction) at the same interval as the interval between the signal lights to be performed may be cut out. On the other hand, when it is found that the direction in which the signal lamps are arranged is the horizontal direction (x direction), the image superimposing unit 21 has the same interval as the interval between adjacent signal lamps as shown in FIG. You may cut out traffic signal area | region CT1 which has a magnitude | size which can arrange | position five signal lights only in a horizontal direction (x direction).

  Since the interval between adjacent signal lamps is determined in advance, if the diameter of the detected signal lamp SG can be detected, the interval between the signal lamps on the camera image can be calculated.

  Although the embodiments of the present invention have been described as described above, it should not be understood that the descriptions and drawings constituting a part of this disclosure limit the present invention. From this disclosure, various alternative embodiments, examples and operational techniques will be apparent to those skilled in the art.

DESCRIPTION OF SYMBOLS 11 Image pick-up part 12 Traffic light detection part 16 Position change calculation part 19 Image adjustment part 21 Image superimposition part 22 Signal lamp detection part 23 Image selection part 26 Traffic signal recognition part 28 Camera image (image)
CT1 to CT3 Signal area DA1 to DA3 Signal lamp LO Reference signal SG, SG1, SG2 Signal lamp

Claims (10)

An imaging unit mounted on a vehicle and capturing an image of the surroundings of the vehicle;
From the image, a signal lamp detection unit that detects a signal lamp whose luminance changes in synchronization with the AC cycle of power,
An image superimposing unit that superimposes the images at different times to generate a superimposed image so that the position between the images of the signal lights detected from the images at different times acquired by the imaging unit is maintained;
In the superimposed image, the signal lamp during lighting, with at least one frame of the signal lights and traffic lights Off, traffic signal detecting apparatus characterized by comprising a recognizing traffic recognizer traffic signal including the signal lamp . An imaging unit mounted on a vehicle and capturing an image of the surroundings of the vehicle;
From the image, a signal lamp detection unit that detects a signal lamp whose luminance changes in synchronization with the AC cycle of power,
An image superimposing unit that superimposes the images at different times to generate a superimposed image so that the position between the images of the signal lights detected from the images at different times acquired by the imaging unit is maintained;
A signal recognition unit that recognizes a signal including the signal light using the superimposed image;
The signal superimposing unit generates the superimposed image by superimposing two or more images at different times at which the luminance of the signal lamp is not more than a threshold value among the images at different times. apparatus.   The image superimposing unit includes an image selecting unit that selects an image at two or more different times at which the luminance of the signal lamp is equal to or less than a threshold value using a phase of a reference signal synchronized with an AC cycle of power. The traffic light detection device according to claim 2, wherein An imaging unit mounted on a vehicle and capturing an image of the surroundings of the vehicle;
From the image, a signal lamp detection unit that detects a signal lamp whose luminance changes in synchronization with the AC cycle of power,
An image superimposing unit that superimposes the images at different times to generate a superimposed image so that the position between the images of the signal lights detected from the images at different times acquired by the imaging unit is maintained;
A signal recognition unit that recognizes a signal including the signal light using the superimposed image;
The traffic light detection apparatus , wherein the image superimposing unit includes an image adjusting unit that adjusts a positional relationship between images at different times to be superimposed so that a position between images of the signal lamp is maintained . The image adjustment unit is configured to detect a difference between images at different times so that a relative position between the images at different times of the two or more signal lights is maintained when the signal lamp detection unit detects two or more signal lights at the same time. 5. The traffic light detection apparatus according to claim 4 , wherein at least one of scale and rotation is adjusted . An imaging unit mounted on a vehicle and capturing an image of the surroundings of the vehicle;
From the image, a signal lamp detection unit that detects a signal lamp whose luminance changes in synchronization with the AC cycle of power,
An image superimposing unit that superimposes the images at different times to generate a superimposed image so that the position between the images of the signal lights detected from the images at different times acquired by the imaging unit is maintained;
A signal recognition unit that recognizes a signal including the signal light using the superimposed image;
The image superimposing unit sets the exposure condition of the imaging unit so that the luminance of at least one of the signal lamp being extinguished and the frame of the traffic light in the superimposed image is equal to or higher than a lower limit value necessary for detection of the traffic light. A traffic signal detection device. An imaging unit mounted on a vehicle and capturing an image of the surroundings of the vehicle;
From the image, a signal lamp detection unit that detects a signal lamp whose luminance changes in synchronization with the AC cycle of power,
An image superimposing unit that superimposes the images at different times to generate a superimposed image so that the position between the images of the signal lights detected from the images at different times acquired by the imaging unit is maintained;
A signal recognition unit that recognizes a signal including the signal light using the superimposed image;
The image superimposing unit sets the number of images at different times to be superimposed so that the luminance of at least one of the signal lamp being turned off and the frame of the traffic light in the superimposed image is equal to or higher than a lower limit value necessary for detection of the traffic light. A traffic light detecting device. An imaging unit mounted on a vehicle and capturing an image of the surroundings of the vehicle;
From the image, a signal lamp detection unit that detects a signal lamp whose luminance changes in synchronization with the AC cycle of power,
An image superimposing unit that superimposes the images at different times to generate a superimposed image so that the position between the images of the signal lights detected from the images at different times acquired by the imaging unit is maintained;
A signal recognition unit that recognizes a signal including the signal light using the superimposed image;
The image superimposing unit is a region that is centered on the detected signal lamp and can arrange five signal lamps in the vertical direction or the horizontal direction at the same interval as the interval between adjacent signal lamps in the traffic light. The region is cut out from the images at different times,
The traffic light detection device , wherein the image superimposing unit generates the superimposed image by superimposing the regions cut out for each of the images at different times . Using an imaging unit mounted on the vehicle, image the surroundings of the vehicle to obtain an image,
From the image, a signal light whose brightness changes in synchronization with the AC cycle of power is detected,
The superimposed images are generated by superimposing the images at different times so that the positions between the images of the signal lights detected from the images at different times acquired by the imaging unit are maintained,
In the superimposed image, the signal light including the signal light is recognized by using a signal light that is turned on and at least one of a signal light that is turned off and a frame of the traffic light.
A traffic light detecting method. Using an imaging unit mounted on the vehicle, image the surroundings of the vehicle to obtain an image,
From the image, a signal light whose brightness changes in synchronization with the AC cycle of power is detected,
In the images at different times, the luminance of the signal lights is equal to or less than a threshold value so that the positions between the images of the signal lights detected from the images at different times acquired by the imaging unit are maintained. Two or more images at different times are superimposed to generate a superimposed image;
Recognize the traffic light including the signal light using the superimposed image.
A traffic light detecting method.

Images