JP2004310525A - Image processing device for vehicles - Google Patents

Abstract

An image processing apparatus for a vehicle capable of selecting an accurate lane marking at the start of detection or at the time of recovery from an undetectable state.
In a case where lane marking information is obtained by image recognition (steps S1 to S5) and a past white line recognition result is not accumulated, an estimated course of the vehicle is estimated from a vehicle state, and the estimated course and the recognition result are determined. Is compared with the traveling lane determined from (step S9). If the difference between the two is within a predetermined range, it is considered that the lane line recognition has been successful, and information on the recognized lane line candidates is output as white line information, and this information is stored in the white line recognition ECU 1 (step S10). .
[Selection diagram] Fig. 4

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to a vehicular image processing device for recognizing a lane in which a vehicle travels.
[0002]
[Prior art]
Development of automatic driving equipment that automatically runs on the set lane, technology to warn of inadvertent departure from the lane of the vehicle, and to activate a system that protects occupants by predicting collision after departure Is being promoted. In such a technology, it is necessary to accurately recognize the lane in which the host vehicle should travel. As a technology for this purpose, a boundary line that separates the lane by image recognition such as edge extraction or the like is acquired by acquiring an image in the traveling direction of the vehicle. A device that detects a white line) is known (for example, see Patent Document 1).
[0003]
[Patent Document 1]
JP-A-7-239996 (paragraphs 0013 to 0029, FIGS. 1 to 5)
[0004]
[Problems to be solved by the invention]
In this type of lane recognition technology, a white line dividing a road is recognized by image recognition based on image data acquired by an imaging device. However, the imaging conditions change significantly due to the movement of the vehicle. As a result, the obtained image data may not be suitable for white line recognition due to backlight, overexposure, underexposure, and the like. In addition, as a result of edge extraction, there may be many lane marking candidates, and it may be difficult to select a correct lane marking. When the image data that is inappropriate for recognition is part of a series of acquired images, it is sufficient to exclude such inappropriate image data from the recognition processing target. Further, a method of selecting a candidate based on past recognition data may be used. However, when it is difficult to use such past recognition data or image data, for example, when a lane recognition device is activated, a white line is detected for the first time, or when a state in which detection is not possible continues and then returns. When there are many lane marking candidates at the time of detection or the like, it is difficult to determine which candidate is adopted as the lane marking line.
[0005]
Therefore, an object of the present invention is to provide a vehicular image processing apparatus capable of selecting an accurate lane marking at the time of starting detection or returning from an undetectable state.
[0006]
[Means for Solving the Problems]
In order to solve the above-described problems, a vehicular image processing apparatus according to the present invention provides a vehicular image processing device that detects a road lane marking of a lane in which a vehicle travels by extracting edges from an image acquired by an imaging unit mounted on the vehicle. The processing device further includes means for detecting the traveling direction of the vehicle, and when the detection of the lane marking is started or when returning from the undetectable state, the traveling direction of the detected vehicle is compared with the direction of the lane marking. When both of them match within a predetermined range, the detection result of the road lane marking is output.
[0007]
At the start of detection or at the time of recovery from the undetectable state, the vehicle is likely to be traveling along the lane. Therefore, the traveling direction of the vehicle is compared with the extending direction of the candidate road lane line determined by the image recognition. If it is determined that the road is a lane marking, an accurate lane marking can be selected.
[0008]
Further, the vehicle is provided with instructing means for giving a predetermined instruction to the driver, and when the detection of the road lane marking is started or when the vehicle returns from the undetectable state, while comparing the traveling direction of the vehicle with the direction of the road lane marking, It is preferable that the instruction means instructs the driver to maintain the traveling lane.
[0009]
BEST MODE FOR CARRYING OUT THE INVENTION
Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In order to facilitate understanding of the description, the same components are denoted by the same reference numerals as much as possible in each drawing, and redundant description is omitted.
[0010]
FIG. 1 is a block diagram of a vehicle control device including a vehicle image processing device according to the present invention, and FIG. 2 is a perspective view showing a vehicle equipped with the same. The vehicle control device 100 includes a front camera 11 that acquires an image in front of the vehicle, and a white line recognition ECU 1 that recognizes a white line on the front course of the vehicle by image processing from the image acquired by the front camera 11 (the vehicle according to the present invention). Image processing device). Output signals of the yaw rate sensor 12, the steering angle sensor 13, and the vehicle speed sensor 14 are also input to the white line recognition ECU 1, and the recognition result of the white line recognition ECU 1 is transmitted to a PCS (Pre-Crash Safety) system via the in-vehicle LAN 2. 3, departure warning system 4, lane keeping system 5, etc. Further, the monitor device 42 and the speaker 43 are connected to the white line recognition ECU 1. Details of the PCS system 3, the departure warning system 4, and the lane keeping system 5 will be described later.
[0011]
The front camera 11 is disposed above a front window of the vehicle 200 (for example, behind a rearview mirror) as shown in FIG. 2, and an image in front of the vehicle 200, that is, an image of a traveling lane 300 in front of the vehicle (section). (Including the line 301).
[0012]
Hereinafter, the process of recognizing the traveling lane 300 from the image acquired by the front camera 11 by image processing will be specifically described. FIG. 3 is a processing flow of the image processing, and FIGS. 4 to 8 are diagrams for explaining the processing contents. The processing described below is performed by the white line recognition ECU 1 unless otherwise specified.
[0013]
First, an image as shown in FIG. 4 is obtained by the front camera 11 (step S1). An opposing lane 310 exists on the right side of the traveling lane 300 with the lane markings 301R and 301L as boundaries, and a lane marking 301X exists on the right side of the opposing lane 310. An oncoming vehicle 210 exists on the oncoming lane 310. The division line may be a continuous line or a discontinuous line such as a dotted line or a broken line. Since these division lines 301 are usually painted white, hereinafter, the painted portion is referred to as a white line. However, in the case where they are painted yellow other than white, or when they are divided by tacks, blocks, etc. It is included in the white line. Further, an auxiliary line may be installed in parallel with the dividing line 301 in some cases.
[0014]
The acquired image is subjected to A / D conversion to obtain a predetermined number of pixels (for example, the horizontal direction in FIG. 4 is the x direction, the vertical direction is the y direction, and x × y is 320 × 240 pixels) (for example, 320 × 240 pixels). The image data is converted into digital image data (8 gradations) and sent to the white line recognition ECU 1 (step S2). Of course, a digital camera capable of directly outputting digital data having a predetermined number of pixels and gradation may be used for the front camera 11, and the analog output of the front camera 11 may be AD-converted inside the white line recognition ECU 1. The converted digital image is substantially identical to the analog image shown in FIG.
[0015]
Next, the white line recognition ECU 1 extracts a white line candidate from the received digital image by performing edge extraction processing on the image (step S3). For example, a pixel whose luminance on the left side is lower than the threshold value and whose luminance on the right side pixel is equal to or higher than the threshold value is extracted as an edge, using the average luminance in the image as a threshold value. As a result, the pixel position corresponding to the left boundary line of the white line (partition line 301L, 301R, 301X, etc.) is extracted. The edge extraction is not limited to this. For example, a pixel whose luminance difference from the left pixel is equal to or larger than a threshold may be extracted as an edge. These were to extract the pixel position at the position corresponding to the left boundary line of the white line by extracting the edge that changes from dark to light. Conversely, by extracting the edge that changes from light to dark, the white line was extracted. May be extracted as the pixel position corresponding to the right boundary line of.
[0016]
FIG. 5 shows the extracted edge position superimposed on the original image. Black circles indicate the extracted edge positions, and broken lines indicate pixel positions in the Y direction. Here, the acquired image of the traveling lane 300 is obtained by projecting the traveling lane 300 on the image plane of the front camera 11. On the other hand, the positions of the lane markings 301L and 301R that are actually desired are positions with respect to the vehicle on the traveling lane 300. Therefore, by means of geometric transformation, with the vehicle as the origin (0, 0), the horizontal direction in the X-axis direction (the right direction as viewed from the vehicle is positive) is changed to the distance from the vehicle in the Y-axis direction (the larger the numerical value, the larger the value). A position corresponding to the edge pixel position on the traveling plane, which is expressed by taking forward (that is, being away from the vehicle), is obtained (step S4). This geometric transformation can be performed based on the mounting position of the front camera 11, the imaging direction, and the imaging area. Note that the image pickup direction and the like may be corrected by detecting the inclination of the vehicle body and the like with a plurality of vehicle height sensors. In addition, it is preferable that the image distortion caused by the optical aberration of the imaging system of the front camera 11 is also corrected. FIG. 6 shows the position of the extracted edge pixel with respect to the vehicle corresponding to the division line 301L obtained by the geometric transformation in this way.
[0017]
Next, a line / curve is extracted by performing Hough transformation on the result of the geometric transformation (step S5). Taking the case of straight line extraction as an example, when the position of the extracted edge point on the traveling plane is represented by (x, y), the inclination θ of each edge point is set to a predetermined angle Δθ from 0 ° to 180 °. And ρ corresponding to each θ is obtained as ρ = xcos θ + ysin θ. Then, the (ρ, θ) obtained in this manner is searched for in which of the areas divided at predetermined intervals in the Huff space, and a vote is made on the searched area. In other words, the number of votes in each area indicates the number of Hough transform results of edge points in this area. This voting is preferably performed separately for the left side and the right side of the traveling lane 300. The center line of the driving lane 300 required to perform the left and right division of the driving lane 300 can be grasped by referring to the lane recognition results, the traveling direction of the vehicle, the yaw rate, the steering angle, and the like. it can. FIG. 7 shows a voting result in the Hough transform area on the left side of the traveling lane 300 obtained in this manner.
[0018]
A (ρ, θ) region having a plurality of votes is shown as a straight line extracted from an edge point (even if the region is intermittent, it is also extracted as one line). This straight line corresponds to y = ρ / sin θ−x × cos θ / sin θ on the (x, y) plane. Here, the larger the voting result (ρ, θ), the closer the edge point is to a straight line. In consideration of the influence of noise and the like, a straight line having a vote count equal to or more than a threshold value (3, preferably) is set as an extraction result. In FIG. 7, the area where the number of votes is 2 indicates this noise.
[0019]
Here, the case of a straight line has been described as an example, but a curve can be similarly extracted in the Hough transform. Then, a straight line or a curve that has a large number of votes and is considered to be the most similar is set as a division line candidate as an extraction result.
[0020]
Next, it is determined whether or not past white line recognition results are stored in the white line recognition ECU 1 (step S6). Here, the past recognition result is determined based on whether or not the lane line information recognition result up to the past which is a predetermined time earlier than the present is stored. At this time, it is not necessary that all the lane line information recognition results up to the past traced back by the predetermined time are stored. Even if erroneous recognition occurs, control based on past recognition information is possible, which is preferable.
[0021]
In step S6, if the lane marking information recognition results up to the past that are traced back by the predetermined time are stored, it indicates that the lane recognition has succeeded continuously for a predetermined time or more after starting the lane recognition. The process proceeds to step S7, and it is checked whether the information of the lane line candidate recognized this time matches the past recognition result. When it is determined that the deviation between the lane marking candidate and the past recognition result is within a predetermined range and that the lane marking is consistent, information on the recognized lane marking candidate is output as white line information, and this information is stored in the white line recognition ECU 1. (Step S10).
[0022]
On the other hand, when it is determined that the deviation is beyond the predetermined range and is not matched, the information of the recognized lane line candidate is discarded and the lane line information estimated from the past recognition result is output (step S1). S8).
[0023]
If it is determined in step S6 that the lane line information recognition result up to the past that has been traced by the predetermined time has not been stored, the process proceeds to step S9. Such a case in which the lane line information recognition result up to the past that is traced back by the predetermined time is not stored is when the predetermined time has not elapsed since the start of the lane recognition (in addition to the activation of the lane recognition means, the parking lot・ There is a time when the vehicle enters a road from a parking area, etc.), when the stored recognition result is lost due to a momentary interruption or some trouble, or when the vehicle returns from a state where lane recognition cannot be performed for a predetermined time or more (for example, , A construction section or a part that is shadowed). In such a case, the consistency between the past recognition result and the current recognition result cannot be checked. Therefore, in such a case, the estimated course of the vehicle 200 is estimated from the vehicle state such as the yaw rate, steering angle, and vehicle speed output from the yaw rate sensor 12, the steering angle sensor 13, and the vehicle speed sensor 14. Is compared with the traveling lane 300 determined from the recognition result of the image processing (step S9).
[0024]
When the deviation between the predicted course and the traveling lane 300 obtained from the recognition result is within a predetermined range, it is considered that the lane line recognition has been successful, and information on the recognized lane line candidate is output as white line information, and this information is output. It is stored in the white line recognition ECU 1 (step S10). Whether or not the deviation between the predicted course and the traveling lane obtained by the image recognition processing is within a predetermined range can be verified as follows, for example. The determination is made based on whether or not the predicted course deviates from the traveling lane for a preset distance or a preset time. Specifically, the determination may be made based on whether or not the angle between the predicted course and the traveling lane is within a predetermined angle. Further, the determination can be made based on whether or not the curvature of the predicted traveling course and the curvature of the traveling lane deviate by a predetermined value or more.
[0025]
On the other hand, when the deviation between the predicted course and the traveling lane 300 obtained from the recognition result exceeds a predetermined range (when the lane marking recognition results 302R and 302L are as shown in FIG. 8), the lane marking is erroneously recognized. Considering this, the recognized lane marking candidate is discarded, and the process is terminated (step S11).
[0026]
In order to determine the success or failure of the lane line information detection based on the vehicle state in this way, the vehicle 200 needs to be traveling correctly in the traveling lane 300, and when the driver changes the lane or the like. However, even if the recognition is successful, it is determined that the recognition is erroneous. Therefore, when the past white line information is not stored, it is desirable that the driver is notified by the monitor device 42 and the speaker 43 that the route and the lane line information are being compared. By this notification, the driver recognizes that the white line recognition ECU 1 is comparing the road and the lane marking information, and it is better to maintain the currently running lane for comparison of these information. Care must be taken to maintain the driving lane, as this will lead to improved accuracy. Furthermore, the correctness of the past lane information may be tested by comparing the trajectory of the vehicle 200 with the actual traveling lane and the traveling lane predicted from the past recognition result.
[0027]
According to the present invention, it is possible to test whether or not the lane marking information obtained by image recognition is correct lane marking information without requiring a special operation by the driver immediately after the start of lane recognition. Therefore, it is possible to suppress vehicle behavior control using erroneously detected lane marking information, so that the driver does not feel uneasy and operability is improved.
[0028]
The image recognition means of the division line (white line) is not limited to the recognition method described above, and various recognition methods can be used. Alternatively, the recognized lane marking information may be verified based on map information or road information. For example, when a curve or the like having an extremely small radius of curvature is detected while traveling on a highway, it can be determined that this is the only erroneous detection.
[0029]
Next, vehicle control in the vehicle control device according to the present invention using the white line information thus obtained will be described. 9 to 11 are block diagrams of the PCS system 3, the departure warning system 4, and the lane keeping system 5 shown in FIG.
[0030]
First, the PCS system 3 shown in FIG. 9 will be described. The PCS system includes a braking system 32 for controlling a braking force, a steering assist motor 33 for applying an assisting force to a driver's steering force, a seat belt pretensioner 34 for winding up a seat belt, and an airbag 35 for occupant protection. And a PCS / ECU 31 for performing these controls. The PCS / ECU 31 has a laser radar 16 for detecting a preceding vehicle and an obstacle, a yaw rate sensor 12, a steering angle sensor 13, a vehicle speed sensor 14, and a brake pedal. The output of the brake pedal switch 15 for detecting the operation or non-operation state is input.
[0031]
The PCS / ECU 31 detects a preceding vehicle and an obstacle on the course of the own vehicle based on lane information (white line information) sent from the lane recognition ECU 1 and outputs of the sensors 12 to 15 and the laser radar 16. Predict the likelihood of a collision. When it is determined that a collision is inevitable, the braking system 32 is instructed to switch to a setting in which the assist force applied during the brake operation is increased in advance, so that when the driver senses a danger and operates the brake pedal, a large force is applied. Make sure that braking force is obtained.
[0032]
Further, the assist steering force by the steering assist motor 33 is increased so that the driver can perform steering with a weaker force. If necessary, steer automatically to avoid collision.
[0033]
Further, by winding the extra length of the seat belt in advance by the seat belt pretensioner 34 before the collision, the occupant is more strongly restrained by the seat and the damage at the time of the collision is reduced. In addition, by operating the airbag at an optimal timing before and after the collision, not after detecting the collision, damage to the occupant at the time of the collision is reduced. With these controls, even when a collision cannot be avoided, it is possible to suppress serious damage to the vehicle and the occupant and reduce the shock of the collision.
[0034]
Next, the departure warning system 4 shown in FIG. 10 will be described. The departure warning system 4 includes a departure warning ECU 41, a monitor device 42 for display, and a speaker 43 for sound output. The departure warning ECU 41 has lane information (white line information), vehicle speed information, yaw rate, and the like. Information has been entered.
[0035]
The departure warning ECU 41 predicts the course of the host vehicle from the vehicle speed / yaw rate information and compares this with the lane information. When the vehicle is expected to depart from the lane, a message indicating that the vehicle is expected to deviate from the lane is displayed on the monitor device 42, and an alarm is generated by the speaker 43 to generate an alarm sound or sound output. Notify to the effect.
[0036]
Next, the lane keeping system 5 shown in FIG. 11 will be described. The lane keeping system 5 includes a lane keeping ECU 51, a braking system 32 for controlling a braking force, a steering assist motor 33 for applying an assisting force to a driver's steering force, and a speed changing means 52 for changing a transmission shift state. The lane keeping ECU 51 includes a laser radar 16 for detecting a preceding vehicle and an obstacle, a yaw rate sensor 12, a steering angle sensor 13, a vehicle speed sensor 14, and operation of a brake pedal. The outputs of a brake pedal switch 15 for detecting a non-operation state and an accelerator opening sensor 17 for detecting an operation amount of an accelerator pedal are input.
[0037]
The lane keeping system 5 travels while maintaining the position of the vehicle in the traveling lane 300 currently traveling. Specifically, by predicting the course of the host vehicle from the outputs of the yaw rate sensor 12, the steering angle sensor 13, and the vehicle speed sensor 14, and comparing this with the lane information, it is determined whether traveling in the traveling lane 300 has been maintained. Is determined. Then, based on the front lane information, the steering assist motor 33 is controlled so that the own vehicle travels within the white line from the position of the own vehicle and the white line, so that the traveling in the traveling lane 300 is maintained.
[0038]
As described above, by detecting the lane marking of the traveling lane of the own vehicle using the vehicular image processing device according to the present invention, it becomes possible to recognize the lane marking, and thus the traveling lane at an early stage. Thus, it is possible to accurately control the behavior of various vehicles using the same.
[0039]
【The invention's effect】
As described above, according to the present invention, the correctness of the lane line information is verified by predicting the course of the vehicle based on the vehicle state and comparing this with lane line information obtained by image recognition. Thus, the lane line information can be verified without any special operation by the driver. At this time, if the driver is instructed to maintain the lane, accurate lane marking information can be obtained.
[Brief description of the drawings]
FIG. 1 is a block diagram of a vehicle control device including a vehicle image processing device according to the present invention.
FIG. 2 is a perspective view showing a vehicle equipped with the vehicle control device of FIG. 1;
FIG. 3 is a processing flow of image processing in the vehicle image processing apparatus of FIG. 1;
FIG. 4 is a diagram illustrating an example of an image acquired by a front camera.
FIG. 5 is a diagram showing a result of performing edge extraction processing from the image of FIG. 4;
FIG. 6 is a diagram showing a result of the geometric conversion processing of FIG. 5;
FIG. 7 is a diagram showing a result of the Hough transform processing of FIG. 6;
8 is a diagram illustrating an example of a case where a division line is erroneously recognized from the image of FIG. 4;
FIG. 9 is a block diagram of the PCS system shown in FIG. 1;
FIG. 10 is a block diagram of the departure warning system shown in FIG. 1;
FIG. 11 is a block diagram of the lane keeping system shown in FIG. 1;
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 1 ... White line recognition ECU, 2 ... In-vehicle LAN, 3 ... PCS (Pre-Crash Safety) system, 4 ... Departure warning system, 5 ... Lane keep system, 11 ... Front camera, 12 ... Yaw rate sensor, 13 ... Steering angle sensor, 14: vehicle speed sensor, 15: brake pedal switch, 16: laser radar, 17: accelerator opening sensor, 31: PCS / ECU, 32: braking system, 33: steering assist motor, 34: seat belt pretensioner, 35: air Bag, 41: departure warning ECU, 42: monitoring device, 43: speaker, 51: lane keeping ECU, 52: transmission means, 53: electronic control throttle, 100: vehicle control device, 200: vehicle, 300: traveling lane, 301 … Compartment line, 302… compartment line recognition result (erroneous recognition line).

Claims (2)

An image processing apparatus for a vehicle that detects a lane marking of a lane in which the vehicle travels by image recognition from an image acquired by an imaging unit mounted on the vehicle,
The vehicle further includes means for detecting the traveling direction of the vehicle. When the detection of the lane marking starts or when the vehicle returns from the undetectable state, the detected traveling direction is compared with the direction of the lane marking to determine whether the vehicle is traveling. An output processing unit that outputs a detection result of a road demarcation line when the distance is within the range. It further includes an instruction means for giving a predetermined instruction to the driver, and at the time of starting detection of a lane marking or returning from an undetectable state, while comparing the traveling direction of the vehicle with the direction of the lane marking, The image processing device for a vehicle according to claim 1, wherein the instruction unit instructs the driver to maintain a driving lane.

Images