KR101699014B1 - Method for detecting object using stereo camera and apparatus thereof - Google Patents

Abstract

The present invention relates to a method and an apparatus for detecting an object using a stereo camera. According to another aspect of the present invention, there is provided a method of generating a stereoscopic image, the method comprising: generating a V-disparity map and a U-disparity map using a depth map acquired from an image of a stereo camera; Determining a position and a size of a plurality of ROIs by adjusting a height of the ROI in consideration of a representative height value of a predetermined ROI; Integrating the objects of interest into a single area of interest, and outputting the detection results of the objects of interest in each of the areas of interest, determining whether the objects of interest are present.
According to the object detection method and apparatus, the size of the region of interest is adjusted by reflecting the height of the object of interest, and the object region is detected by arranging the overlapping regions of interest with high similarity, thereby reducing the search region required for object detection, Can be improved at the same time.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method and apparatus for detecting an object using a stereo camera,

The present invention relates to a method and apparatus for detecting an object using a stereo camera, and more particularly, to a method and apparatus for detecting an object using a stereo camera capable of improving detection speed and performance.

Pedestrian detection technology is used in various fields such as intelligent automobile, robot, and video security. More than 1000 related papers have been published over the past 10 years, and active research is underway, and pedestrian detection performance is gradually improving.

Pedestrian detection technology is applied to intelligent vehicles such as Advanced Driver Assistance Systems (ADAS), which assists the driver and protects the lives of pedestrians. The accuracy and real - time nature of the pedestrian detection technique is very important because it is related to the life of the pedestrian.

Pedestrian detection is a technique for finding a pedestrian from an image of a camera. The pedestrian is detected by extracting features from a learning image and then searching for a test image using the feature. HOG (Histogram of Oriented Gradients) is a typical feature used for pedestrian detection.

A typical pedestrian detection method searches for a multi-scale sliding window in a test image based on characteristics such as HOG, and uses a classifier such as SVM (Support Vector Machine) , And image scale. And eliminates unnecessary results from the detected objects using techniques such as NMS (non-maximum suppression). The multi-scale sliding window technique is simple in structure, but its processing speed is very slow because it searches all areas.

Single camera based pedestrian detection techniques search and search for pedestrians based on multi - scale sliding window technique. These methods have limitations in improving the detection speed because they can not reduce the search area without prior information or additional information.

Stereo camera based object detection method generates depth map using disparity of two cameras and extracts each object using it. Using a depth map of a stereo camera, information of a region of interest (ROI) can be obtained quickly, and the speed and performance of the object can be improved by reducing the object search area.

However, a depth map obtained from a stereo camera generally has a stereo matching error. Such a matching error may cause an error in generating a region of interest, and an object that can not be detected as a classifier may occur.

The technology that provides the background of the present invention is disclosed in Korean Patent No. 0929569 (published on Dec. 3, 2009).

An object of the present invention is to provide a method and an apparatus for detecting an object using a stereo camera capable of reducing detection speed and performance by reducing an object search area.

The present invention provides a method of generating stereoscopic images, the method comprising: generating a V-parallax map and a U-parallax map using a depth map obtained from an image of a stereo camera, Determining a position and a size of a plurality of ROIs, adjusting a height of the ROI in consideration of a representative height value of a predetermined ROI, determining overlapping ROIs having overlapping ROIs among the ROIs, Integrating the object of interest into a single region of interest, and determining whether the object of interest exists in each of the regions of interest and outputting a detection result of the object of interest.

Here, the step of adjusting the height of the ROI may include determining a horizontal position, a width, a vertical position and a height of the ROI in the image using the U-parallax map and the V-parallax map, The height of the ROI can be adjusted by using a height end point obtained by offsetting the representative height value of the ROI at the start point of the ROI detected in the V-parallax map.

The object detection method may further include expanding a size of each region of interest in which the height is adjusted to a predetermined magnification.

In addition, the step of integrating the overlapping ROIs may include calculating the degree of similarity between the ROIs, and then combining the ROIs into one ROI only when the degree of similarity is determined to be equal to or greater than the threshold value.

Further, the step of integrating the overlapping region of interest is, when the _s f (h _i, h _j) of the equation below corresponding to the threshold degree of similarity than can incorporate the overlapping region of interest in a region of interest.

Here, f _s (h _i , h _j ) is a value between 0 and 1, the closest to 1, the higher the degree of similarity, and the f _o (h _i , h _j ) the total area (R ∪R _{hi hj)} and the ratio of the overlapping area (R ∩R _{hi hj) contrast, f d (h i, h} j) are the first and second region of interest (h _i, h _j) (| d _hi -d _hj |) of the (h _i , h _j ), and λ represents a predetermined positive value.

According to another aspect of the present invention, there is provided a stereoscopic image display apparatus comprising a parallax map generator for generating a V-parallax map and a U-parallax map using a depth map obtained from an image of a stereo camera, An interest region generation unit for determining a position and a size of a plurality of ROIs generated in an image and adjusting a height of the ROI by considering a representative height value of a predetermined ROI; An interest detection unit for detecting whether or not the ROI exists in each of the ROIs, and a result output unit outputting a detection result of the ROIs, And an object detecting apparatus using the stereo camera.

According to the method and apparatus for detecting an object using a stereo camera according to the present invention, the size of the region of interest is adjusted by reflecting the height of the object of interest and the overlapping ROIs having high similarity are arranged, There is an advantage that the area can be reduced and the detection speed and performance can be improved at the same time.

1 is a block diagram of an object detecting apparatus using a stereo camera according to an embodiment of the present invention.
2 is a block diagram of an object detection technique according to an embodiment of the present invention.
3 is a flowchart illustrating an object detection method according to an embodiment of the present invention.
4 is a diagram illustrating an example of generating a depth map using an image of a stereo camera in an embodiment of the present invention.
FIG. 5 is a diagram showing the results of generating the V-parallax map and the U-parallax map from the depth map of FIG. 4. FIG.
FIG. 6 is a view showing the V-parallax map shown in FIG. 5. FIG.
FIG. 7 is a diagram illustrating a result of generating a ROI considering the height of a pedestrian in FIG.
8 is a view for explaining a rearrangement technique of a region of interest in an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily carry out the present invention.

The present invention relates to a method and apparatus for detecting an object using a stereo camera, and more particularly, to a technique for effectively generating regions of interest from an image of a stereo camera, a technique for correcting the size of the generated region of interest, We propose a method that can reduce the object search area and improve the detection speed and performance of the object by using the summarization technique.

FIG. 1 is a block diagram of an object detecting apparatus using a stereo camera according to an embodiment of the present invention. FIG. 2 is a diagram illustrating an object detecting method according to an exemplary embodiment of the present invention.

1 and 2, an object detecting apparatus 100 according to an embodiment of the present invention includes a stereo image input unit 110, a depth map generating unit 120, a parallax map generating unit 130, An interest region correcting unit 160, an object detecting unit 170, and a result outputting unit 180. The object region correcting unit 140, the interest region correcting unit 150,

The stereo image input unit 110 receives the image captured by the stereo camera. Specifically, the left and right images of the stereo camera are respectively input. 2 (a) shows one of two images for convenience of explanation.

The depth map generation unit 120 generates a depth map using the left and right images of the input stereo camera. As shown in Fig. 2 (b), the depth map appears as a bright value near the camera and a dark value far from the camera.

The parallax map generator 130 generates a V-parallax map and a U-parallax map using a depth map. The V-parallax map and the U-parallax map are used to create the ROIs in the image.

The depth map generation process, the parallax map generation process, and the ROI generation process may be basically used in the embodiment of the present invention. However, in the embodiment of the present invention, the height of the ROI is adjusted in consideration of the height of the object of interest (ex, pedestrian) when the ROI is generated.

The ROI generator 140 determines the position and size of a plurality of ROIs generated in the image using the V-parallax map and the U-parallax map, and determines a representative height value The height of each area of interest is adjusted. FIG. 2C shows that a plurality of ROIs are generated in the image by using the two parallax maps and reflecting the height of the pedestrian.

The ROI correction unit 150 enlarges and corrects the size of the initial ROI that has been generated. FIG. 2 (d) shows the initial region of interest enlarged and corrected. Initial interest areas created using the pedestrian height can be created with too tight a height. If the height of the area of interest is too small, the detector may not be able to search for pedestrians at a later time. Therefore, the search rate is increased by further performing the operation of increasing the size of the region of interest.

The interest region organizing unit 160 reduces the object search region and increases the detection speed by arranging the overlapping regions of interest among the regions of interest. Figure 2 (e) shows a unified view of overlapping regions of interest. The reason for sorting the overlapping interest regions is that, in the case of the object detection method using a stereo camera, a plurality of regions of interest may be generated redundantly for one object due to a stereo matching error, thereby unnecessarily increasing the object search region This is because the detection speed and performance may be greatly reduced.

The object detecting unit 170 determines whether an object of interest exists in each ROI. The object detection can use a conventional classifier and can identify the area of interest in which the pedestrian exists and confirm at which point the pedestrian is detected in the image.

The result output unit 180 outputs the detection result of the pedestrian by the object detection unit 170 to the screen. Fig. 2 (f) is a view showing a point where the pedestrian is detected in the image by a separate box (ex, rectangle).

Hereinafter, a method for detecting an object using a stereo camera according to an embodiment of the present invention will be described in detail. 3 is a flowchart illustrating an object detection method according to an embodiment of the present invention. Hereinafter, for convenience of explanation, it is exemplified that the object of interest is a pedestrian.

First, the stereo image input unit 110 receives an image of a stereo camera (S310), and the depth map generation unit 120 generates a depth map using the inputted stereo camera image (S320).

4 is a diagram illustrating an example of generating a depth map using an image of a stereo camera in an embodiment of the present invention. You can generate a depth map from left and right images using a common stereo matching method.

Each pixel composing the depth map has a brightness value from 0 (black) to 255 (white). Referring to FIG. 4, it can be seen that the farther from the camera the darker the color is. Thus, the depth map represents the distance (depth) value from the camera to the brightness level.

Thereafter, the parallax map generation unit 130 generates the V-parallax map and the U-parallax map using the depth map (S330). Each of the parallax maps is data generated by accumulating the distance values expressed in the depth map in the horizontal or vertical direction.

FIG. 5 is a diagram showing the results of generating the V-parallax map and the U-parallax map from the depth map of FIG. 4. FIG. It is the V-parallax map represented on the left side of the depth map image and the U-parallax map represented on the lower side.

5, the V-disparity map is data obtained by accumulating the distance value corresponding to the width in the depth map image. The horizontal axis of the V-parallax map corresponds to a distance value of 0 to 255 (left of the horizontal axis: 0, right: 255), and the vertical axis corresponds to the vertical axis of the depth map image.

In a similar principle, the U-disparity map is data in which the distance value corresponding to the vertical is accumulated in the depth map image, and is different from the V-disparity map only in the accumulation direction. The vertical axis of the U-parallax map corresponds to a distance value of 0 to 255 (upper end of the vertical axis: 255, lower end: 0), and the horizontal axis corresponds to the horizontal axis of the depth map image.

FIG. 6 is a view showing the V-parallax map shown in FIG. 5. FIG. FIG. 6 shows a screen magnification adjusted for convenience of explanation. In the V-parallax map, the plane components such as roads are diagonally shaped, and the obstacles on the roads appear as vertical lines.

That is, in FIG. 6, the diagonal type data represents a road line. If there are no obstacles (trees, vehicles, pedestrians, etc.) on the road, only diagonal data appears on the V-parallax map. In Fig. 6, the vertical data (vertical direction) represents an obstacle. The obstacle has a shape in which the distance value (brightness value) is constant in the vertical direction on the V-parallax map.

In this way, the V-parallax map captures the approximate vertical position of the obstacle in the image, and the U-parallax map serves to approximate the horizontal position of the obstacle. That is, the U-disparity map takes charge of the vertical position and the size of the ROI from the V-disparity map, and the U-disparity map takes charge of the horizontal position and size of the ROI.

Therefore, using the V-parallax map and the U-parallax map generated in step S330, ROIs can be generated in the image. If only the data of the V-parallax map and the U-parallax map are used, since the initial interest area is large and the size is large, the embodiment of the present invention adjusts the height of the individual area of interest by considering the height of the pedestrian.

That is, after step S330, the ROI generator 140 determines the position and size of a plurality of ROIs (rectangular boxes) generated in the image through the V-parallax map and the U-parallax map, The height of each ROI is adjusted in consideration of the representative height value (S340).

The ROI generator 140 basically determines the horizontal position and the width of the ROI in the image using the U-parallax map, and determines the vertical position and height of the ROI using the V-parallax map. Here, the height of the area of interest is adjusted considering the general height of the pedestrian.

The region of interest itself is a candidate region that is determined by the parallax map to have an obstructed object. Within this area of interest, there may be pedestrians or other objects. In the embodiment of the present invention, since the object to be detected is a pedestrian, the size of the area of interest is adjusted in consideration of the representative height value of the pedestrian.

FIG. 7 is a diagram illustrating a result of generating a ROI considering the height of a pedestrian in FIG. The dotted line area shown in FIG. 7 indicates that the object to be detected is a pedestrian in the present invention.

Each area of interest is marked with a black square box. In this case, since each depth of interest in the image is different, each of the ROIs is represented by boxes of different heights corresponding to the respective depths. The farther away you are from the actual camera, the smaller the height.

The control principle of the area of interest is as follows. In step S340, the ROI generator 140 offsets the representative height value of the pedestrian based on the height starting point of the ROI detected in the V-parallax map to obtain the height end point of the height starting point. Therefore, the height from the start point to the end point is determined as the height of the region of interest. This process applies the same principle to all areas of interest.

The region of interest determination process will be described mathematically with reference to FIG. The embodiment of the present invention generates an initial ROI using the ROI function considering the height of a pedestrian shown in Equation (1) below.

Here, R _Bk and R _Tk mean the height starting point and the height ending point of the region of interest, and C _{H (k)} represents the representative height value of the pedestrian. The ak + b used to determine the R _Bk value is a linear function of the line of the diagonal line appearing on the V-parallax map. floor (·) is a decimal point decrement function for the value in parentheses, making the value an integer. Of course, it is also possible to use the round function instead of the round function for integer expressions here.

First, the height starting point (R _Bk ) of the ROI is searched using the first equation (R _Bk = floor (ak + b)) of Equation (1). The height starting point corresponds to the ordinate axis coordinate value corresponding to the intersection point where the diagonal line and the vertical line meet in the V-disparity map of FIG. Therefore, by substituting the abscissa coordinate value (k: a value between 0 and 255) corresponding to the intersection into the first order function ak + b, the position of the starting point of the height R _Bk of the corresponding region in the image can be confirmed.

Then, the high end point (R _TK ) is obtained by using the second equation (R _TK = R _Bk -C _{H (k)} +1) in Equation (1). That is, the representative height value (C _{H (k)} ) of the pedestrian is offset at the height starting point (R _Bk ) to obtain the height end point with respect to the height starting point. This is not an essential element, but it can be 1, 0, or some other value depending on the starting position of the (0, 0) coordinate point in the image. As described above, using the height start point and end point of the region of interest, the vertical position and size of the initial region of interest are determined, and the horizontal position and the size can be obtained from the U-parallax map.

In this case, when the pedestrian height value is treated as a minus offset, the left upper end of the image becomes the reference point coordinate (0, 0), so that the vertical axis value of all pixels of the image corresponding to the lower part of (0, 0) . Accordingly, the height of the object in the image has a minus size when viewed from (0, 0). Therefore, if both R _Bk and C _{H (k)} are negative values, it is possible to obtain the C _{H (k)} position offset upward from R _Bk in the actual image even if C _{H (k)} is subtracted mathematically from R _Bk .

Of course, if the lower left corner of the image is (0, 0), the vertical axis values of the image can be represented as positive values, and R _TK can be obtained by adding C _{H (k)} to R _Bk . As described above, the sign used in the offset process can be different according to the reference point position of the image. As described above, the embodiment of the present invention adjusts the height of the initial ROI in consideration of the height of the actual pedestrian, as described above.

Next, the ROI corrector 150 expands the size of each ROI to a predetermined magnification (S350). Before and after enlargement of the area of interest is shown in FIGS. 2 (c) and 2 (d). This enlargement correction of the area of interest enhances the search performance of the pedestrian in the detector.

If the initial interest region shown in (c) of FIG. 2 is extended in four directions with respect to the center point, the shape becomes as shown in (d) of FIG. At this time, the area of the ROI can be enlarged to a predetermined magnification in consideration of the margin of the general pedestrian size. For example, the magnification can be set such that the height of the area of interest is about twice the representative height value of the pedestrian.

Next, the interest region organizing unit 160 integrates the redundant interest regions having mutually overlapping regions among a plurality of regions of interest into a single region of interest (S360). See Fig. 2 (e) for the result of incorporating overlapping regions of interest.

In step S360, f _s (h _i , h _j ) corresponding to the degree of similarity between the overlapping ROIs is calculated using Equation (2) below, and the overlapping ROIs are calculated only when the degree of similarity is determined to be equal to or greater than the threshold To the domain of interest.

Here, f _s (h _i , h _j ) is a value between 0 and 1, and the closer to 1, the higher the degree of similarity. f _s (h _i , h _j ) is the product of f _o (h _i , h _j ) and f _d (h _i , h _j ).

_o f (h _i, h _j) is a redundant area compared to the total area (R ∪R _{hi hj)} of the first and second regions of interest (h _i, h _j) having a mutually overlapping region (R ∩R _{hi hj)} , And f _d (h _i , h _j ) is a value that increases as the depth difference (| d _hi -d _hj |) of the first and second regions of interest (h _i , h _j ) Represents a predetermined positive value.

8 is a view for explaining a rearrangement technique of a region of interest in an embodiment of the present invention. FIG. 8A shows a redundant area (R _hi ∩R _hj ) for two areas of interest, and FIG. 8B shows a total area (R _hi ∪R _hj ) for two areas of interest. If two interest regions are integrated into one, as shown in FIG. 8 (b), one large rectangle can be integrated with respect to the non-overlapping outer edges of the two regions of interest.

f _o (h _i , h _j ) = (R _hi ∩R _hj ) / (R _hi ∪R _hj ), which has a value between 0 and 1. f _o (h _i , h _j ) is a numerical value indicating how much the two regions of interest are overlapped.

f _d (h _i , h _j ) is a numerical value indicating how close the distance (depth) between the two regions of interest is, and has a value between 0 and 1. The reason why the value of f _d (h _i , h _j ) is additionally used in determining the degree of similarity is because two regions of interest may overlap each other but actually correspond to different objects at different depths.

The similarity f _s (h _i , h _j ) is calculated by multiplying f _o (h _i , h _j ), which is the degree of overlap between the two regions of interest, and f _d (h _i , h _j ) If the computed f _s (h _i , h _j ) is greater than or equal to a predetermined threshold value (ex, 0.5), the two regions of interest are grouped together. Depending on how the threshold is set, the number of regions to be merged may vary and the computational complexity may vary.

After the region of interest is determined as described above, the object detecting unit 170 determines whether an object of interest exists in each RO using the pedestrian classifier (S370). Accordingly, it is possible to identify the area of interest in which the pedestrian exists and detect the detection position of the pedestrian in the image.

The result output unit 180 outputs the pedestrian detection result by the object detection unit 170 on the image as shown in (f) of FIG. 2 (S380). For example, a portion where a pedestrian is detected is finally displayed and provided in the form of a black square on the output screen.

As described above, the present invention easily generates a ROI using a stereo camera, determines whether a pedestrian exists in an ROI using a classifier, and displays the ROI on an output screen. In addition, the present invention can effectively improve the detection speed and the performance by using a classifier learned in a high-performance computer for a long time by effectively creating a ROI considering the height of a pedestrian.

The present invention can be utilized in fields such as intelligent vehicles, CCTV, security, etc., which need to accurately and quickly detect pedestrians. In the case of an intelligent vehicle, pedestrian traffic accidents on the road can be reduced by utilizing the pedestrian detection technique of the present invention. CCTV can also be used to detect and track criminals, and can be used as a way to determine whether or not an outside person is intruding into a space where there is no human being in the security field.

According to the method and apparatus for detecting an object using the stereo camera according to the present invention, the size of the region of interest is adjusted by reflecting the height of the object of interest, and the objects are detected by arranging the overlapping regions of interest having high similarity. There is an advantage that the required search area can be reduced and the detection speed and performance can be improved at the same time.

While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims. Accordingly, the true scope of the present invention should be determined by the technical idea of the appended claims.

100: Object detection apparatus 110: Stereo image input unit
120: depth map generating unit 130:
140: ROI generating unit 150: ROI generating unit
160: an interest region rearrangement unit 170:
180: Result output unit

Claims (10)

Generating a V-parallax map and a U-parallax map using a depth map obtained from an image of a stereo camera;
A position and a size of a plurality of ROIs generated in the image are determined through the V-parallax map and the U-parallax map, and a height of the ROI is adjusted considering a representative height value of a predetermined ROI step;
Integrating overlapping interest regions having mutually overlapping regions among the plurality of regions of interest into a single region of interest; And
Determining whether the object of interest exists in each of the regions of interest, and outputting a detection result of the object of interest;
Wherein the step of merging the overlapping ROIs comprises:
Calculating a degree of similarity between the overlapping ROIs, and integrating the ROIs into one ROI only when the degree of similarity is determined to be equal to or greater than a threshold value. The method according to claim 1,
Wherein adjusting the height of the region of interest comprises:
A horizontal position, a width, a vertical position, and a height of the ROI in the image using the U-disparity map and the V-disparity map,
And adjusting a height of the ROI by using a height end point obtained by offsetting a representative height value of the ROI at a starting point of the ROI detected in the V-parallax map. The method according to claim 1,
And expanding the size of each region of interest with the height adjusted to a preset magnification. delete The method according to claim 1,
Wherein the step of merging the overlapping ROIs comprises:
If more than _{f s (h i, h j} ) of the equation below corresponding to the degree of similarity threshold, the duplicate object using a stereo camera, the region of interest integrated into a single region of interest is detected by:

Here, f _s (h _i , h _j ) is a value between 0 and 1, the closest to 1, the higher the degree of similarity, and the f _o (h _i , h _j ) the total area (R ∪R _{hi hj)} and the ratio of the overlapping area (R ∩R _{hi hj) contrast, f d (h i, h} j) are the first and second region of interest (h _i, h _j) (| d _hi -d _hj |) of the (h _i , h _j ), and λ represents a predetermined positive value. A parallax map generator for generating a V-parallax map and a U-parallax map using a depth map obtained from an image of a stereo camera;
A position and a size of a plurality of ROIs generated in the image are determined through the V-parallax map and the U-parallax map, and a height of the ROI is adjusted considering a representative height value of a predetermined ROI A region of interest generating unit;
An interest region rearrangement unit which integrates the overlapping regions of interest having mutually overlapping regions among the plurality of regions of interest into a single region of interest;
An object detection unit for determining whether the object of interest exists in each of the regions of interest; And
And a result output unit for outputting a detection result of the object of interest,
Wherein the interest region organizing unit comprises:
And calculating the degree of similarity between the overlapping ROIs and integrating the ROIs into one ROI only when the degree of similarity is determined to be equal to or greater than a threshold value. The method of claim 6,
Wherein the ROI generating unit includes:
A horizontal position, a width, a vertical position, and a height of the ROI in the image using the U-disparity map and the V-disparity map,
And adjusts a height of the ROI by using a height end point obtained by offsetting a representative height value of the ROI at a starting point of the ROI detected in the V-parallax map. The method of claim 6,
And expanding the size of each of the ROIs having the height adjusted to a predetermined magnification. delete The method of claim 6,
Wherein the interest region organizing unit comprises:
If the _{f s (h i, h j} ) of the equation below corresponding to the threshold degree of similarity than the object detection apparatus using stereo camera incorporating the overlapped region of interest in a region of interest:

Here, f _s (h _i , h _j ) is a value between 0 and 1, the closest to 1, the higher the degree of similarity, and the f _o (h _i , h _j ) the total area (R ∪R _{hi hj)} and the ratio of the overlapping area (R ∩R _{hi hj) contrast, f d (h i, h} j) are the first and second region of interest (h _i, h _j) (| d _hi -d _hj |) of the (h _i , h _j ), and λ represents a predetermined positive value.

Images