JP6406044B2 - Camera calibration unit, camera calibration method, and camera calibration program - Google Patents

Description

  The present invention relates to a technique for calibrating external parameters of each camera in a multi-camera system.

  2. Description of the Related Art Conventionally, as a multi-camera system using a plurality of cameras with different angles, one that generates a three-dimensional image (three-dimensional image) of a target area captured by a plurality of cameras (for example, see Patent Document 1) There is one that generates an all-round image (see, for example, Patent Document 2). In these techniques, a frame image captured by each camera is projected onto a common coordinate system to generate a three-dimensional image or an all-round image.

  In generating a stereoscopic image and an all-round image, a positional relationship between frame images captured by each camera is necessary. For this reason, as described in Patent Documents 1 and 2, etc., camera calibration (camera calibration) is performed, and external parameters of each camera are acquired in advance. The external parameters of the camera are parameters related to the position and orientation of the camera.

JP2013-93787A JP2011-86111A

  However, camera calibration has been performed by a method in which an object having a feature point is set in a target area, and a feature point of the object is extracted from a frame image obtained by capturing the target area. For example, when a chess board is used as an object having a feature point, a method in which a point where two adjacent black squares contact each other is extracted as a feature point.

  As described above, the conventional method for camera calibration has a problem that it is necessary to set an object having a feature point in a target region, which takes time and effort.

  In addition, the mounting state (position and posture) of the camera body changes depending on external factors such as external force. That is, the external parameters of the camera change depending on external factors. For this reason, the external parameters of the camera, unlike the internal parameters of the camera (parameters related to focal length, lens distortion, etc.), need to be updated regularly, and it is desired to reduce the time and labor required for camera calibration. ing.

  An object of the present invention is to provide a technique capable of reducing labor and time for camera calibration.

  In order to achieve the above object, the camera calibration unit of the present invention is configured as follows.

The first moving image input unit receives the first moving image captured by the first camera. The second moving image input unit receives a second moving image captured by the second camera at an angle different from that of the first camera. The imaging area of the first camera and the imaging area of the second camera partially overlap each other. The first feature point extraction unit detects a human face from frame images of a predetermined number of frames applied to the first moving image input to the first moving image input unit immediately after the start timing of the camera calibration process. A frame image captured from the front is selected as the first processing target frame image, and a person whose face is captured from the front is selected as the target person. The first feature point extraction unit extracts a plurality of feature points of the subject 's face captured in the selected first processing target frame image . In addition, the second feature point extraction unit selects, as a second processing target frame image, a frame image having an imaging timing that is temporally closest to the imaging timing of the first processing target frame image. The feature point extraction unit extracts a plurality of feature points of the face of the subject imaged in the selected second frame image.

The feature point association unit associates the feature point of the face of the subject first feature point extraction unit and extracted, and the feature points of the face of the second subject feature point extraction unit and extracted in the. External parameter calculation unit, a correspondence result of the feature points of the face of the subject by the feature point association unit, the positions of the feature points of the face of the subject on the first processing target frame image, the second processing target Using the position of the feature point of the subject 's face on the frame image, an external parameter of the camera related to the relative positional relationship between the first camera and the second camera is calculated. External parameter calculating unit, for example a first camera on the basis, calculates an extrinsic parameter of the second camera relative positional relationship against the first camera.

In this configuration, the first camera and the second camera are utilized by utilizing the characteristics of the face of the subject located in the imaging region (hereinafter referred to as the target region) overlapping in the first camera and the second camera. The camera external parameters relating to the relative positional relationship with the other camera are calculated. That is, camera calibration for calculating the external parameters of the camera related to the relative positional relationship between the first camera and the second camera can be performed without setting an object having a feature point in the target area. Therefore, it is possible to reduce the labor and time required for camera calibration.
Further, since the camera calibration can be performed using the first processing target frame image and the second processing target frame image captured at substantially the same timing, it is possible to suppress a decrease in the calculation accuracy of the external parameters of the camera.

In addition, the second feature point extraction unit detects the subject imaged in the second processing target frame image by face authentication using the face image of the subject imaged in the first processing target frame image. A configuration may be selected.

  If comprised in this way, it can prevent that a 1st feature point extraction part and a 2nd feature point extraction part extract a feature point from a different subject.

  According to the present invention, it is possible to reduce the labor and time required for camera calibration.

It is a figure which shows the structure of the principal part of the camera calibration unit concerning this example. It is a block diagram which shows the function structure of the image processor concerning a camera calibration process. It is a flowchart which shows a camera calibration process. It is a figure which shows a 1st process target frame image and a 2nd process target frame image. It is a figure which shows the matching result of the feature point of the extracted subject's face, and the feature point of the subject's face.

  Hereinafter, a camera configuration unit according to an embodiment of the present invention will be described.

  FIG. 1 is a diagram showing the configuration of the main part of the camera calibration unit according to this example. The camera calibration unit 1 includes an image processor 2, a first moving image input unit 3, a second moving image input unit 4, a storage unit 5, an output unit 6, and an input unit 7. Yes.

  The camera calibration unit 1 according to this example performs camera calibration (camera calibration) for calculating external parameters of the camera related to the relative positional relationship between the first camera 10 and the second camera 11, and A stereoscopic image (three-dimensional image) is generated using the external parameters of the camera calculated by calibration. The external parameters of the camera are the camera rotation R and translation T as is well known. In this example, camera calibration is performed to calculate the rotation R and translation T of the second camera 11 with the first camera 10 as a reference as external parameters of the camera.

  The image processor 2 includes a computer that executes the camera calibration method according to the present invention. The camera calibration program according to the present invention is installed in a computer provided in the image processor 2. The functional configuration of the image processor 2 will be described later. The computer included in the image processor generates a stereoscopic image using the frame image captured by the first camera 10 and the frame image captured by the second camera 11.

  A first camera 10 is connected to the first moving image input unit 3. A second camera 11 is connected to the second moving image input unit 4. The first camera 10 and the second camera 11 are video cameras that capture moving images. The frame rate of the first camera 10 and the second camera 11 is several tens of frames / sec (for example, 30 frames / sec). The first camera 10 and the second camera 11 are at different angles. The imaging area of the first camera 10 and the imaging area of the second camera 11 partially overlap. This overlapping imaging region is referred to as a target region here.

  A frame image related to a moving image captured by the first camera 10 is input to the first moving image input unit 3. A frame image related to a moving image captured by the second camera 11 is input to the second moving image input unit 4. The first moving image input unit 3 and the second moving image input unit 4 have a storage medium (not shown) such as an image memory or a hard disk for storing a frame image related to the input moving image.

  The storage unit 5 stores external parameters of the camera calculated by camera calibration processing described later.

  The output unit 6 outputs a three-dimensional image of the target area captured by the first camera 10 and the second camera 11. This stereoscopic image is generated by the image processor 2. The output unit 6 is connected to a stereoscopic image processing apparatus that processes a stereoscopic image. The stereoscopic image processing device is, for example, a display device that displays a stereoscopic image or an analysis device that analyzes the stereoscopic shape of an object located in a target region.

  An input device for inputting a command to the camera calibration unit 1 main body is connected to the input unit 7. The input device inputs a command related to an execution instruction of a camera calibration process, which will be described later, to the input unit 7.

  FIG. 2 is a block diagram showing a functional configuration of an image processor related to execution of camera calibration processing to be described later. The image processor 2 also has a function of generating a stereoscopic image as described above, but the functional configuration relating to the processing of generating the stereoscopic image is not shown. The image processing processor 2 includes a first processing target frame image selection function unit 21, a second processing target frame image selection function unit 22, a face authentication function unit 23, a first feature point extraction function unit 24, A second feature point extraction function unit 25, a feature point association function unit 26, and an external parameter calculation function unit 27 are provided.

  The first processing target frame image selection function unit 21 selects the first image from the frame images (the frame image input to the first moving image input unit 3) related to the moving image captured by the first camera 10. Select a frame image to be processed. The second processing target frame image selection function unit 22 selects the second image from the frame images (the frame images input to the second moving image input unit 4) related to the moving images captured by the second camera 11. Select a frame image to be processed. In particular, the second processing target frame image selection function unit 22 selects the first processing target frame image selection function unit 21 selected from the frame images related to the moving image captured by the second camera 11. A frame image having an imaging timing closer in time to the imaging timing of the processing target frame image is selected as the second processing target frame image. That is, the imaging timing of the first processing target frame image selected by the first processing target frame image selection function unit 21 and the second processing target frame image selected by the second processing target frame image selection function unit 22. The imaging timing is substantially the same.

  The first feature point extraction function unit 24 extracts a plurality of facial feature points for the subject imaged in the target region on the first processing target frame image. Further, a position on the first processing target frame image is detected for each extracted facial feature point.

  The face authentication function unit 23 performs face authentication on the target person from which the first feature point extraction function unit 24 has extracted the facial feature quantity from among the persons imaged in the target region on the second processing target frame image. To detect.

  The second feature point extraction function unit 25 extracts a plurality of face feature points for the subject detected by the face authentication function unit 23. Further, the position on the second processing target frame image is detected for each extracted facial feature point.

  The feature point association function unit 26 includes feature points of the subject's face extracted by the first feature point extraction function unit 24, and feature points of the subject's face extracted by the second feature point extraction function unit 25. , Are associated.

  The external parameter calculation function unit 27 includes the position of the feature point of the face of the subject associated with the feature point association function unit 26 on the first processing target frame image, and the feature point on the second processing target frame image. An external parameter related to the relative positional relationship between the first camera 10 and the second camera 11 is calculated using the position of the feature point of the subject's face associated with the association function unit 26. .

  Next, the camera calibration operation in the camera calibration unit 1 will be described. FIG. 3 is a flowchart showing camera calibration processing. The camera calibration unit 1 performs the camera calibration process shown in FIG. 3 when a command related to the execution instruction of the camera calibration process is input to the input unit 7.

  In this example, the camera calibration process is executed at the timing when the command related to the camera calibration process execution instruction is input to the input unit 7. However, the camera calibration process may be periodically executed. . For example, the camera calibration unit 1 may be configured to perform scheduling so that camera calibration processing is executed at midnight every Sunday or at midnight on the first day of every month.

  In the camera calibration unit 1, a frame image related to a moving image captured by the first camera 10 is input to the first moving image input unit 3, and a frame image related to a moving image captured by the second camera 11. Is input to the second moving image input unit 4.

  The image processing processor 2 uses a first image used for a camera calibration process to be described later from frame images (frame images input to the first moving image input unit 3) related to the moving images captured by the first camera 10. A processing target frame image is selected (s1). In s1, a frame image of a predetermined number of frames (for example, 10 frames) input immediately after the start timing of the camera calibration processing (timing at which a command related to the camera calibration processing execution instruction is input to the input unit 7). Then, the first processing target frame image is selected. For example, a frame image in which a person whose face is imaged from substantially the front in the target region is selected as the first processing target frame image. Also, this person (a person in the target area whose face is imaged from substantially the front) is selected as the target person. The processing related to s1 is performed by the first processing target frame image selection function unit 21 described above.

  The image processor 2 uses a second image used for camera calibration processing to be described later from frame images (frame images input to the second moving image input unit 4) of the moving images captured by the second camera 11. A processing target frame image is selected (s2). In s2, a frame image having an imaging timing that is closer in time to the imaging timing of the first processing target frame image selected in s1 is selected as the second processing target frame image. The processing related to s2 is performed by the second processing target frame image selection function unit 22 described above.

  Therefore, the imaging timing of the first processing target frame image selected in s1 and the imaging timing of the second processing target frame image selected in s2 are substantially the same.

  FIG. 4 is a diagram illustrating the first processing target frame image selected in s1 and the second processing target frame image selected in s2. In FIG. 4, illustrations of people other than the subject being imaged, the background, and the like are omitted. Depending on the angle difference between the first camera 10 and the second camera 11, the imaging position of the subject, the orientation of the subject, etc. in the first processing target frame image and the second processing target frame image are different. ing.

  The image processor 2 detects the target person selected in s1 by face authentication from the persons captured in the target area on the second processing target frame image (s3). The processing related to s3 is performed by the face authentication function unit 23 described above.

  The image processor 2 extracts a plurality of feature points of the face of the target person (target person selected in s1) captured in the target area on the first processing target frame image (s4). Further, the image processor 2 extracts a plurality of facial feature points of the target person (target person detected by face authentication in s3) imaged in the target area on the second processing target frame image (s5). The number of feature points of the subject's face extracted in s4 and the number of feature points of the subject's face extracted in s5 may or may not be the same. The processing relating to s4 is performed by the first feature point extraction function unit 24 described above. Further, the processing relating to s5 is performed by the second feature point extraction function unit 25 described above.

  The first feature point extraction unit referred to in the present invention includes a first processing target frame image selection function unit 21 and a first feature point extraction function unit 24. The second feature point extraction unit referred to in the present invention includes a second processing target frame image selection function unit 22, a face authentication function unit 23, and a second feature point extraction function unit 25.

  The image processor 2 associates the feature point of the subject's face extracted in s4 with the feature point of the subject's face extracted in s5 (s6). FIG. 5 is a diagram illustrating a correspondence result of the feature points of the subject's face extracted in s4 and s5 and the feature points of the subject's face in s6. The hatched small circles shown in FIG. 5 are the facial feature points of the subject extracted at s4 and s5. In s6, there may be a feature point that is not associated with the feature point extracted in s5 among the feature points of the subject's face extracted in s4, and conversely, Among the feature points, there may be a feature point that is not associated with the feature point extracted in s4. A straight line with double-ended arrows shown in FIG. 5 indicates that feature points are associated with each other. Also, feature points that are not marked with a straight line with double-ended arrows are feature points that could not be associated. The process related to s6 is performed by the above-described feature point association function unit 26.

  The image processor 2 determines whether or not the number of sets of feature points that can be associated in s6 is equal to or greater than a predetermined number (s7). In s7, the number of sets of feature points that can be associated in s6 is equal to or greater than the number of sets that can calculate the external parameters related to the relative positional relationship between the first camera 10 and the second camera 11. Judging whether or not. That is, in s7, it is determined whether or not an external parameter related to the relative positional relationship between the first camera 10 and the second camera 11 can be calculated.

  The camera calibration unit 1 determines the position (coordinate) on the first processing target frame image and the position (coordinate) on the second processing target frame image of the feature point associated with the external parameters of the camera in s6. Use to calculate. At this time, based on the first camera 10, the rotation R and translation T of the second camera 11 are calculated as external parameters of the camera. Various calculation methods are known for calculating the camera external parameters. For example, it is described in Patent Document 1 mentioned above. The camera calibration unit 1 may calculate the external parameters of the camera by any known calculation method.

  If the image processor 2 determines that the number of feature point pairs that can be associated in s6 is less than a predetermined number, the image processor 2 updates the start timing (s10), returns to s1, and repeats the above processing. In s10, the current time is set as the start timing.

  If the image processor 2 determines that the number of sets of feature points that can be associated in s6 is equal to or greater than a predetermined number, the relative position relationship between the first camera 10 and the second camera 11 is determined. Such external parameters are calculated (s8).

  The camera calibration unit 1 stores the external parameter calculated in s8 in the storage unit 5 (s9), and ends this process.

  Thus, the camera calibration unit 1 according to this example has a relative positional relationship between the first camera 10 and the second camera 11 without setting an object such as a chess board having a feature point in the target area. It is possible to calculate an external parameter of the camera according to the above. Therefore, it is possible to reduce the labor and time required for camera calibration.

  Further, since the target person imaged in the first process target frame image and the target person imaged in the second process control frame image are associated by face authentication, feature points are extracted from different target persons. Can be prevented. That is, processing after s5 is not performed wastefully. Further, in the first processing target frame image, a person whose face is imaged from substantially the front is selected as the target person, so that more feature points of the target person's face can be extracted.

  Moreover, since the imaging timings of the first processing target frame image and the second processing target frame image are substantially the same, a decrease in the accuracy of the external parameters of the camera to be calculated can be suppressed.

  In addition, the camera calibration unit 1 uses the external parameters stored in the storage unit 5 and applies the frame image related to the moving image captured by the first camera 10 and the moving image captured by the second camera 11. A stereoscopic image of the target area is generated using the frame image, and the generated stereoscopic image is output from the output unit 6.

  Further, the camera calibration unit 1 may be configured such that three cameras are connected and external parameters of each camera are calculated. In this case, by using one arbitrary camera as a reference, external parameters relating to the relative positional relationship with respect to the reference camera are calculated for the other two cameras.

  In the above example, the first processing target frame image and the second processing target frame image having substantially the same imaging timing used in camera calibration are set as one set, but the imaging timing is substantially the same. Camera calibration may be performed using a plurality of sets of one processing target frame image and second processing target frame images. If a plurality of sets of first processing target frame images and second processing target frame images having substantially the same imaging timing are used, the number of sets of associated feature points used for calculating external parameters in s8 can be increased. it can. Therefore, the accuracy of camera calibration can be improved.

  In addition, here, an example has been described in which a stereoscopic image of a target area is generated. However, even with a plurality of cameras that generate a full-circumference image described in Patent Document 2 described above, Parameters can be calculated.

  Further, the camera calibration unit 1 may be configured to be incorporated in any camera, or may be configured to be incorporated in an information processing apparatus such as a personal computer.

DESCRIPTION OF SYMBOLS 1 ... Camera calibration unit 2 ... Image processor 3 ... 1st moving image input part 4 ... 2nd moving image input part 5 ... Memory | storage part 6 ... Output part 7 ... Input part 10 ... 1st camera 11 ... 2nd Camera 21 ... first processing target frame image selection function unit 22 ... second processing target frame image selection function unit 23 ... face authentication function unit 24 ... first feature point extraction function unit 25 ... second feature point extraction Function unit 26: Feature point correspondence function unit 27: External parameter calculation function unit

Claims (5)

A first moving image input unit that inputs a first moving image captured by the first camera;
A second moving image input unit for inputting a second moving image captured by the second camera at an angle different from that of the first camera;
Selecting a first processing target frame image from among a predetermined number of frame images applied to the first moving image input to the first moving image input unit immediately after the start timing of the camera calibration process; A first feature point extraction unit that extracts a plurality of feature points of the face of the subject imaged in the first processing target frame image ;
Among the frame images related to the second moving image input to the second moving image input unit according to the imaging timing of the first processing target frame image selected by the first feature point extracting unit. A second feature point extraction unit that selects a second processing target frame image and extracts a plurality of feature points of the face of the subject captured in the second frame image;
And feature points of the face of the subject in which the first feature point extraction unit and extracted, and the feature point of the second of the subject's face feature point extraction unit and extracted, and the feature point association unit associating the ,
The result of associating the feature points of the subject 's face by the feature point association unit, the position of the feature points of the subject 's face on the first processing target frame image, and the second processing target frame An external parameter calculation unit that calculates external parameters of the camera related to the relative positional relationship between the first camera and the second camera using the position of the feature point of the face of the subject on the image; , equipped with a,
The first feature point extraction unit selects, from the frame images of the predetermined number of frames, a frame image in which a human face is captured from the front as the first processing target frame image, and the front The person whose face is imaged from is selected as the subject,
The second feature point extraction unit selects, as the second processing target frame image, a frame image having an imaging timing closest in time to the imaging timing of the first processing target frame image. Calibration unit. The second feature point extraction unit, said by the face authentication using the subject's face images imaged in the first processing target frame image, the imaged in the second processing target frame image The camera calibration unit according to claim 1, wherein the camera calibration unit detects a subject. The external parameter calculation unit, the first camera as a reference, calculates an extrinsic parameter of the relative positional relationship of the second camera relative to said first camera, according to claim 1 or 2, Camera calibration unit. The first moving image captured by the first camera is input to the first moving image input unit, and the second moving image captured by the second camera at an angle different from that of the first camera is the second moving image. Input to the image input section,
Selecting a first processing target frame image from among a predetermined number of frame images applied to the first moving image input to the first moving image input unit immediately after the start timing of the camera calibration process; A first feature point extracting step of extracting a plurality of feature points of the face of the subject imaged in the first processing target frame image ;
Among the frame images related to the second moving image input to the second moving image input unit in accordance with the imaging timing of the first processing target frame image selected in the first feature point extracting step. A second feature point extracting step of selecting a second processing target frame image from the image and extracting a plurality of feature points of the subject 's face imaged in the second frame image;
And feature points of a face of the first of the subject extracted by the feature point extraction step, a feature point of the said subject's face extracted by the second feature point extraction step, and with the feature point corresponding step of associating ,
And associating results of the feature points of the face of the subject by the feature point correspondence step, the positions of the feature points of the face of the subject on the first processing target frame image, the second processing target frame An external parameter calculating step of calculating an external parameter of the camera related to a relative positional relationship between the first camera and the second camera using the position of the feature point of the face of the subject on the image; A camera calibration method performed by a computer ,
In the first feature point extraction step, a frame image in which a human face is captured from the front is selected as the first processing target frame image from the predetermined number of frame images, and the front surface The person whose face is imaged from is selected as the subject,
The second feature point extracting step selects a frame image having an imaging timing closest in time to the imaging timing of the first processing target frame image as the second processing target frame image. Calibration method. The first moving image captured by the first camera is input to the first moving image input unit, and the second moving image captured by the second camera at an angle different from that of the first camera is the second moving image. Input to the image input section,
Selecting a first processing target frame image from among a predetermined number of frame images applied to the first moving image input to the first moving image input unit immediately after the start timing of the camera calibration process; A first feature point extracting step of extracting a plurality of feature points of the face of the subject imaged in the first processing target frame image ;
Among the frame images related to the second moving image input to the second moving image input unit in accordance with the imaging timing of the first processing target frame image selected in the first feature point extracting step. A second feature point extracting step of selecting a second processing target frame image from the image and extracting a plurality of feature points of the subject 's face imaged in the second frame image;
And feature points of a face of the first of the subject extracted by the feature point extraction step, a feature point of the said subject's face extracted by the second feature point extraction step, and with the feature point corresponding step of associating ,
And associating results of the feature points of the face of the subject by the feature point correspondence step, the positions of the feature points of the face of the subject on the first processing target frame image, the second processing target frame An external parameter calculating step of calculating an external parameter of the camera related to a relative positional relationship between the first camera and the second camera using the position of the feature point of the face of the subject on the image; A camera calibration program executed by a computer ,
In the first feature point extraction step, a frame image in which a human face is captured from the front is selected as the first processing target frame image from the predetermined number of frame images, and the front surface The person whose face is imaged from is selected as the subject,
The second feature point extracting step selects a frame image having an imaging timing closest in time to the imaging timing of the first processing target frame image as the second processing target frame image. Calibration program.

Images