JP2006236255A - Person tracking device and person tracking system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a person tracking device for tracking a person with high accuracy.
A person tracking apparatus according to the present invention includes a photographed image recording unit 22 that acquires and records a photographed image in which a photographing area and a photographing time are specified, and a person image that detects a person image from the photographed image and creates a person image. A detection unit 24, a feature amount extraction unit 28 that extracts a feature amount from a person image, and collating and determining the two feature amounts make it possible to identify the persons of the two person images from which the feature amounts have been extracted as the same person or another person. A person authentication unit 30, an authentication result determination unit 32 that determines whether the same authentication result of the person authentication unit is correct based on a shooting area and a shooting time corresponding to each person image of the same person, and an authentication result A photographing area corresponding to a person image of a person who is authenticated as the same in the same authentication result that the determination means is correct, and a flow line calculating means 38 for calculating the flow line of the person based on the photographing time. .
[Selection] Figure 1

Description

  The present invention relates to a person tracking apparatus that tracks a moving person using a plurality of cameras.

  When a person moving in a certain area is tracked, a plurality of cameras that respectively photograph different predetermined places in the area are used. The tracked person is photographed by a plurality of cameras, and the tracked person appears in the captured image of the camera. It can be seen from the captured image that the tracked person is shown that the tracked person has moved in the order of the time when the captured area of the captured image that the tracked person is reflected is reflected.

At this time, a person to be tracked (the same person) is specified among the persons shown in the captured images of the plurality of cameras. Whether or not the person shown in the image taken by a camera is the same as the person from another camera is determined by a predetermined determination method, for example, by extracting a feature amount indicating the feature of the person from the image and determining the difference between the feature amounts. The If the difference in feature amount is within a predetermined range, it is determined to be the same, otherwise it is determined to be another person. Thus, the identification of the same person among a plurality of persons shown in a plurality of photographed images (paradoxically speaking, identification with other persons) is called authentication. As an apparatus for authenticating the same person among the persons shown in the images taken by a plurality of cameras and tracking the same person, for example, there is one disclosed in Patent Document 1. In this method, a moving person is tracked to obtain information on the movement locus (hereinafter referred to as “flow line”) of the person. For example, it is carried out at a supermarket and the customer's flow line information is acquired in order to know the customer's purchasing tendency and movement pattern.
JP 2003-263641 A

  However, the tracking accuracy of the person and the correctness of the acquired flow line naturally depend on the authentication result. When another person is authenticated as a tracked person or when a tracked person is authenticated as another person, the person tracking device tracks a person other than the tracked person based on an incorrect authentication result and derives an incorrect flow line. Sometimes.

  Therefore, an object of the present invention is to provide a person tracking device that can accurately track a tracked person and derive an accurate flow line of the tracked person.

A person tracking apparatus according to the present invention is a person tracking apparatus that calculates a flow line by tracking the same person shown in a captured image obtained in a plurality of imaging areas,
Captured image recording means for acquiring and recording a captured image in which the capturing area and the capturing time are specified;
A person image detecting means for detecting a person image from the photographed image and creating a person image in which the photographing area of the photographed image and the information of the photographing time are associated;
Feature amount extraction means for extracting feature amounts from a person image;
A person authentication unit that performs authentication by comparing the two feature amounts and making a determination with a predetermined determination method so that each person of the two person images from which the feature amounts are extracted is the same or another person;
An authentication result judging means for judging whether or not the same authentication result authenticated by the person authenticating means is correct based on the information of the photographing area and the photographing time corresponding to each of the two person images of the person who is authenticated as the same; ,
A flow line calculating means for calculating a flow line of the person based on a shooting area and a shooting time corresponding to the person image of the same authenticated person in the same authentication result determined to be correct by the authentication result determining means; It is characterized by having.

  According to the present invention, the same authentication result determined to be correct by the authentication result determination means becomes more correct, and the flow line derived based on the same authentication result becomes more accurate.

  The person tracking device according to the present invention tracks a person moving in a certain area, for example, a building (tracked person) with a plurality of cameras each shooting a predetermined place (photographing area) in the area. It is.

  In the person tracking apparatus according to the present invention, “tracking” means specifying the location of the tracked person. The range where the location is specified is the shooting area of the camera. For example, the past location of the tracked person can be specified by a captured image showing the tracked persons of a plurality of cameras taken in the past. Further, when a tracked person enters a captured image of a camera (appears in a captured image of the camera), it is specified that the tracked person exists in the shooting area of the camera.

Embodiment 1 FIG.
A person tracking apparatus according to Embodiment 1 of the present invention is shown in FIG. The person tracking system 10 tracks a plurality of cameras 12-1 to 12 -M (M is an integer) that captures a predetermined area in a certain area and a person who moves in the area based on the captured image of the camera. It consists of a person tracking device 14.

  The person tracking device 14 includes a control unit 16 that controls the entire person tracking device 14, an output unit 18 such as a display on which captured images are displayed, and a user who uses the device 14 (for example, a person who monitors a tracked person). ) Has an input unit 20 such as a keyboard for receiving commands.

  The control unit 16 of the person tracking device 14 acquires a captured image of a plurality of cameras 12-1 to 12-M and records the captured image recording unit 22, and detects a person area (person image) from the captured images. A person image detection unit 24 that creates an image, an image processing unit 26 that performs image processing on the person image, a feature amount extraction unit 28 that extracts a feature amount from the person image, and two person images based on the feature amount A person authentication unit 30 that authenticates each person as the same or another person, and an authentication result determination unit 32 that determines whether the authentication result (hereinafter referred to as “the same authentication result”) that the person authentication unit 30 has authenticated is correct. And a tracked person information recording unit 34 for recording information (data) about the tracked person such as a person image and a feature amount.

  The control unit 16 also moves the person's flow line (movement) based on the shooting area and the shooting time corresponding to the person image of the person who is authenticated as the same in the same authentication result determined to be correct by the authentication result determination unit 32. A flow line calculation unit 36 for calculating the locus) and a flow line information recording unit 38 for recording the calculated flow line information.

  Hereinafter, each component of the person tracking system 10 will be described with reference to an embodiment. In one embodiment, the person tracking system 10 is provided in a building B having a passage made up of one entrance E and three passages T1 to T3 as shown in FIG.

  In order to track the tracked person, the plurality of cameras 12-1 to 12 -M are arranged such that a place where the tracked person can move (part of a route along which the tracked person can move) is an imaging area. Is done. In the embodiment shown in FIG. 2, the camera C1 images the entrance E of the passage, and the cameras C2 to C4 image the character paths T1 to T3. FIG. 2 shows captured images P1 to P4 of the cameras C1 to C4.

  In addition, the plurality of cameras 12-1 to 12-M are arranged so that the branching path, the entrance, and the exit serve as an imaging area in the path along which the tracked person can move.

  A camera having a branch road as an imaging area is arranged so that a branch destination is reflected in a captured image. For example, in the embodiment, the cameras C <b> 2 to C <b> 4 are arranged so that the shooting direction coincides with the passage where the cue roads join. For example, as shown in FIG. 3, in the captured image P2 of the camera C2, the passage R2 is out of frame (disappears from the captured image) from the left side LS2 of the captured image P2, the passage R3 is out of frame from the upper side US2, and the right side RS2 In the state in which the passage R1 is out of the frame, the choji T1 is shown.

  In a camera with an entrance or exit as a shooting area, the entrance camera is arranged so that the front of the person entering through the entrance is reflected in the photographed image, and the exit camera is a photographed image of the person leaving the exit through the exit. (In the embodiment, there is no camera for photographing the front of the person leaving the room). For example, as illustrated in FIG. 4, the entrance E is captured from inside the building B in the captured image P1 of the camera C1.

  The captured images of the plurality of cameras 12-1 to 12-M are recorded in the captured image recording unit 22 of the person tracking device 14 in association with the information of the capturing area and the capturing time.

  The control unit 16 of the person tracking device 14 has a function of tracking a person to be tracked (components necessary for tracking) and, for example, a subject image such as a photographed image or a person image of a plurality of cameras 12-1 to 12-M. Information on the tracker is displayed on the output unit 18 (display), and an instruction from the user via the input unit 20 (for example, an instruction to output information on the tracked person such as displaying a recorded person image) It plays the role of receiving.

  The output unit 18 is a display device such as a display, and is configured to display captured images and person images of a plurality of cameras 12-1 to 12 -M, and tracking information such as flow lines as will be described later. ing.

  The input unit 20 receives an instruction from a user such as a keyboard, and is configured to receive an instruction related to tracking such as an instruction to display an image on the output unit 18.

  The photographed image recording unit 22 records a photographed image in which the information of the photographing area and the photographing time is associated with each other.

  The person image detection unit 24 is configured to detect a person image from the captured image. Here, “detection” refers to specifying a person's region in a captured image. As a method of detecting a human image from an image, for example, there is a method of detecting a region that changes in time series in an image as an image of a moving object. If the image of the moving object is similar to various types of human-type images prepared in advance (for example, human-type images with different body shapes and postures), the image of the moving object is that of the moving person. For example, an image of the person H1 is detected from the captured image P2 of the camera C2 shown in FIG.

  The person image detected by the person image detection unit 24 may be an entire image of a person or a ghost image of a person. As a method for detecting a wrinkle image (a wrinkle region) from an image, for example, as described above, a human image is detected from the image, and a head portion in the human image is detected from the contour or structure of the human body. Further, in the detected head image, the eye part and the mouth part are specified from the color shading, and the eyelid part (the eyelid image) is detected.

  The human image detection unit 24 preferably detects both the whole image and the heel image. The person image detected by the person image detection unit 24 is used for authentication performed by the person authentication unit 30 as described later. For example, in the case where authentication is performed using only the whole image, there is a possibility that two other people wearing similar clothes may be authenticated as the same. In some cases, a ghost image does not appear in the captured image due to the positional relationship between the camera and the person. It is preferable that the person image detection unit 24 detects the entire image and the samurai image so that authentication can be performed using both the entire image and the samurai image.

  The person image detection unit 24 records the detected person image in the tracked person information recording unit 34 as a person image in which the shooting area of the shooting image of the detection source is associated with the shooting time information. For example, in the embodiment shown in FIG. 3, a person image of the person H1 is recorded in association with the shooting area T1 and the shooting time t1.

  The image processing unit 26 performs image processing on the person image. A person shown in a photographed image is not always photographed under the same photographing conditions, for example, the photographing direction (angle), the photographing distance, and the brightness. Accordingly, the feature amounts described later extracted from a plurality of person images of the same person are naturally different. For example, if there are two person images of the same person and the feature amounts are significantly different, the person of the two person images may be authenticated as another person. In order to prevent the person of the person image from being authenticated as another person despite the person image of the same person due to the difference in the shooting conditions, the image processing unit 26 makes the shooting conditions substantially equal to the person image. Perform image processing.

  Specifically, the image processing performed by the image processing unit 26 includes simple processing such as processing for aligning the size of a person image to a predetermined size, processing for aligning the brightness and contrast of a human image to a predetermined value, for example. and so on. Further, as advanced processing, there is an angle correction of a person image, for example, a process of correcting a person image photographed from the front left to a person image photographed from the front. The angle of the person with respect to the camera can be obtained from the inclination of the eye line (line connecting two eyes) in the captured image (for example, when the image is taken from the front, the eye line substantially matches the horizontal direction of the image). To do.)

  The feature amount extraction unit 28 extracts a feature amount from the image-processed person image. The feature amount is a unique value of the person shown in the captured image of the camera and is used in authentication. For example, the feature amount is obtained by quantifying the shape (body shape), color scheme (clothing color scheme), and the like of the whole image by a predefined calculation method when the human image is the whole image. In the case of an image, the arrangement pattern of eyes, nose and mouth (intervals between eyes, etc.), the outline of a face (shape of jaw), etc. are digitized. As will be described later, the feature quantity extraction unit 28 uses a plurality of types of feature quantities from one person image in order to improve the accuracy of authentication performed based on the feature quantities (so that a plurality of feature quantities can be used in authentication). To extract.

  The feature quantity extraction unit 28 records the extracted feature quantity in the tracked person information recording unit 34 in association with the information of the person image as the extraction source.

  The person authentication unit 30 performs authentication based on the feature amount extracted by the feature amount extraction unit 28. In authentication performed on a person of one arbitrary person image, a feature amount extracted from the arbitrary person image is compared with a feature amount extracted from another person image, and a predetermined determination method, for example, the feature When the amount difference is within a predetermined range, the person of the arbitrary person image is authenticated as the same as the person of the other person image, while when it is out of the range, the person is authenticated with another person. The predetermined range is clearly a range for determining that the person is another person (for example, an adult and a child). In other words, if the feature amount difference is within the range, strictly speaking, the same person is possible. It means that the nature is high. Therefore, in authentication performed on a person of one arbitrary person image, there are a plurality of other person images in which the difference in feature amount is within a predetermined range (in other words, a plurality of the same authentication results are derived). ).

  For example, when the person appears in any of the shooting areas of the plurality of cameras 12-1 to 12-M (when the person has entered a frame in any of the shot images), The feature amount of the person image of the person (hereinafter referred to as “authentication target image”) is collated. For example, as shown in FIG. 3, the feature amount of the person image of the person H1 that appears in the captured image P2 of the camera C2 is collated.

  First, the person image detection unit 24 detects an image to be an authentication target image, and then the feature amount extraction unit 28 extracts a feature amount from the authentication target image. The person authentication unit 28 compares the feature amount of the authentication target image with the feature amounts of a plurality of person images (hereinafter referred to as “authentication destination images”) recorded in the tracked person information recording unit 34.

  As a result of the collation, for example, when an authentication destination image having a feature amount whose difference from the feature amount of the authentication target image is within a predetermined range is recorded in the tracked person information recording unit 34, the person authentication unit 30 The person of the authentication target image is authenticated as being the same as the person of the authentication destination image. For example, the person H1 shown in the photographed image P2 shown in FIG. 3 is authenticated as the same as the person H2 shown in the photographed image P1 shown in FIG.

  The person authentication unit 30 preferably performs determination based on a plurality of types of feature values, for example, the feature values of the body shape and color arrangement pattern extracted from the whole image, the arrangement pattern of the eyes and nose extracted from the eyelid image, and the feature value of the contour. . As a matter of course, the authentication result for each of the plurality of feature amounts is correct as compared with the result of the determination for one feature amount.

For confirmation, the person authentication unit 30 does not specify the same person and perform the same authentication. For example, each person image ha, hb, hg is authenticated as person 1, each person image hc, hf is authenticated as person 2, each person image hd, he is each person 3, and so on. Authenticate to identify someone about a person.

  The authentication result determination unit 32 determines whether or not the same authentication result authenticated by the person authentication unit 30 is correct. The same authentication result of the person image authentication unit 30 indicates that the person of the authentication target image is the same as, for example, the person of the authentication destination image having a feature amount whose difference from the feature amount is within a predetermined range. Thereby, the person existing in the shooting area of the authentication target image and the person existing in the shooting area of the authentication destination image are authenticated as the same. However, since the authentication performed by the person authentication unit 30 is strictly a person image authentication, it is difficult to say that the person has been reliably authenticated, and the authentication result is not guaranteed (therefore, the conventional image authentication). In some cases, there was an error in the authentication result.) The authentication result determination unit 32 functions to determine whether the same authentication result of the person authentication unit 30 is correct and to prove the correctness of the same authentication result.

  The authentication result determination unit 32 refers to information on shooting areas and shooting times corresponding to (corresponding to) two person images of a person who has been authenticated as the same, and between each shooting time in each shooting area. The person authentication unit 30 determines whether or not the same authentication result authenticated by the person authentication unit 30 is correct based on a temporal restriction existing in the image and a spatial restriction existing between the shooting areas. The authentication result determination unit 32 determines that the authentication result is correct, and persons having the same authentication result are determined to be the same person.

  Specifically, in two shooting areas Ax and Ay in a plurality of shooting areas, the person authentication unit 30 takes a person Hx that appears in a shot image of the shooting area Ax at time tx and shots of the shooting area Ay at time ty. It is determined whether or not the result obtained by authenticating the same person as the person Hy shown in the image is correct based on the spatial constraint between the two shooting areas Ax and Ay and the temporal constraint between the times tx and ty. In the present specification, the symbols with the subscripts x and y indicate that they are arbitrary.

  As an example of determination based on time constraints, the authentication result determination unit 32, at the same time (tx = ty), in the example shown in FIG. 2, the person shown in the captured images P2 and P3 in the shooting areas T1 and T2 The same authentication result of the person authentication unit 30 authenticated as the same is determined as an error. It is impossible for the same person to appear in different shooting areas at the same time.

  As an example of determination based on spatial constraints, the authentication result determination unit 32, when there are two shooting areas and the movement between them is impossible, the same authentication result that makes the person of the shot images in the two shooting areas the same Is determined to be an error. It is impossible for the same person to appear in two immovable shooting areas.

  As an example of determination based on spatial constraints, the authentication result determination unit 32 refers to a captured image corresponding to each of the two person images of the person who is authenticated as the same by the person authentication unit 30, and sets one of the captured images. It is determined whether or not the person's frame-in position / frame-out position shown in the image corresponds to the person's frame-out position / frame-in position shown in the other photographed image. Judge the result as correct. The frame-in position is a position where a person appears first in the captured image, and the frame-out position is a position where a person appears last.

  More specifically, the authentication result determination unit 32 authenticates the person Hx of the photographed image in the photographing area Ax at time tx to be the same as the person Hy in the photographed image in the photographing area Ay at time ty. When the result corresponds to the frame-in position of the person Hx and the frame-out position of the person Hy, or the frame-out position of the person Hx corresponds to the frame-in position of the person Hy, the same authentication result of the person image authentication unit 30 Is determined to be correct.

  An example will be described. FIG. 5 shows captured images P1 to P4 at times t3 and t4 of the cameras C1 to C4 shown in FIG. Persons H3 to H6 having substantially the same feature quantity (the person authentication unit 30 can authenticate the same) are shown in the captured image. It is assumed that the person H3 shown in the captured image P3 at time t3 is tracked.

  It is assumed that the person H3 shown in the photographed image P3 at time t3 disappears from the vicinity of the upper side US3 of the photographed image P3 (assuming that the person H3 is out of the frame from the upper side US). The upper side US3 is the frame-out position. Subsequently, at time t4 after a lapse of time, the person H4 appears in the captured image P2 of the camera C2 (the person H4 enters the frame), and the persons H5 and H6 appear in the captured image P4 of the camera C4 almost simultaneously. It is assumed that the person image authenticating unit 30 authenticates any one of the persons H4 to H6 as H3.

  On the other hand, the authentication result determination unit 32 authenticates the person H5 appearing from the frame-in position corresponding to the frame-out position US3 of the person H3, that is, the person H5 appearing from the frame-in position US4 of the captured image P4 as the same as the person H3. Is determined to be correct. On the other hand, if the person authentication unit 30 authenticates the persons H4 and H6 as the same as the person H3, the authentication result is determined to be an error.

As shown in FIG. 2, the person H3 goes out of the frame from the upper side US3 of the captured image P3 when heading for the path R4. When the person H3 moves along the path R4, the next shooting area appears at T3. Therefore, it can be seen that the person H4 appearing in the photographed image P2 in the photographing area T1 is another person.

  In addition, the person H6 appears in the captured video P4, but enters the frame from the right side RS4 that does not correspond to the frame-out position US3. In order to frame in from the right side RS4, as shown in FIG. 2, the path R5 must be moved. Therefore, it can be seen that the person H3 and the person H6 who have moved in the passage R4 are other persons.

  As in the above-described example, the authentication result determination unit 32 uses the same authentication result information determined to be correct based on the shooting area and shooting time information corresponding to each of the two person images of the same authenticated person. It is associated with the person image to be authenticated and recorded in the tracked person information storage unit 34.

  The person authentication unit 30 removes the authentication destination image of the person who has been authenticated as the same person as the authentication target image in the authentication of the same authentication result determined as an error by the authentication result determination unit 32 (the same authentication determined as an error). It is configured to re-authenticate (so as not to derive the result again). In other words, the person authentication unit corrects the same authentication result for all the person images.

A routine in which the person authentication unit 30 performs re-authentication so that the same authentication result determined to be incorrect is not derived again, and the authentication result determination unit makes a correct / incorrect determination for all the same authentication results corrected by re-authentication. By repeating, the same authentication result with respect to the person of all the person images becomes correct. In other words, the person authentication unit and the authentication result determination unit function so that the same authentication result for all persons in the person image is correct.

  The tracked person information recording unit 34 includes a person image associated with the photographing area recorded in the person image detecting unit 24 and a photographing time, and a feature associated with the person image recorded in the feature amount extracting unit 28. The amount and the authentication result information associated with the person image recorded in the authentication result determination unit 32 are recorded. A table showing these correspondences is shown in FIG. As shown in FIG. 6, the shooting area, shooting time, feature amount, and authentication result are associated with a person image. For example, the person images h1, h4, and h99 in different shooting areas are shown to be images of the same person 1.

  The flow line calculation unit 36 refers to the tracked person information recording unit 34 and calculates the flow line of the same person. According to the example shown in FIG. 6, the flow lines in the order of E, T1, and T3 of the person 1 are calculated. The calculated flow line is recorded in the flow line information recording unit 38.

  In the above, the component of the person tracking device which concerns on one Embodiment of this invention was demonstrated. Next, an example of an operation flow related to the same person authentication of the person tracking device shown in FIG. 7 will be described. Here, an operation flow related to authentication in which the person Hm starts to appear in the captured image Pm at time tm will be described.

  In step 100, the person image detection unit 24 detects an image of the person Hm shown in the captured image Pm at time tm.

  In step 110, an image of the person Hm is recorded in the tracked person information recording unit 34 as a person image hm.

  In step 120, the image processing unit 26 performs image processing on the person image hm.

  In step 130, the feature amount extraction unit 28 extracts feature amounts am to zm from the person image hm.

  In step 140, the person authentication unit 30 collates the feature amounts am to zm between the person image and the person image hm recorded in the tracked person information recording unit 34.

  In step 150, it is determined whether or not a person image having a feature amount difference within a predetermined range is recorded (exists) as a result of the collation in step 140. If not, the process proceeds to step 160. If it exists, go to Step 170.

  In step 160, the person authentication unit 30 authenticates the person Hm shown in the captured image Pm at time tm as a new person.

  In step 170, as a result of the collation in step 140, it is determined whether or not there are a plurality of person images whose feature amount difference is within a predetermined range. If there are more than one, proceed to Step 180. If not, go to step 230.

  In step 180, the person authentication unit 30 authenticates the person Hm shown in the captured image Pm at time tm as the same person as the person image having the smallest feature amount difference among the plurality of person images.

  In step 190, the authentication result determination unit 32 determines whether or not the same authentication result in step 180 is correct.

  In step 200, if the authentication result is correct in the determination result in step 190, the process proceeds to step 210. If there is an identical authentication result determined to be an error, the process proceeds to step 220.

  In step 210, the authentication result determination unit 32 ensures that the same authentication result in step 180 is correct.

  In step 220, the person authentication unit 30 deletes the person image of the person who is authenticated as the same as the person Hm shown in the captured image Pm at the time tm in the same authentication result determined as an error. Return to step 170.

  In step 230, the person authentication unit 30 authenticates the person Hm shown in the captured image Pm at time tm as being the same as the person in the person image whose feature value difference is within a predetermined range.

  In step 240, as in step 190, the authentication result determination unit 32 determines whether the authentication result in step 230 is correct.

  In step 250, if the authentication result is correct in the determination result in step 240, the process proceeds to step 260. If there is an error, go to step 270.

In step 260, the authentication result determination unit 32 ensures that the authentication result in step 240 is correct.

  In step 270, the person authentication unit 30 authenticates the person Hm shown in the captured image Pm at time tm as a new person.

  According to the above-described operation flow relating to authentication, the person Hm shown in the captured image Pm at time tm is determined to be the same as the person transferred to any other captured image, or determined as a new person. As can be seen from this flow, the person authentication unit 30 authenticates that two person images having the smallest feature amount difference are likely to be the same, in other words, the person image only has a high possibility of being the same. The authentication result determination unit 32 may make an error in the authentication result. As a result, the erroneous authentication that has occurred by authenticating the person who is most likely to be the same in the conventional person authentication using only the person image is suppressed, and the tracked person can be tracked accurately and the tracked person can be tracked. The exact flow line can be derived.

Embodiment 2. FIG.
The person tracking apparatus according to the present embodiment is an improved form of the first embodiment. The difference is the authentication result determination unit. Only this will be described.

  The authentication result determination unit of the present embodiment refers to the information of the shooting areas Au and Aw and the shooting times tu and tw corresponding to the two person images Pu and Pw of the person who are authenticated as the same in the same authentication result, Based on the temporal constraints that exist between each shooting time tu, tw in each shooting area Au, Aw and the spatial constraints that exist between each shooting area Au, Aw, the person is at one shooting time tu. A contradiction amount (authentication contradiction amount) indicating the temporal or / and spatial uncertainty of being present in the corresponding one shooting area Au and existing in the other shooting area Aw corresponding to the other shooting time tw is calculated. Then, it is determined that the same authentication result with a small amount of authentication contradiction is correct. In addition, in this specification, the code | symbol attached with the subscript of u and w shows that it is arbitrary.

  Specifically, in the authentication of the person authentication unit, the person Hu existing in the shooting area Au (person shown in the person image Pu) at the time tu is authenticated as the same person Hw existing in the shooting area Aw at the time tw. And At this time, the authentication result determination unit calculates an authentication contradiction amount indicating temporal or / and spatial uncertainties for the person Hu existing in the imaging area Aw at time tw. The amount of authentication contradiction indicates the uncertainty of the same authentication result, in other words, the probability of being incorrect. The authentication result determination unit determines that the same authentication result with a small amount of authentication contradiction, in other words, the most probable same authentication result is correct. Hereinafter, the temporal uncertainty is referred to as a temporal contradiction amount, and the spatial uncertainty is referred to as a spatial contradiction amount.

  In the first embodiment, the same authentication result is determined to be correct, but this can also be regarded as an authentication contradiction amount. This can be dealt with by making positive the smallest amount of authentication contradiction and wrong making the largest amount of authentication contradiction.

  An example of the authentication contradiction amount will be described. FIG. 8 shows captured images P1 to P4 at times t5 and t6 of the cameras C1 to C4 shown in FIG. Persons H7 and H8 having substantially the same feature amount (person authentication unit 30 can authenticate the same person) are shown in the captured image. It is assumed that the person H7 shown in the captured image P1 at time t5 is tracked.

  It is assumed that the person H7 shown in the photographed image P1 disappears from the photographed image P1 at time t5 (time t5 is defined as a frame-out time). Subsequently, it is assumed that a person H8 appears in the captured image P2 of the camera C2 at time t6 (future time relative to t5 and frame-in time), and the person authentication unit authenticates the same.

  On the other hand, the authentication result determination unit determines the amount of authentication contradiction based on the spatial constraint between the shooting areas E and T1 of the cameras C1 and C2 in which the persons H7 and H8 are shown and the temporal constraint between the times t5 and t6. Calculate

For example, it is assumed that the distance (passage R1) between the imaging areas E and T1 is 10 m. When the difference between the frame-out time t5 and the frame-in time t6 is less than 1 second, it is impossible for the person to move during that time. Therefore, the authentication contradiction amount for the same authentication result in this case is set to 100. Here, the maximum amount of authentication contradiction is 100.

  If the difference between the frame-out time t5 and the frame-in time t6 is 1 to 2 seconds, it is possible to run, so the authentication contradiction amount for the same authentication result in this case is 70. In addition, when the difference between t5 and t6 is 2 to 10 seconds, it corresponds to the normal human walking speed, so the amount of authentication contradiction for the same authentication result in this case is 10. Furthermore, if it is 10 seconds or longer, there is a possibility of walking at a slow speed, and the authentication contradiction amount for the same authentication result in this case is set to 30. In this way, an authentication contradiction amount is defined in advance between predetermined imaging areas.

  When the person H7 disappears from the photographed image P1 at time t5 and a plurality of persons appear in the photographed image P2 of the camera C2, the person with the smallest amount of authentication contradiction (for example, the person who appeared without passing one second from t5) If there is a person who appears 1.5 seconds after t5 and a person who appears 12 seconds after t5, among these, the person authentication unit that makes the person who appears 12 seconds after t5 with a small authentication contradiction amount the same It is determined that the same authentication result is correct.

  The amount of authentication contradiction in the above example may be defined in the time range as described above, or the time is calculated from the standard walking speed of the person and the distance between the shooting areas, and depending on the deviation from this calculated time The amount of authentication contradiction may be obtained. Further, the walking speed may be determined by the body feature amount. For example, when the person can be identified as a dwarf from the feature quantity of the body shape, the authentication contradiction amount may be obtained from the walking speed of the dwarf.

The authentication result determination unit determines that the same authentication result having a large authentication contradiction amount is not correct.

  The person authentication unit removes the authentication destination image of the person authenticated as the same as the person of the authentication target image corresponding to the same authentication result determined to be incorrect by the authentication result determination unit (the same authentication determined to be incorrect). It is configured to re-authenticate (so as not to derive the result again). In other words, the person authentication unit corrects the same authentication result for all the person images.

When the person authentication unit re-authenticates so that the same authentication result determined to be incorrect is not derived again, and the authentication result determination part obtains a large amount of authentication contradiction for all the same authentication results corrected by re-authentication By repeating the routine of determining that the image is correct, the amount of authentication contradiction in the same authentication result for all persons in the person image becomes small. In other words, the person authentication unit and the authentication result determination unit function so that the total amount of authentication contradiction of the same authentication result for all the person images is minimized.

  Thus, the authentication result determination unit can determine that the most probable identical authentication result is correct by using the authentication contradiction amount. Therefore, the person tracking apparatus according to the present embodiment can track the tracked person with high accuracy and derive the accurate flow line of the tracked person.

  In the above description, the authentication contradiction amount is indicated by the spatial contradiction amount and the temporal contradiction amount. Instead, the sum of the same authentication result score indicating the accuracy of the same authentication in the person authentication unit (for example, small feature quantity difference), the temporal contradiction amount, and the value obtained by weighting the spatial contradiction amount with a constant coefficient. The amount of authentication contradiction may be configured.

Embodiment 3 FIG.
In the person tracking device of the above-described embodiment, the flow line of the person corresponding to the same authentication result is formed based on the same authentication result that is correct by the authentication result determination unit. In the present embodiment, a flow line is created based on the same authentication result authenticated by the person authentication unit, it is determined whether the flow line is correct, and the flow line determined to be correct corresponds to the person corresponding to the same authentication result. The flow line of

  FIG. 9 shows a person tracking apparatus according to the present embodiment. In the person tracking device 110 shown in the figure, the same constituent elements as those of the first embodiment are given the reference numerals of 100 plus the reference numerals of the constituent elements of the first embodiment. Components different from those of the first embodiment are a person authentication unit 140, a flow line calculation unit 142, and a flow line determination unit 144. These will be mainly described.

  The person authentication unit 140 performs authentication based on the feature amount extracted by the feature amount extraction unit 128, similarly to the person authentication unit 30 of the first embodiment. In authentication performed on a person of one arbitrary person image, a feature amount extracted from the arbitrary person image is compared with a feature amount extracted from another person image, and a predetermined determination method, for example, the feature When the amount difference is within a predetermined range, the person of the arbitrary person image is authenticated as the same as the person of the other person image, while when it is out of the range, the person is authenticated with another person.

  The flow line calculation unit 142 calculates the flow line of the person based on the shooting area and the shooting time corresponding to the person image of the person who is authenticated as the same. Specifically, a flow line that moves the shooting area in the order of the shooting time is calculated from the shooting area and the shooting time corresponding to a plurality of person images of the person who is authenticated as the same.

  The flow line determination unit 144 refers to the shooting area corresponding to the person image of the person who is authenticated as the same, the shooting time, and the positional relationship between the plurality of shooting areas, and the time existing between the shooting times in each shooting area. It is determined whether or not the flow line is correct based on the physical constraints and the spatial constraints existing between the imaging areas. Since the same authentication result of the person authentication unit 140 is an authentication result of only the person image, the flow line calculation unit 142 exists between the shooting areas of the two person images of the person who is authenticated as the same in the same authentication result. The flow line is created by ignoring the spatial constraints and / or the temporal constraints existing between the photographing times. Therefore, in the calculated flow line, there may be a case where a spatial restriction is included between the photographing areas and a time restriction is included between the photographing times in the photographing area. In that case, of course, the flow line is inconvenient. The flow line determination unit 144 detects whether there is a problem with the flow line, and determines whether the flow line is correct.

  As an inconvenient example of the flow line, for example, there are cases where two imaging areas Ab and Ac that are consecutive in the flow line of Aa, Ab, Ac, and Ad in that order cannot actually move. In this case, the person authentication unit 140 erroneously authenticates the person existing in one of the two shooting areas Ab and Ac as the same regardless of the other person.

  As another inconvenient example of the flow line, there is a problem in the continuity of the shooting area in the flow line. For example, it is assumed that the flow lines of the imaging areas E, T2, T3, T1, and E are calculated in order from the building B as shown in FIG. In this case, T1 needs to exist between E and T2 of the flow line from the positional relationship of the plurality of shooting areas in the building B. In this case, at the time between the shooting time corresponding to the shooting area E and the shooting time corresponding to the shooting area T2, the person authentication unit 140 erroneously authenticates the other person even though the person existing in the shooting area T1 is the same. Will be.

  The flow line determination unit 144 determines that the flow line is correct when there is no inconvenience as in the above-described example. The flow line determined to be correct is stored in the flow line information recording unit 138.

  The flow line determination unit 144 determines that the flow line is an error when there is a problem such as the above-described example in the flow line.

  The person authentication unit 140 captures a shooting area (shooting area Ab and Ac or T1 in the above example) or shooting time (shooting time and shooting corresponding to the shooting area E in the above example) of the inconvenient part of the flow line determined to be an error. Re-authentication is performed so as not to derive again the same authentication result regarding the shooting time corresponding to the area T2. In other words, the person authentication unit 140 corrects the same authentication result for all the person images.

The person authentication unit 140 re-authenticates, the flow line calculation unit 142 calculates a new flow line based on all the same authentication results corrected by the re-authentication, and the flow line determination unit 144 determines whether or not the new flow line is correct. By repeating the routine of determining whether or not, the flow lines for all the person images are correct. In other words, the person authentication unit 140, the flow line calculation unit 142, and the authentication result determination unit 144 function so that the flow lines for all persons in the person image are correct.

Thereby, the error of the flow line which generate | occur | produced by authenticating the person with the highest possibility of the same in the person authentication only of the conventional person image as the same is suppressed.

Embodiment 4 FIG.
The person tracking apparatus according to the present embodiment is an improved form of the third embodiment. The difference is the flow line determination unit. Only this will be described.

  The mobile flow line determination unit of the present embodiment refers to the shooting area corresponding to the person image of the person who is authenticated as the same, the shooting time, and the positional relationship between the plurality of shooting areas, and between each shooting time in each shooting area. The temporal or / and spatial uncertainty of the movement of the person corresponding to the flow line in accordance with the calculated flow line based on the temporal restriction existing in the camera and the spatial restriction existing between the shooting areas. A flow line contradiction amount is calculated, and a flow line with a small flow line contradiction amount is determined to be correct. Hereinafter, the temporal uncertainty is referred to as a temporal contradiction amount, and the spatial uncertainty is referred to as a spatial contradiction amount.

  As an example, the flow line determination unit calculates a large flow line contradiction amount when the time between shooting times of a certain continuous shooting area in the calculated flow line is not suitable for the moving speed of the person, and when it is suitable Calculates a small amount of traffic line contradiction. Further, as an example, the flow line contradiction amount may be calculated corresponding to the time between the shooting time for the first shooting area and the shooting time for the last shooting area in the flow line.

  As another example, the flow line determination unit may calculate a flow line contradiction amount from a deviation amount with respect to a predefined flow line. For example, an exhibition route is set in a painting exhibition, and the flow line of a person is determined to some extent. The flow line determination unit calculates a flow line contradiction amount with respect to the flow line calculated by the flow line calculation unit according to the deviation amount from the flow line that can be defined in advance like the exhibition route.

  The flow line determination unit determines that a flow line having a small flow line contradiction amount is correct. The flow line determined to be correct is stored in the flow line information recording unit.

  The flow line determination unit determines that a flow line having a large flow line contradiction amount is an error.

  The person authentication unit determines whether there is an inconsistent portion of the flow line determined to be erroneous (for example, in the above example, the two shooting areas having a short time between corresponding shooting times, or the most deviated from the predefined flow line. It is configured to perform re-authentication so as not to derive again the same authentication result regarding a large amount portion or the like). In other words, the person authentication unit corrects the same authentication result for all the person images.

The person authentication unit re-authenticates, and based on all the same authentication results corrected by re-authentication, the flow line calculation unit calculates a new flow line, and the flow line determination unit determines whether the new flow line is correct. By repeating this routine, the flow line contradiction amount of the flow lines with respect to the persons of all the person images becomes small. In other words, the person authentication unit, the flow line calculation unit, and the authentication result determination unit function so that the total of the flow line contradiction amounts of the flow lines for all persons in the person image is minimized.

  As described above, the flow line determination unit can determine that the most likely flow line is correct by using the flow line contradiction amount. Therefore, the person tracking apparatus according to the present embodiment can track the tracked person with high accuracy and derive the accurate flow line of the tracked person.

  In the above description, the flow line contradiction amount is indicated by the spatial contradiction amount and the temporal contradiction amount. Instead, the sum of the same authentication result score indicating the accuracy of the same authentication in the person authentication unit (for example, small feature quantity difference), the temporal contradiction amount, and the value obtained by weighting the spatial contradiction amount with a constant coefficient. The flow line contradiction amount may be constituted by.

  The person tracking apparatus (system) according to the present invention has been described with reference to the four embodiments, but the present invention is not limited to the above-described embodiments. There are various modifications (improved examples).

  As an example, the shooting area of the shooting camera is not limited to a branch path, an entrance, and an exit, and may be provided in a path connecting the branch path and the branch path as shown in FIG.

  As another example, in the person tracking device according to the present invention, real-time tracking (tracking that identifies a person every time a person moves to a captured image) is also tracked in the past (tracking a person who moves to a past captured image). Obviously, it is possible. In the case of tracking in the past, tracking a person appearing in a photographed image from a plurality of photographed images that have been photographed by other devices (for example, a surveillance system using a plurality of surveillance cameras) and whose photographing time is specified. Is possible. In this case, the person tracking device may be provided with a photographed image input unit for receiving a photographed image from the outside and inputting the photographed image to the photographed image recording unit.

  In addition, the image processing may be performed on a person image to be authenticated that has been authenticated as a new person in the authentication by the person authentication unit without being performed on all the person images created by the person image detection unit ( In other words, this is performed when there is no person image of the same person as the person of the person image to be authenticated.) This is because there is a possibility that a person image of a person who is authenticated to be the same as the person of the person image to be authenticated exists in the tracked person information recording unit even if it is a person image to be authenticated before image processing. It is.

  In this case, the image processing unit performs image processing on the person image to be authenticated that has been authenticated as another person in the authentication by the person authentication unit, and the person authentication unit authenticates again the person image to be authenticated that has undergone image processing. Do. As a result, it is not necessary to perform image processing on all the human images created by the human image detection unit, and the time required for authentication is reduced. Further, in this case, when a plurality of processes are performed as image processing, the plurality of processes are not performed at one time, but one process is performed on the person image to be authenticated and one process is performed. When the person of the person image is authenticated as a new person by the person authentication unit, image processing may be performed such that the following processing is performed.

  Further, not only the person image recorded in the tracked person information recording unit but also other person images may be compared. For example, a person image prepared in advance (for example, a person image requiring attention such as a wanted photo) and a person image of a person shown in a captured image of the entrance camera (for example, a captured image P1 of the camera C1 shown in FIG. 2). Collate. The person who needs to be watched can be traced by tracking the person shown in the image taken by the entrance camera that has been authenticated as the person who needs the attention. In this case, a person image input device (for example, a scanner) for inputting a person image requiring attention to the person tracking device and a person information storage unit requiring attention are provided, and the person authentication unit includes the person image requiring attention in the person information storage unit requiring attention. What is necessary is just to comprise so that collation may be performed.

  Furthermore, the suspicious person who performs abnormal behavior is identified using the flow line calculated from the flow line calculation unit of Embodiment 1 or 2 or the flow line determination unit of Embodiment 3 or 4 determined to be correct. May be. For example, an abnormal flow line defined in advance, for example, a flow line that moves between the same imaging areas is compared with a calculated flow line (target flow line), and the result matches the abnormal flow line. In this case, the person corresponding to the target flow line is regarded as a suspicious person, and the person image and the flow line are displayed on a display device as an output unit. Thereby, the user of a person tracking device can know the person who performs abnormal action, and its action, without confirming all the flow lines. In this case, the target flow line is compared with the predefined abnormal flow line, and if they match, the information on the flow line, for example, the corresponding human image and the abnormal flow line determination unit that outputs the flow line information to the output unit May be provided in the person tracking device.

  In addition, only the person designated by the user may be tracked. For example, when the control unit displays each captured image of a plurality of cameras on a display device which is an output unit, and the user designates a person appearing in one captured image via the input unit (for example, clicks a person with a mouse) Only the designated person is tracked (for example, the control unit displays only a photographed image showing the designated person on the display device). Thereby, the user can pay attention to one person.

  In addition, when the person image recorded in the tracked person information recording unit is displayed as a thumbnail on the display device of the output unit and the user designates one person image via the input unit, information about the person image, For example, the control unit may be configured to display a captured image in which the person of the person image is reflected and a corresponding flow line.

  In addition, the control unit may be configured to display the person image of the person on the display device of the output unit if the person frames out of the photographed image and the person does not frame in the photographed image for a predetermined time. Good. The user can know a person who acts for a predetermined time between shooting areas (a place where no shooting is performed).

It is a schematic block diagram of the person tracking system which concerns on Embodiment 1 of this invention. It is a figure which shows the structure of the building of one Example in which a person tracking system is used. It is a figure which shows an example captured image. It is a figure which shows the captured image of another example. It is a figure for demonstrating the determination which an authentication result determination part performs. It is a figure which shows the correspondence of the information currently recorded on the trackee information recording part. It is a figure which shows an example of the operation | movement flow regarding authentication. It is a figure for demonstrating another determination which an authentication result determination part performs. It is a schematic block diagram of the person tracking system which concerns on Embodiment 3 of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Person tracking system, 12 Cameras, 14 Person tracking apparatus, 16 Control part, 18 Output part, 20 Input part, 22 Captured image recording part, 24 Person image detection part, 26 Image processing part, 28 Feature-value extraction part, 30 Person Authentication unit, 32 authentication result determination unit, 34 tracked person information recording unit, 36 flow line calculation unit, 38 flow line information recording unit,
110 human tracking system, 112 camera, 114 human tracking device, 116 control unit, 118 output unit, 120 input unit, 122 captured image recording unit, 124 human image detection unit, 126 image processing unit, 128 feature quantity extraction unit, 134 tracking Person information recording unit, 138 flow line information recording unit, 140 person authentication unit, 142 flow line calculation unit, 144 flow line determination unit

Claims (12)

A person tracking device that calculates a flow line by tracking the same person in a captured image obtained in a plurality of shooting areas,
Captured image recording means for acquiring and recording a captured image in which the capturing area and the capturing time are specified;
A person image detecting means for detecting a person image from the photographed image and creating a person image in which the photographing area of the photographed image and the information of the photographing time are associated;
Feature amount extraction means for extracting feature amounts from a person image;
A person authentication unit that performs authentication by comparing the two feature amounts and making a determination with a predetermined determination method so that each person of the two person images from which the feature amounts are extracted is the same or another person;
An authentication result judging means for judging whether or not the same authentication result authenticated by the person authenticating means is correct based on information on the photographing area and the photographing time corresponding to each of the two person images of the person who is authenticated as the same; ,
A flow line calculating means for calculating a flow line of the person based on a shooting area and a shooting time corresponding to the person image of the same authenticated person in the same authentication result determined to be correct by the authentication result determining means; A person tracking device comprising: Authentication result determination means
Refers to information on the shooting area and shooting time corresponding to each of two person images of the person who is authenticated as the same, and exists between each shooting area and the time constraints existing between the shooting times in each shooting area. Based on spatial constraints, temporal or / and spatial uncertainty that the person exists in one shooting area corresponding to one shooting time and in the other shooting area corresponding to the other shooting time Calculate the amount of authentication contradiction that indicates
The person tracking apparatus according to claim 1, wherein the same authentication result with a small amount of authentication contradiction is determined to be correct. Authentication result determination means
Refers to information on the shooting area and shooting time corresponding to each of two person images of the person who is authenticated as the same, and exists between each shooting area and the time constraints existing between the shooting times in each shooting area. Based on spatial constraints, temporal or / and spatial uncertainty that the person exists in one shooting area corresponding to one shooting time and in the other shooting area corresponding to the other shooting time Calculate the amount of authentication contradiction that indicates the likelihood, determine that the same authentication result with a large amount of authentication conflict is incorrect,
The person authentication means
Each time the authentication result determination means determines that the same authentication result is not correct,
The person tracking apparatus according to claim 1, wherein all authentication results are corrected so that a total of authentication contradiction amounts of the same authentication result for all persons in the person image is minimized. The amount of authentication contradiction is
Consists of the sum of the same authentication score indicating the accuracy of the same authentication of the person authentication means, the amount of temporal contradiction indicating the uncertainty of temporal uncertainty, and the value of the spatial contradiction indicating the uncertainty of spatial uncertainty weighted by a constant coefficient The person tracking device according to claim 2, wherein the person tracking device is provided. A person tracking device that calculates a flow line by tracking the same person in a captured image obtained in a plurality of shooting areas,
Captured image recording means for acquiring and recording a captured image in which the capturing area and the capturing time are specified;
A person image detecting means for detecting a person image from the photographed image and creating a person image in which the photographing area of the photographed image and the information of the photographing time are associated;
Feature amount extraction means for extracting feature amounts from a person image;
A person authentication unit that performs authentication by comparing two feature amounts and determining by a predetermined determination method that each person of the two person images from which the feature amounts are extracted is the same or different; and
A flow line calculating means for calculating a flow line of the person based on the shooting area and the shooting time corresponding to the person image of the person who is authenticated as the same;
And a flow line determination means for determining whether or not the flow line is correct based on a shooting area corresponding to a person image of a person who is authenticated as the same, a shooting time, and a positional relationship between the plurality of shooting areas. Person tracking device. The flow line determination means is
Refer to the shooting area corresponding to the person image of the person who is authenticated as the same, the shooting time, and the positional relationship between multiple shooting areas, and the time restrictions that exist between each shooting time in each shooting area and between each shooting area Based on the spatial constraints existing in the flow line, a flow line contradiction amount indicating temporal or / and spatial uncertainty of movement of the person corresponding to the flow line according to the flow line is calculated. The person tracking apparatus according to claim 5, wherein a small flow line is determined to be correct. The flow line determination means is
Refer to the shooting area corresponding to the person image of the person who is authenticated as the same, the shooting time, and the positional relationship between multiple shooting areas, and the time restrictions that exist between each shooting time in each shooting area and between each shooting area Based on the spatial constraints existing in the flow line, a flow line contradiction amount indicating temporal or / and spatial uncertainty of movement of the person corresponding to the flow line according to the flow line is calculated. Determine that the large flow line is not correct,
The person authentication means
Each time the authentication result determination means determines that the same authentication result is not correct,
The person tracking apparatus according to claim 5, wherein the authentication result is corrected so that a total of the flow line contradiction amounts of the flow lines with respect to the persons of all the person images is minimized. The amount of traffic line contradiction is
Consists of the same authentication score that shows the accuracy of the same authentication of the person authentication means, the amount of temporal contradiction that indicates the uncertainty of time, and the value that is weighted by a constant coefficient for the amount of spatial contradiction that indicates the uncertainty of spatial The person tracking device according to claim 6, wherein the person tracking device is performed. Image processing means for performing predetermined image processing on a person image;
The person tracking apparatus according to claim 1, wherein the feature quantity extraction unit extracts a feature quantity from the human image that has undergone image processing. The person image extraction means extracts the frog image as a person image,
The person tracking apparatus according to claim 1, wherein the feature amount extraction unit extracts a feature amount from the hail image.   The abnormal flow line determination means for determining by comparing the flow line calculated by the flow line calculation means or the flow line determined to be correct with the predefined abnormal flow line, and the determination of the abnormal flow line determination means The person tracking apparatus according to claim 1, further comprising an output unit that outputs a result. A plurality of cameras that shoot a predetermined shooting area and create a shot image with the shooting area and shooting time information added thereto;
A person tracking system comprising the person tracking apparatus according to claim 1.

Images