KR101712136B1 - Method and apparatus for detecting a fainting situation of an object by using thermal image camera - Google Patents

Abstract

A method for detecting a fainting condition according to an object tracking method using a thermal camera, the method comprising the steps of: extracting an object using difference between consecutive frames; Extracting and tracking an object using a camshift and a Kalman filter; Sensing a fainting condition of the object based on a change in trajectory and size of the tracked object and a change time analysis; And a step of outputting that the emergency situation occurs when a faulty situation occurs. This makes it possible to always extract objects regardless of the surrounding environment and has accurate tracking detection performance.

Description

[0001] The present invention relates to a method and apparatus for detecting a fainting condition of an object using an infrared camera,

The present invention relates to a method and apparatus for detecting a fainting situation of an object using a thermal imaging camera, and more particularly, to an object tracking method using an infrared camera capable of always extracting an object regardless of the surrounding environment, And a recording medium storing a software program for performing the method.

As a part of the national research and development project, "Gyeonggi Provincial Cooperation Center GRRC Suwon 2014-B3, Research Project Name: Gyeonggi Provincial Cooperation Research Center Support Project, Project Title: Development of cloud-based intelligent video surveillance monitoring system ".

Nowadays, aging society is an important social issue and it is causing big problems. At the same time, the number of patients with geriatric diseases is increasing. Especially, in the case of the elderly living alone, if an emergency such as a fainting situation occurs, since a countermeasure is insufficient, a fainting situation detection and response system is required to prevent this.

The elderly living alone or the patients alone may suffer a serious injury due to the shock of fainting in the event of an emergency such as a fainting condition, and a secondary emergency can occur if the person can not cope with it quickly. The inventor has developed a system for detecting fainting situations in order to prevent the occurrence of such a secondary emergency situation. In addition, it should be possible to accurately detect and detect a person in any day and night situations by using a thermal camera, and to accurately judge the fainting situation.

The object of the present invention is to provide a method and apparatus for detecting a fainting situation, which provides accurate fainting situation detection performance not only during the daytime but also during the daytime by tracking an object using a thermal imaging camera, a recording medium storing a software program for performing the method, And to provide the above-mentioned objects.

A problem to be solved by the present invention is to provide an apparatus and a method for tracking an object that can accurately detect a fainting situation at night as well as during a day by tracking an object using a thermal camera, And a recording medium storing a software program for performing the method. The present invention also provides a recording medium storing a software program for performing the method.

According to one aspect of the present invention, there is provided a method for detecting a fainting condition according to object tracking using a thermal camera, the method comprising: extracting an object using difference between consecutive frames; Extracting and tracking an object using a camshift and a Kalman filter; Sensing a fainting condition of the object based on a change in trajectory and size of the tracked object and a change time analysis; And a step of outputting that the emergency situation occurs when a faulty situation occurs.

In some embodiments, the method further comprises a re-tracing step of finding the object again by comparing the histogram between the object detected through labeling and the tracking object before missing if the object being tracked is missed.

= 블랍 라벨링을 통한 객체들의 히스토그램, Hist_CT = 현재 추적중인 객체의 히스토그램, Hist_PT = 추적에 실패할 때 프레임에서의 추적 객체 히스토그램임을 특징으로 한다. In some embodiments, if Hc < Wc and Wc = Wp and Hc = Hp and f &gt; F and fn-fm &lt; S, a person is considered to be a syncope, where Hp = the height of a person in the previous frame, Hc = Fc = frame number at which the person starts to bend; fn = frame number at which the person is laid on the floor; f = width of the person in the current frame; F = total frame (time) for determining whether a person is fainting, S = interval frame (time) for determining whether a person is fainting,

= Histogram of objects through bubble labeling, Hist _CT = histogram of the object being tracked, and Hist _PT = tracking object histogram in the frame when tracking fails.

(k = 검출된 객체의 수) 중 어떠한 것도 Hist_CT와 같지 않으면 추적에 실패한다고 가정하고

중 Hist_PT와 가장 유사한 객체를 재추적하며, 여기서,

= 블랍 라벨링을 통한 객체들의 히스토그램, Hist_CT = 현재 추적중인 객체의 히스토그램, Hist_PT = 추적에 실패할 때 프레임에서의 추적 객체 히스토그램임을 특징으로 한다.In some embodiments,

(k = number of objects detected) is not equal to Hist _CT ,

Retraces the object most similar to the Hist _PT ,

= Histogram of objects through bubble labeling, Hist _CT = histogram of the object being tracked, and Hist _PT = tracking object histogram in the frame when tracking fails.

In some embodiments, f is set to 0 and Wp is updated to Wc and Hp = Hc, and if Yc < Yp, Yc < Yth, fm = 0, = Maximum height of the person in the previous frame, Yc = maximum height of the person in the current frame, Yth = arbitrary height for judging that the person starts to fall down.

In some embodiments, set f = f + 1, update Wp = Wc and Hp = Hc, and if Yc < Yp, Wc Hmax, fm > 0, fn = 0, , Where Hmax = maximum vertical width of a person, Yp = maximum height of a person in a previous frame, Yc = maximum height of a person in the current frame, and Yth = arbitrary height for judging that a person starts to fall.

According to another aspect of the present invention, there is provided an apparatus for detecting a fainting condition according to object tracking using a thermal imaging camera, the apparatus comprising: Extracts an object using the difference between two consecutive frames; The extracted objects are subjected to object detection and object tracking using a camshift and a Kalman filter; Detecting a fainting condition of the object based on a change in trajectory and size of the tracked object and a change time analysis; And a controller for performing an operation of outputting an emergency state when a falsely falsified situation occurs.

In some embodiments, the control unit retrains an object when the object being tracked is missed by comparing the object detected through labeling and the tracked object before the missed object.

= 블랍 라벨링을 통한 객체들의 히스토그램, Hist_CT = 현재 추적중인 객체의 히스토그램, Hist_PT = 추적에 실패할 때 프레임에서의 추적 객체 히스토그램임을 특징으로 한다. ]In some embodiments, the control unit considers a person to be a syncope if Hc <Wc and Wc = Wp and Hc = Hp and f? F and fn-fm <S, where Hp = = Wp = width of a person in the previous frame, Wc = width of a person in the current frame, fm = frame number at which bending of a person starts, fn = frame number when a person lies on the floor, f = Cumulative number of people lying on a lying frame, F = Total frame (time) for determining whether a person is fainting, S = Interval frame (time)

= Histogram of objects through bubble labeling, Hist _CT = histogram of the object being tracked, and Hist _PT = tracking object histogram in the frame when tracking fails. ]

(k = 검출된 객체의 수) 중 어떠한 것도 Hist_CT와 같지 않으면 추적에 실패한다고 가정하고

중 Hist_PT와 가장 유사한 객체를 재추적하며, 여기서,

= 블랍 라벨링을 통한 객체들의 히스토그램, Hist_CT = 현재 추적중인 객체의 히스토그램, Hist_PT = 추적에 실패할 때 프레임에서의 추적 객체 히스토그램임을 특징으로 한다. In some embodiments, the control unit

(k = number of objects detected) is not equal to Hist _CT ,

Retraces the object most similar to the Hist _PT ,

= Histogram of objects through bubble labeling, Hist _CT = histogram of the object being tracked, and Hist _PT = tracking object histogram in the frame when tracking fails.

In some embodiments, the controller sets f = 0, updates Wp = Wc and Hp = Hc, and if Yc < Yp, Yc < Yth, fm = 0, the person assumes that he / , Where Yp = maximum height of a person in the previous frame, Yc = maximum height of a person in the current frame, and Yth = arbitrary height for determining that a person starts to fall.

In some embodiments, the control unit sets f = f + 1 and updates Wp = Wc and Hp = Hc, and if Yc < Yp, Wc = Hmax, fm > 0, fn = 0, Yc = maximum height of a person in the current frame, Yth = arbitrary height for determining that a person starts to fall down .

According to another aspect of the present invention, there is provided a computer-readable recording medium having recorded thereon a software program for performing the method for detecting a fainting condition.

A method and apparatus for detecting a fainting condition according to an object tracking method using a thermal imaging camera according to the present invention and a recording medium storing a software program for performing the method are capable of always extracting an object regardless of a surrounding environment, .

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a schematic block diagram illustrating the operation of an apparatus for detecting a fainting condition according to object tracking using a thermal imaging camera of the present invention.
FIG. 2 is a schematic flowchart for explaining an object tracking and re-tracking operation applied to a method for detecting a fainting condition according to an object tracking using a thermal imaging camera of the present invention.
FIG. 3 is a schematic flowchart for explaining the operation of the false-state detection method according to the object tracking using the thermal imaging camera of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, a method and apparatus for detecting a fainting condition according to an object tracking method using a thermal imaging camera according to the present invention will be described in detail with reference to the accompanying drawings.

FIG. 1 is a schematic diagram for explaining the operation of a device for detecting a fainting condition according to object tracking using a thermal imaging camera of the present invention. In FIG. 1, the thermal imaging camera 20 photographs an object to be sensed, mainly an elderly person or a patient 30, and sends the sensed image signal to the sensory position detecting apparatus 10 of the present invention. The control unit 15 in the control unit 15 detects whether or not the object is in a fainting state and outputs an alarm signal to the outside when a fainting situation occurs. Hereinafter, the operation of detecting a faulty situation in the control unit will be described in more detail with reference to FIG. 1, FIG. 2 and FIG.

FIG. 2 is a schematic flowchart for explaining an object tracking and re-tracking operation applied to a method for detecting a fainting condition according to an object tracking method using the thermal imaging camera of the present invention. FIG. 2 is a schematic flowchart for explaining the operation of a method for detecting a fainting condition according to an embodiment of the present invention; FIG.

1 and 2, the image input from the thermal imaging camera 20 is converted into a HSV (Hue, Saturation, Brightness Value) form (S210) Set the location to set the area of interest. That is, an object of interest (S212).

Next, object detection is performed by applying a camshift algorithm for object detection using color information (S214).

Then, a Kalman filter is applied to the detected object to track the detected object (S216).

Next, as will be described later with reference to FIG. 3, it is determined whether the object to be tracked is falsely applied (S310-S356) by applying the proposed sync state detection algorithm.

The object tracking signal (A) in step S216 of Figure 2 is the output signal (A) of the object tracking / re-tracking algorithm application step of Figure 3,

On the other hand, if the tracked object suddenly changes direction or changes its shape and fails to track, the histograms of the objects detected through the blob labeling and the histogram of the tracked object, as shown in steps S220 to S238 in Fig. 2 To perform the re-tracing.

Camshift algorithm

The camshift algorithm applied to the object tracking of the present invention is improved for use in an environment for detecting a mean shift algorithm which is a color segment method and is a technique for adjusting the size of a search window by itself It compensates for the shortcomings of shifting. It is used to detect objects at high speed and has poor performance in backgrounds with high luminance variation and noise. The object is continuously detected by locating the center after predicting and detecting the position to be changed by using the distribution of the hue value of the detected object area.

The RGB color model is much more sensitive to illumination changes. Therefore, this algorithm converts the RGB color space into the color of the HSV color model to reduce the influence of lighting on the tracking object. A one-dimensional histogram is constructed and stored in the region of interest and used as a model for tracking.

Kalman filter algorithm

Kalman Filter is a well known method in the field of motion prediction. It is a recursive algorithm to find an optimal estimate minimizing the error in the state vector in a linear dynamic system interfered by white Gaussian noise. The Kalman filter is divided into three stages: prediction, measurement, and modification. Efficient body tracking of the Kalman filter requires establishment of an appropriate tracking model. In the present invention, the state vector is set as the center coordinates (x, y) of the detected body and the amount of change (DELTA X, DELTA Y) between the previous frame and the current frames.

The state vector x (t) of the Kalman filter at time t is defined as:

...(1)

...(One)

The Kalman filter evolves along with the system state vector x (t) and time.

...(2)

...(2)

w (t) is defined as the covariance Q (t) and the mean zero Gaussian noise and is defined as the covariance Q (t).

...(3)

... (3)

The measurement vector is given by z (t).

...(4)

...(4)

V (t) is a Gaussian noise factor with a mean other than covariance R (t) and is defined as the covariance R (t).

...(5)

... (5)

In the present invention, it is defined as a state change matrix Φ (t), assuming that the face moves at a uniform velocity and in a linear direction.

...(6)

... (6)

The input vector is a four-dimensional vector using the center x and y coordinates and is defined by the measurement matrix H (t) according to the Δx, Δy axis and the body change.

...(7)

... (7)

Re-tracking algorithm

The image input from the thermal imaging camera 20 is converted into a gray scale (S220), and the image of the gray scale is converted into a binary image (S222).

At this time, remove objects other than objects between the lowest temperature and the highest temperature that a person can have to separate the foreground from the background. The bubble labeling excludes objects other than those between the minimum and maximum sizes a human can have. Bubble labeling checks neighboring pixel values and recognizes them as the same object if these pixel values are the same as the surrounding values. That is, after the morphology technique is performed on the binary image, the object is detected using the blob labeling (S224). Then, histograms of the detected objects are extracted (S226). The histograms of the detected objects are compared with the histogram of the object just before the failure, and the most similar objects are found and retraced. The histogram comparison method examines the similarity using the Bhattacharyya distance calculation method.

...(8)

...(8)

If there is no similarity between the histogram of the object currently being tracked and the histogram of the objects through the blub labeling (S230), it is regarded that the tracking is unsuccessful (S232) and the histogram of the tracking object is extracted from the previous frame (S234).

If the two histogram values are completely matched, then the batachaya distance returns 0, and if it matches only half, it returns 0.55. Returns 1 if there is a complete discrepancy. That is, the histograms of the detected objects are compared with the histograms of the objects immediately before the failure, and the similarity of the histograms is determined using the Batachaya distance calculation method (S236). The object having the histogram closest to 0 or 0 is retrained (S238).

Fainting situation recognition flowchart

Since the Kalman filter can successfully predict the state of the object in the next frame, the object can be properly tracked and the Camshift algorithm can change the size of the window according to the movement of the objects, Based on the analysis, we can recognize the fainting situation and the flow of the overall situation recognition algorithm is as follows.

Stage 1

Global parameter definition: Defines the global parameters used in the algorithm.

= 블랍 라벨링으로 검출된 객체의 히스토그램, Hp = 이전 프레임에서 사람의 세로 폭, Hc = 현재 이미지에 있는 사람의 세로 폭, Wp = 이전 이미지에서 사람의 가로 폭, Wc = 현재 이미지에서 사람의 가로 폭, Hmax = 사람의 최대 세로 폭, Yp = 이전 프레임에서 사람의 최대 높이, Yc = 현재 프레임에서 사람의 최대 높이 fc = 현재 프레임넘버, fm = 사람의 구부림이 시작되는 프레임넘버, fn = 사람이 바닥에 누웠을 때의 프레임넘버, f = 사람의 누워있는 이미지에 대한 누적, F = 사람의 실신 여부를 결정하는 전체 프레임 (시간), S = 사람의 실신 여부를 결정하는 구간 프레임(시간)이다.Hist _CT = Histogram of the object currently being tracked, Hist _PT = Histogram of the tracked object just before the trace failed,

= Histogram of object detected by bubble labeling, Hp = Height of person in previous frame, Hc = Height of person in current image, Wp = Width of person in previous image, Wc = Width of person in current image , The maximum height of the person in the previous frame, Yc = the maximum height of the person in the current frame, fc = the current frame number, fm = the frame number at which the human bend starts, fn = F = the total frame (time) for determining whether a person is fainting; S = the interval frame (time) for determining whether a person is fainting.

Step 2

The thermal imager 20 acquires an image and stores it as an RGB or gray-scale image (S210). The object is extracted from the difference between the current frame and the previous frame and displayed as a rectangular frame (S212).

Step 3

The camshift algorithm holds the initial position of the search window and performs object detection (S214). The Kalman filter predicts the position of the detected object and performs tracking (S216). At this time, the histogram information of the tracked object is stored in the Hist _CT in real time (S218).

Step 4

At the same time as tracking is performed, object detection is performed using blob labeling separately. The image obtained in step 2 is converted into a gray area (S220) and binarization is performed (S222). The gray image is called G (x, y), the binary image is called g (x, y), and the threshold is called the minimum temperature a person can have.

Step 5

The morphology technique is applied to the binarized image to fill the empty space and remove the noise.

Step 6

(k = 검출객체의 수) 에 저장한다(S228).After performing the morphology technique on the binarized image to fill the empty space and remove the noise, the binarized image is subjected to the blot labeling (blob labeling) to detect objects between the minimum size and the maximum size that a human can have (S224). A histogram of the detected objects is extracted (S226). The histogram of detected objects

(k = number of detected objects) (S228).

Step 7

들중 어떠한 것도 유사성을 띄우지 않는다면(S230) 추적에 실패한 것으로 간주하고(S232), 이전 프레임에서 추적 객체의 히스토그램을 추출하여(S234), Hist_PT 와

을 비교하여(S236) 가장 유사한 객체를 찾아 찾은 객체의 좌표값을 가져와 재추적을 수행한다(S238).If Hist _CT and

(S232), the histogram of the tracked object is extracted in the previous frame (S234), and the Hist _PT and the histogram of the tracked object are extracted

(S236), finds the closest object, obtains the coordinate value of the found object, and retries the object (S238).

들이 유사한 경우에는 단계 S216으로 진행하여 단계 S216 내지 S230을 반복하고, 단계 S238에서 가장 유사한 값을 가지는 객체를 관심 객체로 지정하는 경우에는 단계 S212로 진행하여 단계 단계 S212 내지 S238을 반복한다.If it is determined in step S230 that Hist _CT

The flow advances to step S216 to repeat steps S216 to S230. If an object having the most similar value is designated as an object of interest in step S238, the flow advances to step S212 to repeat steps S212 to S238.

Step 8

Calculate the size of the tracking window and perform syncope detection. If Hc ≥ Wc (S310), the person is considered to be standing or walking and continues at step 9 (S340).

If Hc <Wc, Wc ≠ Wp and Hc ≠ Hp and f <F (S320), a lying or bending is considered, and the process continues at step 9 (S340).

If Hc > = Wc, Wc = Wp and Hc = Hp and f > = F (S330), it is regarded as a lie.

Step 9

Reset the sync frame counter and start a new cycle. In step 9, setting f = 0, update Wp = Wc, and Hp = Hc (S340). If Yc <Yp, Yc <Yth, and fm = 0 (S342), the person is considered to have started to lie down or start fainting, and fm = fc and Yp = Yc (S344). If Yc < Yp, Yc < Yth, and fm = 0 are not satisfied (S342), step 8 is repeated.

Step 10

Increase the frame counter and start a new cycle. In step 10, setting f = f + 1, update Wp = Wc and Hp = Hc (S350). If Yc <Yp, Wc? Hmax, fm> 0, and fn = 0 (S352), the person is considered to be lying down or fainted and fn = fc and Yp = Yc (S354). If f? F and fm-fn < S (S356), it is regarded as a syncope and continues at step 11, and if not satisfied, the previous steps S210-S354 are repeated.

Step 11

In the case of fainting, an alarm sounds with a signal to perform the structure and manually reset the system (S358).

As described above, in the method and apparatus for detecting a fainting condition according to the present invention, the object being tracked has a flexible search window and it can be determined whether the object is lying or not by using this feature. In addition, it is determined whether the difference between the frame in the standing state of the object and the frame in the lying state is below the threshold value and the time during which the object is laid has lapsed for a certain period of time.

In addition, the present invention provides a histogram comparison between the objects detected through the blur labeling and the tracking object immediately before the tracking failure when the tracking object suddenly changes direction or the shape changes, You can find it again as the original object.

10 ... syncope sensor
15 ... control unit
20 ... Thermal Cameras
30 ... patient

Claims (13)

delete delete A method for detecting a fainting condition according to object tracking using an infrared camera,
Extracting an object using differences between two consecutive frames;
Extracting and tracking an object using a camshift and a Kalman filter;
Sensing a fainting condition of the object based on a change in trajectory and size of the tracked object and a change time analysis;
Outputting an emergency situation when a faulty situation occurs; And
Histogram of object detected through bubble labeling when missing tracked object

And a retrace step of finding the object again by comparing the histogram Hist _CT of the track object before the miss,
The re-tracing step may be performed with Hist _CT

If no similarity is found, it is assumed that the tracking has failed, and the histogram of the tracking object is extracted from the previous frame, and Hist _PT and

, Finds the most similar object, obtains the coordinate value of the found object,
The human fingers are considered to be fainted if Hc <Wc and Wc = Wp and Hc = Hp and fc = Hp and fc = F and fn-fm <S, where Hp = the height of the person in the previous frame, Hc = Fc = frame number at which human bending starts, fn = frame number at which the person is lying on the floor, f = width of the person in the previous frame, F = total frame (time) for determining whether a person is fainting, S = interval frame (time) for determining whether a person is fainting,

= Histogram of objects through bubble labeling, Hist _CT = Histogram of the object being tracked, Hist _PT = Tracking object histogram in the frame when tracking fails. delete The method according to claim 3, wherein the falsenal situation detection step sets f = 0, updates Wp = Wc and Hp = Hc, and if Yc <Yp, Yc <Yth, and fm = 0, Wherein Yp is a maximum height of a person in a previous frame, Yc is a maximum height of a person in a current frame, and Yth is an arbitrary height for determining that a person starts to fall down. The method according to claim 3, wherein the falsenal situation detection step sets f = f + 1, updates Wp = Wc and Hp = Hc, and if Yc < Yp, Wc? Hmax, fm > 0, Yp = maximum height of a person in the previous frame, Yc = maximum height of a person in the current frame, Yth = number of people to judge that a person starts to fall down Wherein the false alarm is an arbitrary height. delete delete An apparatus for detecting a fainting condition according to object tracking using an infrared camera,
Receiving an object signal photographed by the thermal imaging camera; Extracts an object using the difference between two consecutive frames; The extracted objects are subjected to object detection and object tracking using a camshift and a Kalman filter; Detecting a fainting condition of the object based on a change in trajectory and size of the tracked object and a change time analysis; And a controller for performing an operation of outputting an emergency state when a faulty situation occurs,
When the control unit misses the object being tracked, the histogram of the object detected through the blob labeling

And a retest operation for finding an object again by comparing the histogram Hist _CT of the track object before the miss,
The control unit in the re-tracking operation, and Hist _CT

If no similarity is found, it is assumed that the tracking has failed, and the histogram of the tracking object is extracted from the previous frame, and Hist _PT and

, Finds the most similar object, obtains the coordinate value of the found object,
The controller considers a person to be a fainted person if Hc <Wc and Wc = Wp and Hc = Hp and fc = F and fn-fm < , Wc = width of a person in the current frame, fm = frame number at which bending of a person starts, fn = frame when a person lies on the floor, F = total frame (time) for determining whether a person is fainting, S = interval frame (time) for determining whether a person is fainting or not,

= Histogram of objects through bubble labeling, Hist _CT = Histogram of the object being tracked at the moment, Hist _PT = Tracked object histogram in the frame when tracking fails,
The controller considers a person to be a fake if Hc <Wc and Wc = Wp and Hc = Hp and fc-Fn and fn-fm <S, where Hp = the height of the person in the previous frame, Hc = Fn is the frame number at which the person is laid on the floor, f is the length of the person's lying on the floor, Wp is the width of the person in the previous frame, Wc is the width of the person in the current frame, fm = F = total frame (time) for determining whether a person is fainting, S = interval frame (time) for determining whether a person is fainting,

= Histogram of objects through bubble labeling, Hist _CT = Histogram of the object being tracked, Hist _PT = Tracking object histogram in frame when tracking fails. delete 10. The method of claim 9, wherein the controller sets f = 0, updates Wp = Wc and Hp = Hc, and if Yc < Yp, Yc < Yth, fm = 0, Wherein Yp = a maximum height of a person in a previous frame, Yc = a maximum height of a person in the current frame, and Yth = an arbitrary height for judging that a person starts to fall down. 10. The method of claim 9, wherein the controller sets f = f + 1, updates Wp = Wc and Hp = Hc, and if Yc <Yp, Wc? Hmax, fm> 0, fn = 0, The maximum height of the person in the previous frame, Yc = the maximum height of the person in the current frame, and Yth = the arbitrary height for judging that the person starts to fall down. (1). A computer-readable recording medium recording a software program for performing the method for detecting a fainting condition according to any one of claims 3, 5, and 6.

Images