JP7786061B2 - Control device, control method, control program, and information processing system - Google Patents

Description

The present invention relates to a control device, a control method, a control program, and an information processing system.

At nursing homes for the elderly, hospitals, and other long-term care facilities, care recipients (hereinafter referred to as "care recipients") who are residents or hospitalized may fall while walking, fall out of bed and injure themselves, or abnormalities may be detected in their biometric information. Therefore, development is underway to develop monitoring systems that can constantly monitor the condition of care recipients, so that nurses, caregivers, etc. (hereinafter referred to as "caregivers") can quickly rush to the scene when such situations occur.

Patent Document 1 below describes a monitoring system that uses an image recognition unit to recognize a patient's body parts as objects or parts from images captured by a monitor camera attached to an electric bed.

JP 2018-134278 A

The technology in Patent Document 1 has the advantage of installing a camera on an electric bed, eliminating the need to align the camera with the electric bed or the surrounding environment, thereby improving convenience for caregivers. However, the technology in Patent Document 1 has the potential to cause frequent false detections based on images. After careful consideration, the inventors found that problems can arise due to the operation of the electric bed. Specifically, electric beds have functions for adjusting the bed height, back lift angle, knee lift angle, foot lift angle, and tilt angle. Installing a camera on an electric bed changes the orientation of the camera in response to movements such as tilting of the electric bed, and it was found that this can cause problems in detecting the condition of the care recipient based on images.

The present invention was made to solve these problems. Specifically, its purpose is to provide a control device, control method, control program, and information processing system that can properly detect the condition of a care recipient using images even if the bed moves, such as by tilting.

The above-mentioned problems of the present invention are solved by the following means.

(1) A control device comprising an image acquisition unit that acquires images taken by an imaging unit installed in a bed, an image analysis unit that analyzes the images, and a control unit that controls the imaging unit or the image analysis unit based on the movement or state of the bed, wherein the bed is an electric bed, and the control unit determines the movement or state of the bed based on a signal from the electric bed .

(2) A control device comprising an image acquisition unit that acquires an image taken by an imaging unit installed on a bed, an image analysis unit that analyzes the image, and a control unit that controls the imaging unit or the image analysis unit based on the movement or state of the bed , wherein the image analysis unit detects objects included in the image taken by the imaging unit, compares the position of the top end of a bed object corresponding to the bed in the image with the position of the top end of a person object corresponding to a person in the image, and determines that the person has not fallen if the position of the top end of the person object is higher than the position of the top end of the bed object, but determines that the person has fallen if the position is lower .

(3) A control device comprising: an image acquisition unit that acquires images taken by an imaging unit installed on a bed; an image analysis unit that analyzes the images; a sensor that detects the movement of the bed, including the movement of tilting the mattress of the bed and the movement of moving the mattress of the bed, or the state of the bed, including a reclining state in which the mattress is tilted relative to the floor; and a control unit that controls the imaging unit or the image analysis unit based on the movement or state of the bed, wherein the control unit controls the orientation or position of the imaging unit based on the movement or state of the bed .

(4) A control device comprising: an image acquisition unit that acquires images taken by an imaging unit installed on a bed; an image analysis unit that analyzes the images; a sensor that detects the movement of the bed, including the movement of tilting the mattress of the bed and the movement of moving the mattress of the bed, or the state of the bed, including a reclining state in which the mattress is inclined relative to the floor; a control unit that controls the imaging unit or the image analysis unit based on the movement or state of the bed; and a reception unit that receives detection results from the sensor that detects the movement or state of the bed, wherein the control unit determines the movement or state of the bed based on the detection results .

(5) A control device comprising: an image acquisition unit that acquires images taken by an imaging unit installed on a bed; an image analysis unit that analyzes the images; a sensor that detects the movement of the bed, including the movement of tilting the mattress of the bed and the movement of moving the mattress of the bed, or the state of the bed, including the state in which the mattress is tilted relative to the floor surface; and a control unit that controls the imaging unit based on the movement or state of the bed.

(6) The control unit controls the image analysis unit to change the state determination threshold used to determine the state of the person being cared for, or the posture determination standard used to determine the posture of the person being cared for, based on the movement or state of the bed, and the posture determination standard includes a normal state standard set when the bed is in a normal state and a reclining state standard set when the bed is in a reclining state, and the normal state standard is a standard for determining that the person being cared for is in a non-falling state when the posture of the person being cared for is determined to be standing, and for determining that the person being cared for is in a falling state when the posture of the person being cared for is determined to be sitting, and the reclining state standard is a standard for determining that the person being cared for is in a falling state when the posture of the person being cared for is determined to be standing, a control device described in any one of (1) to (4) above.

(7) A control device comprising an image acquisition unit that acquires images taken by an imaging unit installed on a bed, an image analysis unit that analyzes the images, and a control unit that controls the imaging unit or the image analysis unit based on the movement or state of the bed, wherein the control unit controls the orientation or position of the imaging unit based on the movement or state of the bed, and the control unit converts the position of each pixel of the image acquired by the image acquisition unit to the position of an image obtained when the bed is in a normal state based on the movement or state of the bed, and controls the image analysis unit to correct the image acquired by the image acquisition unit by performing calculation processing to interpolate between the pixels.

(8) The control device described in (6) above further includes a memory unit that stores the state determination threshold, and the image analysis unit detects objects included in an image captured by the imaging unit and determines the state of the person based on the comparison result between the position of the top end of a human object corresponding to the person in the image and the state determination threshold.

(9) The control device described in (8) above, wherein the image analysis unit changes the state determination threshold based on the movement or state of the bed.

(10) The control device described in (6) above further includes a memory unit that stores the posture determination criteria, and the image analysis unit detects the skeleton and joint points of a human object corresponding to a person included in an image captured by the imaging unit, and determines the posture of the person and the state of the person corresponding to the posture based on the detected skeleton and joint points and the posture determination criteria.

(11) The control device described in (10) above, wherein the image analysis unit changes the posture determination criteria based on the movement or state of the bed.

(12) A control method comprising: a control step of controlling the imaging unit based on the operation or state of a bed on which the imaging unit is installed; a photographing step of photographing an object to be photographed, including the bed, using the imaging unit; an acquisition step of acquiring the image photographed by the imaging unit using an image acquisition unit; and an analysis step of analyzing the image.

(13) A control method comprising: an imaging step of photographing an object including the bed using an imaging unit installed on the bed; an acquisition step of acquiring the image photographed by the imaging unit using an image acquisition unit; a step of receiving a detection result from a sensor that detects the movement or state of the bed; and a control step of determining the movement or state of the bed based on the detection result and controlling an image analysis unit that analyzes the image based on the movement or state of the bed.

(14) A control method comprising: a detection step of detecting the operation of the bed, including the operation of tilting the mattress of the bed and the operation of moving the mattress of the bed, or the state of the bed, including a reclining state in which the mattress is inclined relative to the floor surface; a control step of controlling the imaging unit based on the operation or state of the bed in which the imaging unit is installed; a photographing step of photographing a subject including the bed using the imaging unit; an acquisition step of acquiring the image photographed by the imaging unit using an image acquisition unit; and an analysis step of analyzing the image .

(15) A control method comprising: an imaging step of photographing an object to be photographed, including the bed, using an imaging unit installed on the bed; an acquisition step of acquiring the image photographed by the imaging unit using an image acquisition unit; a detection step of detecting, using a sensor, the movement of the bed, including the movement of tilting the mattress of the bed and the movement of moving the mattress of the bed, or the state of the bed, including a reclining state in which the mattress is inclined relative to the floor surface; a step of receiving a detection result from the sensor that detects the movement or state of the bed; and a control step of determining the movement or state of the bed based on the detection result and controlling an image analysis unit that analyzes the image based on the movement or state of the bed .

(16) A control program for causing a computer to execute the control method according to any one of (12) to (15) above.

(17) An information processing system having the control device according to any one of (1) to (11) above.

According to the present invention, the imaging unit or image analysis unit is controlled based on the movement or state of the bed, so the condition of the care recipient can be properly detected from images even if the bed moves due to tilting, etc.

1 is a diagram showing the overall configuration of a monitoring system according to a first embodiment; 2 is a schematic diagram illustrating a bed of a care recipient on which the detection device shown in FIG. 1 is installed. FIG. FIG. 2 is a block diagram showing a hardware configuration of the detection device shown in FIG. 1 . 4 is a block diagram illustrating the main functions of the arithmetic and control unit shown in FIG. 3. FIG. FIG. 2 is a block diagram showing a hardware configuration of the server shown in FIG. 1 . 4 is a flowchart illustrating a processing procedure of a control method for the detection device according to the first embodiment. FIG. 2 is a schematic side view of the bed in a normal state and the detection device. FIG. 10 is a schematic diagram illustrating an example of an image including a bed and a care recipient in a non-falling state. FIG. 10 is a schematic diagram illustrating an example of an image including a bed and a care recipient in a fallen state. FIG. 2 is a schematic side view of the bed in a reclining state and the detection device. FIG. 10 is a schematic diagram illustrating an example of an image including a bed and a care recipient in a non-falling state. FIG. 10 is a schematic diagram illustrating an example of an image including a bed and a care recipient in a fallen state. FIG. 2 is a schematic side view of the bed in a reclining state and the detection device. FIG. 10 is a schematic diagram illustrating an example of an image including a bed and a care recipient in a non-falling state. FIG. 10 is a schematic diagram illustrating an example of an image including a bed and a care recipient in a fallen state. 10 is a flowchart illustrating a processing procedure of a control method for a detection device according to a second embodiment. FIG. 10 is a schematic diagram illustrating an image after distortion correction including a bed and a care recipient in a non-falling state. 10 is a schematic diagram illustrating an example of an image after distortion correction, including a bed and a care recipient in a fallen state; FIG. 10 is a flowchart illustrating a processing procedure of a control method for a detection device according to a third embodiment. FIG. 10 is a schematic diagram illustrating an example of an image including a bed and a care recipient in a non-falling state. FIG. 10 is a schematic diagram illustrating an example of an image including a bed and a care recipient in a fallen state. 10 is a schematic diagram illustrating skeletal detection when the bed is in a normal state and the care recipient is not in a fallen state. FIG. 10 is a schematic diagram illustrating skeletal detection when the bed is in a normal state and the care recipient has fallen. FIG. 10 is a schematic diagram illustrating skeletal structure detection when the bed is in a reclining position and the care recipient is not in a fallen position. FIG. 10 is a schematic diagram illustrating skeletal structure detection when the bed is in a reclining position and the care recipient has fallen. FIG. FIG. 13 is a schematic side view of a bed in a normal state and a detection device according to the fifth embodiment. 10 is a schematic diagram of a bed and a detection device viewed from the side in a state where the mattress on the foot side is inclined. FIG. FIG. 13 is a schematic side view of a bed in a normal state and a detection device according to the sixth embodiment. FIG. 2 is a schematic side view of the bed in a reclining state and the detection device.

The following describes a control device, control method, control program, and information processing system according to an embodiment of the present invention, with reference to the drawings. Note that in the drawings, identical elements are given the same reference numerals, and duplicate explanations will be omitted. Also, the dimensional proportions in the drawings have been exaggerated for the sake of explanation, and may differ from the actual proportions.

(First embodiment)
FIG. 1 illustrates the overall configuration of a monitoring system 1 according to a first embodiment. In this embodiment, the monitoring system 1 constitutes an information processing system and is used in a nursing care facility or the like. The monitoring system 1 includes a detection device 10, a server 20, multiple mobile terminals 30, and a wireless AP (access point) 31. The detection device 10, the server 20, and the mobile terminals 30 are communicably connected to each other via a local area network (LAN) and the wireless AP 31. The mobile terminals 30 are carried by a caregiver 70 who cares for or provides care to a care recipient 80. The detection device 10 constitutes a control device either alone or together with an arithmetic and control unit 201 (see FIG. 5 ) of the server 20. The following description will primarily focus on the case where the detection device 10 constitutes a control device alone. The care recipient 80 constitutes a subject to be photographed. While the following description will be given assuming that the subject to be photographed is the care recipient 80, the subject to be photographed may also include a clinical trial participant or the like.

(Detection device 10)
Fig. 2 is a schematic diagram illustrating a bed 90 of a care recipient 80 on which the detection device 10 shown in Fig. 1 is installed. Fig. 3 is a block diagram illustrating a hardware configuration of the detection device 10 shown in Fig. 1.

As shown in FIG. 2, the bed 90 includes a frame unit 91, a headboard 92, a footboard 93, a mattress 94, and a sensor unit 95. In this embodiment, the bed 90 is, for example, an electric bed, and is equipped with a first drive device (not shown) that partially tilts the mattress 94 corresponding to the back, knees, and feet of the care recipient 80 when lying supine on the bed 90. For example, by tilting the portion of the mattress 94 corresponding to the back of the care recipient 80, a tilted (reclining) state can be created. The operation of the first drive device is controlled by a control signal from a control device (not shown) of the electric bed.

Furthermore, the frame portion 91 has an upper frame 911 and a lower frame 912, and the height (floor height) of the mattress 94 relative to the floor surface can be adjusted by moving the lower frame 912 toward or away from the upper frame 911 (see Figures 2, 7A, etc.). Also, by using the first drive device to independently move the head side and foot side of the lower frame 912 toward or away from the upper frame 911, the upper frame 911 can be tilted relative to the floor surface.

The sensor unit 95 has a sensor that detects the movement or state of the bed. The sensor may be a known acceleration sensor, tilt sensor, angle sensor, etc. The detection results from the sensor are sent to the calculation control unit 101.

As shown in FIG. 3, the detection device 10 has an arithmetic and control unit 101, a communication unit 102, and a camera 103, all of which are interconnected by a bus. The detection device 10 is installed, for example, on the upper end of the mattress 94 of a bed 90, and captures images within the monitoring area, which includes the bed 90 and the care recipient 80 as the subject of the image capture. The detection device 10 also has a rotation drive unit that rotates the housing that houses the arithmetic and control unit 101, communication unit 102, and camera 103 around the Z-axis and Y-axis as rotation axes, and can freely change the orientation of the housing in response to commands from the arithmetic and control unit 101 (see FIG. 7A for coordinate axis display). This allows the arithmetic and control unit 101 to control the direction in which the camera 103 captures images. The detection device 10 can also be equipped with a mechanism for moving the housing in the X-axis or Z-axis direction.

The arithmetic and control unit 101 includes a CPU, RAM, ROM, etc. The arithmetic and control unit 101 may also include an HDD or SSD (Solid State Drive). The arithmetic and control unit 101 controls each unit of the detection device 10 and performs arithmetic processing in accordance with a control program. The main functions of the arithmetic and control unit 101 will be described later. The RAM stores the state determination thresholds (first threshold, second threshold) and posture determination criteria (normal state criteria, reclined state criteria) described later.

The communication unit 102 is an interface circuit (e.g., a LAN card, wireless communication circuit, etc.) for communicating with, for example, the server 20 via a LAN.

Camera 103 captures images within the monitoring area and outputs the captured images (image data). Camera 103 constitutes the imaging unit. The captured images include still images and video. Video may include multiple still images captured intermittently in time series. Camera 103 is a near-infrared camera, but a visible light camera may be used instead, or a combination of these may be used. Camera 103 is fixed in the housing of detection device 10 so that the optical axis of the lens faces the foot of bed 90, and camera 103 also rotates and moves in accordance with the rotation and movement of the housing.

Figure 4 is a block diagram illustrating the main functions of the calculation and control unit 101 shown in Figure 1. The calculation and control unit 101 functions as an image acquisition unit 111, an image analysis unit 112, a reception unit 113, and a control unit 114.

The image acquisition unit 111 acquires images captured by the camera 103 installed on the bed 90. The image analysis unit 112 analyzes the images acquired by the image acquisition unit 111. In this embodiment, the image analysis unit 112 determines whether the care recipient 80 has fallen or is not falling based on the images. The reception unit 113 receives the detection results from the sensor of the sensor unit 95 and a control signal from the control device of the bed 90.

The control unit 114 determines the movement or state of the bed based on the detection results of the sensor in the sensor unit 95 and/or a control signal from the control device of the bed 90, and controls the camera 103 based on the determined movement or state. The movement of the bed 90 includes, for example, an operation of tilting a part of the mattress 94 relative to the upper frame 911, an operation of tilting the upper frame 911 relative to the floor surface, etc. The state of the bed 90 also includes, for example, a state in which the mattress 94 is tilted relative to the floor surface, a state in which it is not tilted, etc.

(Server 20)
5 is a block diagram showing the hardware configuration of the server 20. The server 20 has an arithmetic and control unit 201, a communication unit 202, and a storage unit 203, which are interconnected by a bus.

The arithmetic and control unit 201 includes a CPU, RAM, ROM, etc. The arithmetic and control unit 201 controls each unit of the server 20 and performs arithmetic and control processing in accordance with a program. The arithmetic and control unit 201 also constitutes a control device together with the detection device 10, and can be configured to cooperate with the image analysis unit 112 to carry out image analysis processing.

The communication unit 202 is composed of an interface circuit for communicating with, for example, a mobile terminal 30 via a LAN. The communication unit 202 receives the assessment results of the condition of the care recipient 80 and/or captured images from the detection device 10.

The storage unit 203 is composed of an HDD or SSD and stores various programs and data.

(Control Method)
FIG. 6 is a flowchart illustrating the processing procedure of the control method of the control device in the first embodiment. The processing of the flowchart in FIG. 6 is realized by the CPU of the arithmetic and control unit 101 executing a control program. FIG. 7A is a schematic diagram of the bed 90 with the mattress 94 horizontal to the floor, and the detection device 10, viewed from the side. FIG. 7B is a schematic diagram illustrating an image including the bed 90 and the care recipient 80 in a non-falling state. FIG. 7C is a schematic diagram illustrating an image including the bed 90 and the care recipient 80 in a fallen state. Also, FIG. 8A is a schematic diagram of the bed 90 in a reclining state, and the detection device 10, viewed from the side. FIG. 8B is a schematic diagram illustrating an image including the bed 90 and the care recipient 80 in a non-falling state. FIG. 8C is a schematic diagram illustrating an image including the bed 90 and the care recipient 80 in a fallen state. Also, Figure 9A is a schematic diagram of the bed 90 in a reclining state and the detection device 10 viewed from the side, Figure 9B is a schematic diagram illustrating an image including the bed 90 and the cared-for person 80 in a non-falling state, and Figure 9C is a schematic diagram illustrating an image including the bed 90 and the cared-for person 80 in a fallen state.

In Figures 7A, 8A, and 9A, the direction from left to right on the paper (the direction from the head side to the foot side of the bed 90) is defined as the X-axis direction, the direction from the front to the back of the paper (the width direction of the bed 90) is defined as the Y-axis direction, and the direction from bottom to top of the paper (the height direction of the bed 90) is defined as the Z-axis direction. Also, to simplify the explanation, the object corresponding to the footboard 93 is omitted from Figures 7B, 7C, 8B, 8C, 9B, and 9C.

First, the orientation or position of the camera 103 is controlled to set the monitoring area (step S101). The control unit 114 controls the orientation or position of the camera 103 based on the state of the bed 90 and sets the monitoring area. For example, as shown in FIG. 7A, the control unit 114 sets the parameters of the camera 103 to initial values when the mattress 94 of the bed 90 is horizontal, i.e., not tilted, relative to the floor (hereinafter referred to as the "normal state"). In this embodiment, the parameters include, for example, pan P, tilt T, and magnification M (or zoom Z). The control unit 114 sets the parameters (P, T, M) to initial values (P0, T0, M0). The state of the camera 103 in which the parameters are set to their initial values is referred to as the initial state. In the initial state, pan P0 and tilt T0 are set so that the optical axis of the camera 103 is directed toward the foot side of the mattress 94. Furthermore, in the initial state, magnification M0 is set so that the monitoring area of the camera 103 includes at least the foot side portion of the mattress 94. By setting the parameters in this way, the monitoring area of the camera 103 is set.

Alternatively, instead of setting the pan P, tilt T, and magnification M based on the state of the bed 90, the position of the camera 103 may be controlled. For example, when the bed 90 is in a normal state, moving the position of the camera 103 in the X-axis direction can achieve an effect equivalent to changing the magnification M.

Next, camera 103 begins capturing images of the area being monitored (step S102). Camera 103 begins (continues) capturing images of the area being monitored that has been set based on the parameters. As an example, consider a case in which care recipient 80, who is lying in bed 90, gets up from bed 90, gets off the bed, and attempts to walk. Camera 103 captures images of care recipient 80 and bed 90 within the area being monitored.

Next, the captured image is acquired (step S103). The image acquisition unit 111 acquires the image captured by the camera 103 and transmits it to the image analysis unit 112.

Next, objects are detected in the image (step S104). The image analysis unit 112 detects objects included in the image acquired by the image acquisition unit 111. For example, as shown in FIG. 7B, the image analysis unit 112 detects an object 940 corresponding to a bed 90 (hereinafter referred to as a "bed object") and an object 800 corresponding to a care recipient 80 who is trying to walk (hereinafter referred to as a "person object") 800. Note that object detection in an image can be achieved using a known algorithm, so a detailed description of the object detection method will be omitted.

Next, the state of the care recipient 80 is determined (step S105). The image analysis unit 112 determines the state of the care recipient 80 based on the position of the upper end of the detected person object 800. More specifically, the position of the upper end of the bed object 940 is set as the first threshold TH1, which is the state determination threshold, and the position B of the upper end of the person object 800 is compared with the state determination threshold. For example, as shown in FIG. 7B, the image analysis unit 112 determines that the care recipient 80 is in a non-falling state if the position B of the upper end of the person object 800 is greater than the state determination threshold. On the other hand, for example, as shown in FIG. 7C, the image analysis unit 112 determines that the care recipient 80 is in a falling state if the position B of the upper end of the person object 800 is smaller than the state determination threshold. Note that the first threshold TH1 may be a value pre-stored in the RAM of the calculation control unit 101.

Next, the determination result is output (step S106). The calculation and control unit 101 outputs the determination result obtained by the image analysis unit 112. The determination result is transmitted to the mobile terminal 30 by the communication unit 202.

In this way, when the bed 90 is in a normal state, i.e., when the mattress 94 is not tilted relative to the floor surface, the camera 103 captures an image within the monitoring area with its parameters set to their initial values, and the state of the care recipient 80 (fall/not fall) can be determined based on the acquired image.

However, when the bed 90 is in a reclined position as shown in FIG. 8A, the distance between the camera 103 and the mattress 94 in the monitoring area becomes closer as shown in FIGS. 8B and 8C, causing the shapes of the person object 800 and bed object 940 to change (e.g., become larger) compared to their shapes when the bed 90 is in a normal position. As a result, the position B of the top edge of the person object 800 is not compared correctly with the state determination threshold, and it may not be possible to properly determine the state of the care recipient 80.

As shown in Figure 9A, the pan P and tilt T of camera 103 are changed (controlled) to P1 and T1, respectively, so that the angle between the optical axis of camera 103 and the head end of mattress 94 corresponds to the inclination angle a1 of mattress 94, thereby making the optical axis of camera 103 roughly parallel to the foot end of mattress 94. Furthermore, as shown in Figures 9B and 9C, the magnification of camera 103 is set to an appropriate value M1 so that the shape of bed 90 in the captured image is equivalent to the shape of bed 90 in its normal state. By setting the parameters (P, T, M) in this way and capturing an image within the monitoring area, the condition of care recipient 80 can be appropriately determined based on the captured image.

In addition, instead of controlling the pan P, tilt T, and magnification M, the position of the camera 103 may be controlled to move in the X-axis and/or Z-axis directions.

Furthermore, although the case where the orientation or position of the camera 103 is controlled based on the state of the bed 90 (e.g., normal state or reclined state) has been described, the orientation or position of the camera 103 may also be controlled based on the movement of the bed 90 (e.g., movement of the mattress 94). Movement of the mattress 94 can be detected, for example, by an acceleration sensor of the sensor unit 95 installed on the mattress 94.

The detection device 10 of this embodiment described above controls the camera 103 based on the movement or state of the bed 90, so even if the bed 90, i.e., the camera 103, moves due to the bed 90 tilting, for example, the condition of the care recipient can be appropriately detected from the image.

Second Embodiment
In the first embodiment, the camera 103 parameters are controlled based on the movement or state of the bed 90 to capture an image of a set monitoring area, and the state of the care recipient 80 is determined based on the captured image. In the second embodiment, the camera 103 corrects the distortion (shape) of the image captured based on the movement or state of the bed 90, and the state of the care recipient 80 is determined based on the corrected image. Note that the correction of "distortion" in this specification refers to correcting (converting) an image captured when the bed 90 is in a reclined position into an image equivalent to an image captured when the bed 90 is in a normal position, and is unrelated to distortion in the optical system (lens) of the camera 103. To avoid duplication, detailed descriptions of components identical to those in the first embodiment will be omitted below.

Figure 10 is a flowchart illustrating the processing steps of the control method for the detection device 10 in the second embodiment. The processing of the flowchart in this figure is realized by the CPU of the calculation control unit 101 executing a control program. Figure 11 is a schematic diagram illustrating an image after distortion correction, including a bed and a care recipient in a non-falling state. Figure 12 is a schematic diagram illustrating an image after distortion correction, including a bed and a care recipient in a fallen state.

As shown in FIG. 10, first, the camera 103 starts capturing images of the monitoring area (step S201). The control unit 114 sets the camera 103 parameters (P, T, M) to initial values (P0, T0, M0). This puts the camera 103 into its initial state (see FIG. 7A).

Next, the captured image is acquired (step S202). The image acquisition unit 111 acquires the image captured by the camera 103. As described above, when the state of the bed 90 changes from the normal state to the reclined state, the distance between the camera 103 and the mattress 94 in the monitored area decreases, and the shapes of the person object 800 and the bed object 940 change from their shapes in the normal state (see Figures 8B and 8C).

Next, distortion of the acquired image is corrected (step S203). The image analysis unit 112 corrects distortion of the image acquired by the image acquisition unit 111 based on the state of the bed 90. The image analysis unit 112 performs predetermined arithmetic processing to correct distortion on the image acquired by the image acquisition unit 111, thereby generating an image equivalent to the image obtained when the bed 90 is in its normal state. For example, as shown in Figures 11 and 12, the image analysis unit 112 uses existing morphing technology or the like to convert the position of each pixel in the image acquired by the image acquisition unit 111 to the position of the image obtained when the bed 90 is in its normal state, and performs arithmetic processing to interpolate between pixels.

Next, objects are detected in the image (step S204). The image analysis unit 112 detects objects contained in the distortion-corrected image.

Next, the condition of the care recipient 80 is determined (step S205). Since the distortion-corrected image is equivalent to an image captured under normal conditions, the same method as for determining the condition of the care recipient 80 in the first embodiment can be applied, and detailed explanations thereof will be omitted.

Next, the determination result is output (step S206). The calculation and control unit 101 outputs the determination result obtained by the image analysis unit 112. The determination result is transmitted to the mobile terminal 30 by the communication unit 202.

In this way, in the processing of the flowchart shown in Figure 10, the camera 103 captures an image of the monitored area and acquires the image. Then, based on the movement or state of the bed 90, distortion in the image is corrected, and the state of the care recipient 80 is determined based on the distortion-corrected image, and the determination result is output.

The detection device 10 of this embodiment described above determines the condition of the care recipient 80 based on images in which distortion has been corrected, and is therefore able to properly detect the condition of the care recipient from the image even if the bed 90 moves, for example, by tilting.

(Third embodiment)
In the third embodiment, a case will be described in which the image analysis method is changed based on the operation or state of the bed 90. In the following, to avoid duplication of explanation, detailed explanation of the same configuration as in the first embodiment will be omitted.

Figure 13 is a flowchart illustrating the processing steps of the control method for the detection device 10 in the third embodiment. The processing of the flowchart shown in Figure 13 is realized by the CPU of the calculation control unit 101 executing a control program. Also, Figure 14 is a schematic diagram illustrating an example of changing the state determination threshold when the bed 90 is in a reclining state and the care recipient 80 is not in a fallen state, and Figure 15 is a schematic diagram illustrating an example of changing the state determination threshold when the bed 90 is in a reclining state and the care recipient 80 is in a fallen state.

Below, we will consider a case where the bed 90, which is initially in a normal state, changes to a reclining state. First, as shown in Figure 13, the camera 103 begins capturing images of the monitoring area (step S301). Initially, the bed 90 is in a normal state, and the parameters of the camera 103 are set to their initial values (P0, T0, M0) (see Figure 7A).

Next, the captured image is acquired (step S302). The image acquisition unit 111 acquires the image captured by the camera 103. As described above, when the state of the bed 90 changes from the normal state to the reclined state, the distance between the camera 103 and the mattress 94 in the monitored area decreases, and the shapes of the person object 800 and the bed object 940 change from their shapes in the normal state (see Figures 14 and 15).

Next, objects are detected in the image (step S303). The image analysis unit 112 detects objects included in the image acquired by the image acquisition unit 111. For example, as shown in Figures 14 and 15, the image analysis unit 112 detects a bed object 940 corresponding to the bed 90 and a person object 800 corresponding to the care recipient 80.

Next, the image analysis method is changed (step S304). When the state of the bed 90 changes from the normal state to the reclining state, the image analysis unit 112 changes the state determination threshold from the first threshold TH1 to the second threshold TH2. The first threshold TH1 is the upper end of the bed object 940 when the bed 90 is in the normal state, and the second threshold TH2 is the upper end of the bed object 940 when the bed 90 is in the reclining state.

Next, the state of the care recipient 80 is determined (step S305). The image analysis unit 112 compares the position B of the top end of the person object 800 with the state determination threshold. For example, as shown in FIG. 14, if the position B of the top end of the person object 800 is greater than the state determination threshold, the image analysis unit 112 determines that the care recipient 80 is not in a fallen state. Note that because the person object 800 exceeds the range of the monitoring area, the position B of the top end is taken as the top end of the image of the person object 800 for convenience. On the other hand, for example, as shown in FIG. 15, if the position B of the top end of the person object 800 is smaller than the state determination threshold, the image analysis unit 112 determines that the care recipient 80 is in a fallen state. Note that the second threshold TH2 may use a value pre-stored in the RAM of the calculation control unit 101.

Next, the determination result is output (step S306). The calculation and control unit 101 outputs the determination result obtained by the image analysis unit 112. The determination result is transmitted to the mobile terminal 30 by the communication unit 202.

In this way, in the processing of the flowchart shown in Figure 13, the camera 103 captures an image within the monitoring area and acquires the image. Then, based on the movement or state of the bed 90, the condition determination threshold used to determine the condition of the care recipient 80, as an image analysis method, is changed. The changed condition determination threshold is then used to determine the condition of the care recipient 80, and the determination result is output.

The detection device 10 of this embodiment described above determines the state of the care recipient 80 using the changed state determination threshold, so it can appropriately determine the state of the care recipient 80 from images even if the bed 90 moves, for example, by tilting.

(Fourth embodiment)
In the third embodiment, a case was described in which the state determination threshold used to determine the state of the care recipient 80 is changed as an image analysis method based on the movement or state of the bed 90. In the fourth embodiment, a case will be described in which the posture determination standard used to determine the posture of the care recipient 80 is changed as an image analysis method based on the movement or state of the bed 90. In the following, detailed description of the same configuration as in the first embodiment will be omitted to avoid duplication.

Figure 16 is a schematic diagram illustrating skeletal detection when the bed 90 is in a normal position and the care recipient 80 is not in a fallen position, and Figure 17 is a schematic diagram illustrating skeletal detection when the bed 90 is in a normal position and the care recipient 80 is in a fallen position. Figure 18 is a schematic diagram illustrating skeletal detection when the bed 90 is in a reclining position and the care recipient 80 is not in a fallen position, and Figure 19 is a schematic diagram illustrating skeletal detection when the bed 90 is in a reclining position and the care recipient 80 is in a fallen position.

The posture of the care recipient 80 can be determined using, for example, human posture estimation technology that uses machine learning. Human posture estimation technology, such as Open Pose (see, e.g., https://arxiv.org/abs/1611.08050) and Deep Pose (see, e.g., https://arxiv.org/abs/1312.4659), is a technology that recognizes posture by estimating the position of a person's skeleton and joints from two-dimensional video data. These technologies recognize the skeleton that makes up a person, such as the head, neck, shoulders, arms, waist, and legs, and further obtain coordinate values as position information for the joints, which are the joints and ends of these.

For example, in Figure 16, the thick lines indicate the skeleton 801, and the parts connecting the skeletons and the ends of the skeleton (circled areas in the figure) indicate joint points 802. The head 810 also includes joint points. Note that the skeleton and joint points are displayed in the same way in Figures 17 to 19, so explanations and reference numerals will be omitted. In this embodiment, the positions of the joint points roughly correspond to the anatomical positions of the joints of the human body. The image analysis unit 112 acquires coordinate data for each joint point of an object recognized as a person and analyzes the posture, etc. of the recognized person from the coordinate data.

The processing steps of the control method for the detection device 10 in this embodiment are the same as those in the processing steps of the third embodiment (flowchart in FIG. 13) except for step S304 (i.e., steps S301 to S303, S305, and S306). Therefore, in the following, we will omit the explanation of steps S301 to S303, S305, and S306 in this embodiment and will only explain the processing of step S304.

The image analysis unit 112, for example, changes the posture determination criteria used to determine the posture of the care recipient 80 as an image analysis method. More specifically, as shown in Figures 16 and 17, when the bed 90 is in a normal state, the image analysis unit 112 sets the posture determination criteria to the normal state criteria. The normal state criteria are criteria for determining that an upright posture is not a fall and that a sitting posture is a fall, and are pre-stored in the RAM of the calculation control unit 101. In the example shown in Figure 16, since the bed 90 is in a normal state, the image analysis unit 112 uses the normal state criteria to determine that the care recipient 80 is in an upright posture, for example, based on the positions of the joint points of the head, neck, shoulders, arms, waist, and feet of the care recipient 80. In this case, since the posture is upright, the image analysis unit 112 determines that the care recipient 80 is in a non-fall state. Similarly, in the example shown in FIG. 17, the image analysis unit 112 determines that the care recipient 80 is in a sitting position. In this case, because the position is sitting, the image analysis unit 112 determines that the care recipient 80 has fallen.

On the other hand, when the state of the bed 90 changes from the normal state to the reclined state, the image analysis unit 112 changes the posture determination criterion from the normal state criterion to the reclined state criterion. The reclined state criterion is a criterion for determining that the care recipient 80 is in a fallen state when the care recipient's posture is determined to be standing, and is pre-stored in the RAM of the calculation control unit 101. In the example shown in FIG. 18, the bed 90 is in a reclined state, so the image analysis unit 112 determines the posture of the care recipient 80 using the reclined state criterion. Because the head and neck of the care recipient 80 are outside the monitoring area, the image analysis unit 112 determines the posture of the care recipient 80 based on the positions of the joint points of the shoulders, arms, waist, and feet. In this case, the image analysis unit 112 may determine that the posture is other than standing, but in either posture, it determines that the care recipient 80 is not in a fallen state. In the example shown in FIG. 19, the bed 90 is in a reclined position, so the image analysis unit 112 similarly uses the reclined position standard to determine the posture of the care recipient 80. The image analysis unit 112 determines that the care recipient 80 has fallen because the positions of the joint points of the head, neck, shoulders, arms, waist, and feet of the care recipient 80 are similar in terms of skeletal balance to the posture of the care recipient 80 in a normal standing position.

In this way, in this embodiment, the posture determination criteria used to determine the posture of the care recipient 80 as an image analysis method are changed based on the movement or state of the bed 90. The posture of the care recipient 80 is then determined using the changed posture determination criteria, and the condition of the care recipient 80 is determined based on the determination result.

The detection device 10 of this embodiment described above determines the condition of the care recipient 80 based on the results of the posture determination of the care recipient 80, so it can appropriately determine the condition of the care recipient 80 from images even if the bed 90 moves, for example, by tilting.

Fifth Embodiment
In the first to fourth embodiments, the detection device 10 is installed on the mattress 94. In the fifth embodiment, the detection device 10 is installed on the headboard 92.

Figure 20 is a schematic side view of the bed 90 and detection device 10 in the normal state in the fifth embodiment. Also, Figure 21 is a schematic side view of the bed 90 and detection device 10 when the foot-side mattress 94 is inclined.

As shown in FIG. 20, the detection device 10 is installed, for example, on the headboard 92, and sets the area including the bed 90 as the monitoring area of the camera 103, capturing images of the monitoring area. When the bed 90 is in a normal state, the camera 103 parameters are set to their initial values to capture images of the monitoring area, and the state of the care recipient 80 (fall/not fall) can be determined based on the captured image.

On the other hand, as shown in FIG. 21, for example, if the portion of the mattress 94 corresponding to the waist of the care recipient 80 is tilted, the distance between the camera 103 and the mattress 94 in the monitoring area will be reduced, and the shapes of the person object 800 and bed object 940 will change from their normal shapes. As a result, the image analysis unit 112 may not be able to correctly determine the condition of the care recipient 80.

In this embodiment, therefore, the pan P and tilt T of the camera 103 are changed (controlled) to P2 and T2, respectively, and the optical axis of the camera 103 is set to an angle corresponding to the inclination angle a2 of the mattress 94, thereby making the optical axis of the camera 103 roughly parallel to the foot side of the mattress 94. Furthermore, the magnification of the camera 103 is set to an appropriate value M2 so that the shape of the bed 90 in the captured image is equivalent to the shape of the bed 90 in its normal state. By setting the parameters (P, T, M) in this way and capturing an image within the monitoring area, the condition of the care recipient 80 can be appropriately determined based on the captured image.

Sixth Embodiment
In the sixth embodiment, a case where the detection device 10 is installed on a footboard 93 will be described.

Figure 22 is a schematic side view of the bed 90 and detection device 10 in the normal state according to the sixth embodiment. Also, Figure 23 is a schematic side view of the bed 90 and detection device 10 in the reclining state.

As shown in FIG. 22, the detection device 10 is installed, for example, on the footboard 93, and sets the area including the bed 90 as the monitoring area of the camera 103, capturing images of the monitoring area. When the bed 90 is in a normal state, the camera 103 parameters are set to their initial values to capture images of the monitoring area, and the state of the care recipient 80 (fall/non-fall) can be determined based on the captured image.

On the other hand, as shown in FIG. 23, for example, if the portion of the mattress 94 corresponding to the back of the care recipient 80 is tilted, the distance between the camera 103 and the mattress 94 in the monitoring area will decrease, causing the shapes of the person object 800 and the bed object 940 to change from their normal shapes. As a result, the image analysis unit 112 may not be able to correctly determine the condition of the care recipient 80.

In this embodiment, therefore, the pan P and tilt T of the camera 103 are changed (controlled) to P3 and T3, respectively, and the optical axis of the camera 103 is set to an angle corresponding to the inclination angle a3 of the mattress 94, thereby making the optical axis of the camera 103 roughly parallel to the head end of the mattress 94. Furthermore, the magnification M of the camera 103 is set to an appropriate value M3 so that the shape of the bed 90 in the captured image is equivalent to the shape of the bed 90 in its normal state. By setting the parameters (P, T, M) in this way and capturing an image within the monitoring area, the condition of the care recipient 80 can be appropriately determined based on the captured image.

The configuration of the monitoring system 1 described above is the main configuration used to explain the features of the above-mentioned embodiment, but it is not limited to the above configuration and can be modified in various ways within the scope of the claims.

For example, in the first to sixth embodiments described above, the detection device 10 is installed on the mattress 94, headboard 92, or footboard 93, but the detection device 10 may be installed in a location on the bed 90 other than the mattress 94, headboard 92, or footboard 93. Furthermore, the detection device 10 is not limited to being installed on the upper end of the headboard 92 or footboard 93, and can also be installed, for example, in a plane facing the center of the bed 90.

In addition, while the first to sixth embodiments described above have mainly been described as cases in which the state of the bed 90 changes due to the tilting of the mattress 94, the present invention can also be applied to cases in which the state of the bed 90 changes due to the tilting of the upper frame 911. It can also be applied to cases in which the state of the bed 90 changes due to changes in the bed height. The control unit 114 controls the camera 103 or the image analysis unit 112 in response to changes in the orientation or position of the camera 103.

Furthermore, in the first to sixth embodiments described above, the bed 90 is an electrically operated bed, but the bed 90 may be configured to tilt the mattress 94 or upper frame 911 not only electrically but also hydraulically or by other means.

Furthermore, the above-described flowcharts may include steps other than those shown in the flowcharts, or may not include some steps. The order of the steps is not limited to the above-described embodiment. Furthermore, each step may be combined with other steps and executed as a single step, may be included in other steps and executed, or may be divided into multiple steps and executed.

Furthermore, the means and methods for performing the various processes in the above-described embodiments can be realized by either dedicated hardware circuits or a programmed computer. The above programs may be provided, for example, by computer-readable recording media such as USB memory or DVD (Digital Versatile Disc)-ROM, or may be provided online via a network such as the Internet. In this case, the programs recorded on the computer-readable recording media are typically transferred to and stored in a storage unit such as a hard disk. Furthermore, the above programs may be provided as standalone application software, or may be incorporated as a function into the software of a device such as its detection unit.

1. monitoring system,
10 detection device,
101 arithmetic and control unit,
102 Communications Department,
103 camera,
111 image acquisition unit,
112 image analysis unit,
113 Reception department,
114 control unit,
20 servers,
201 arithmetic and control unit,
202 Communications Department,
203 storage unit,
30 mobile terminal,
31 access points,
90 beds,
91 frame part,
911 upper frame,
912 lower frame,
92 headboard,
93 footboard,
94 mattresses,
95 Sensor section.

Claims (17)

an image acquisition unit that acquires an image captured by an imaging unit installed on the bed;
an image analysis unit that analyzes the image;
A control unit that controls the imaging unit or the image analysis unit based on the operation or state of the bed,
the bed is an electric bed,
The control unit determines the operation or state of the electric bed based on a signal from the electric bed. an image acquisition unit that acquires an image captured by an imaging unit installed on the bed;
an image analysis unit that analyzes the image;
A control unit that controls the imaging unit or the image analysis unit based on the operation or state of the bed,
The image analysis unit detects objects included in the image captured by the imaging unit, compares the position of the top end of a bed object corresponding to the bed in the image with the position of the top end of a person object corresponding to the person in the image, and determines that the person has not fallen if the position of the top end of the person object is higher than the position of the top end of the bed object, but determines that the person has fallen if the position is lower. an image acquisition unit that acquires an image captured by an imaging unit installed on the bed;
an image analysis unit that analyzes the image;
a sensor that detects the operation of the bed, including an operation of tilting a mattress of the bed and an operation of moving the mattress of the bed, or the state of the bed, including a reclining state in which the mattress is tilted relative to a floor surface;
A control unit that controls the imaging unit or the image analysis unit based on the operation or state of the bed,
The control unit is a control device that controls the orientation or position of the imaging unit based on the movement or state of the bed. an image acquisition unit that acquires an image captured by an imaging unit installed on the bed;
an image analysis unit that analyzes the image;
a sensor that detects the operation of the bed, including an operation of tilting a mattress of the bed and an operation of moving the mattress of the bed, or the state of the bed, including a reclining state in which the mattress is tilted relative to a floor surface;
A control unit that controls the imaging unit or the image analysis unit based on the operation or state of the bed;
a reception unit that receives a detection result from the sensor that detects the movement or state of the bed,
The control unit determines the operation or state of the bed based on the detection result. an image acquisition unit that acquires an image captured by an imaging unit installed on the bed;
an image analysis unit that analyzes the image;
a sensor that detects the operation of the bed, including an operation of tilting a mattress of the bed and an operation of moving the mattress of the bed, or the state of the bed, including a state in which the mattress is tilted with respect to a floor surface;
A control device comprising: a control unit that controls the imaging unit based on the operation or state of the bed. the control unit controls the image analysis unit to change a state determination threshold used to determine the state of the care recipient or a posture determination standard used to determine the posture of the care recipient based on the movement or state of the bed ;
the posture determination criteria include a normal state criterion that is set when the bed is in a normal state, and a reclined state criterion that is set when the bed is in a reclined state,
The standard for the normal state is a standard for determining that the care recipient is not in a fallen state when the posture of the care recipient is determined to be standing, and determining that the care recipient is in a fallen state when the posture of the care recipient is determined to be sitting,
A control device as described in any one of claims 1 to 4 , wherein the criterion for the reclining state is a criterion for determining that the person being cared for is in a fallen state when the posture of the person being cared for is determined to be upright . an image acquisition unit that acquires an image captured by an imaging unit installed on the bed;
an image analysis unit that analyzes the image;
A control unit that controls the imaging unit or the image analysis unit based on the operation or state of the bed,
The control unit controls the orientation or position of the imaging unit based on the movement or state of the bed,
The control unit converts the position of each pixel in the image acquired by the image acquisition unit to the position of the image that would be obtained if the bed were in a normal state based on the operation or state of the bed , and controls the image analysis unit to correct the image acquired by the image acquisition unit by performing calculations to interpolate between the pixels. further comprising a storage unit that stores the state determination threshold value;
The control device according to claim 6, wherein the image analysis unit detects an object included in an image captured by the imaging unit, and determines the state of the person based on the result of comparing the position of the top end of a human object corresponding to the person in the image with the state determination threshold. The control device according to claim 8 , wherein the image analysis unit changes the state determination threshold value based on the movement or state of the bed. a storage unit that stores the posture determination criteria;
7. The control device according to claim 6, wherein the image analysis unit detects a skeleton and joint points of a human object corresponding to a person included in an image captured by the imaging unit, and determines a posture of the person and a state of the person corresponding to the posture based on the detected skeleton and joint points and the posture determination criterion. The control device according to claim 10 , wherein the image analysis unit changes the posture determination criterion based on a movement or state of the bed. an imaging step of imaging a subject including the electric bed by an imaging unit installed in the electric bed;
an acquisition step of acquiring the image captured by the imaging unit by an image acquisition unit;
A control method comprising a control step of determining the operation or state of the electric bed based on a signal from the electric bed, and controlling an image analysis unit that analyzes the image based on the operation or state of the electric bed. an imaging step of imaging an object including the bed by an imaging unit installed on the bed;
an acquisition step of acquiring the image captured by the imaging unit by an image acquisition unit;
A control step of controlling an image analysis unit that analyzes the image based on the operation or state of the bed,
In the control step, an object included in the image captured by the imaging unit is detected, and the position of the top end of a bed object corresponding to the bed in the image is compared with the position of the top end of a person object corresponding to the person in the image.If the position of the top end of the person object is higher than the position of the top end of the bed object, it is determined that the person has not fallen, but if it is lower, it is determined that the person has fallen. a detection step of detecting a bed operation including an operation of tilting a mattress of the bed and an operation of moving the mattress of the bed, or a state of the bed including a reclining state in which the mattress is tilted with respect to a floor surface;
a control step of controlling the imaging unit based on an operation or a state of the bed on which the imaging unit is installed;
an imaging step of imaging an object including the bed by the imaging unit;
an acquisition step of acquiring the image captured by the imaging unit by an image acquisition unit;
and analyzing the image. an imaging step of imaging an object including the bed by an imaging unit installed on the bed;
an acquisition step of acquiring the image captured by the imaging unit by an image acquisition unit;
a detection step of detecting, by a sensor, the operation of the bed, including the operation of tilting the mattress of the bed and the operation of moving the mattress of the bed, or the state of the bed, including a reclining state in which the mattress is tilted with respect to a floor surface;
receiving a detection result from the sensor that detects the movement or state of the bed;
A control step of determining the movement or state of the bed based on the detection result and controlling an image analysis unit that analyzes the image based on the movement or state of the bed;
A control method comprising: A control program for causing a computer to execute the control method described in any one of claims 12 to 15. An imaging unit installed on the bed,
A control device according to any one of claims 1 to 11;
An information processing system having: