WO2025253795A1 - Imaging device, imaging system, imaging method, and storage medium - Google Patents

Abstract

This imaging device comprises: an imaging unit that acquires video information including a plurality of subjects in a video display area; and a storage unit that identifies one first subject from the plurality of subjects included in the video information, detects a second subject closest to the first subject from among the plurality of subjects, and stores tracking control information for tracking the first subject and the second subject to acquire the video information.

Description

Imaging device, imaging system, imaging method, and storage medium

This disclosure relates to an imaging device, an imaging system, an imaging method, and a storage medium.

Patent Literature 1 discloses an information processing device that includes an image control unit that generates a virtual camera image to be used in an output image based on an image captured by a camera, and controls updating of the composition of the virtual camera image based on determination information related to a subject in the captured image or the output image.
According to the information processing device configured in this way, it is possible to realize the production of video content that maintains the quality of the content while automatically determining the composition of the camera image.

International Publication Number: WO 2023/074355 A1

In the above-mentioned information processing device, when, for example, people participating in an event are taken as subjects and multiple subjects are photographed with a camera to produce video content, no consideration is given to which subjects to track and which video to produce.
Therefore, there has been a demand for the development of an imaging device, imaging system, and imaging method that can produce a captured video while tracking the appropriate subject when multiple subjects are present. Furthermore, there has been a demand for the development of tracking control information (tracking control program) that can realize such an imaging device, imaging system, and imaging method, and a storage medium that non-temporarily stores this tracking control information.

The imaging device according to the first embodiment of the present disclosure includes an imaging unit that acquires video information including multiple subjects within a video display area, and a memory unit that stores tracking control information that identifies a first subject from the multiple subjects included in the video information, detects a second subject from the multiple subjects that is closest to the first subject, and tracks the first and second subjects to acquire video information.

An imaging system according to a second embodiment of the present disclosure includes an imaging unit that acquires video information including multiple subjects within an image display area; a display unit that displays the video information; a memory unit that stores tracking control information that identifies a first subject from the multiple subjects included in the video information, detects a second subject from the multiple subjects that is closest to the first subject, and tracks the first and second subjects to acquire video information; a control unit that controls the operation of the imaging unit, acquires video information based on the tracking control information stored in the memory unit, and displays the video information on a display device; and an operation unit that operates the control unit.

An imaging method according to a third embodiment of the present disclosure acquires video information containing multiple subjects within an image display area, identifies a first subject from the multiple subjects included in the video information, detects a second subject from the multiple subjects that is closest to the first subject, and tracks the first and second subjects to acquire video information.

A storage medium according to a fourth embodiment of the present disclosure non-temporarily stores tracking control information that causes a computer to execute each of the following steps: acquire video information containing multiple subjects within a video display area; identify a first subject from the multiple subjects included in the video information; detect a second subject from the multiple subjects that is closest to the first subject; and track the first and second subjects to acquire video information.

FIG. 1 is a system configuration diagram showing the overall configuration of an imaging device and an imaging system according to a first embodiment of the present invention. FIG. 2 is a perspective view of the appearance of the imaging device shown in FIG. 1 as seen from the front side. FIG. 3 is a perspective view of the rear surface of the imaging device shown in FIG. FIG. 4 is a block diagram illustrating the functions of the imaging device shown in FIGS. FIG. 5 is a diagram showing, in table form, a list of setting values for the PTZ auto-framing function of the image pickup apparatus shown in FIG. FIG. 6 is a diagram showing a display screen displayed on the display of a personal computer that constitutes the imaging system. FIG. 7 is a diagram of a display screen when the number of subjects to be tracked and video information to be acquired is set to two in the video display area of the display screen shown in FIG. FIG. 8 is a diagram of a display screen when the number of subjects to be tracked and video information to be acquired is set to three in the video display area of the display screen shown in FIG. FIG. 9 is a first display screen diagram for explaining, in a step-by-step manner, the procedure for increasing the number of subjects in the video display area of the display screen shown in FIG. FIG. 10 is a diagram of a second display screen for explaining the procedure for increasing the number of subjects step by step. FIG. 11 is a diagram of a third display screen for explaining the procedure for increasing the number of subjects in a step-by-step manner. FIG. 12 is a first display screen diagram for explaining, in a step-by-step manner, the procedure when the number of subjects cannot be increased in the video display area of the display screen shown in FIG. FIG. 13 is a diagram of a second display screen for explaining, in stages, the procedure when the number of subjects cannot be increased. FIG. 14 is a diagram of a third display screen for explaining, in stages, the procedure to be followed when the number of subjects cannot be increased. FIG. 15 is a first display screen diagram for explaining, in a step-by-step manner, the procedure for reducing the number of subjects in the video display area of the display screen shown in FIG. FIG. 16 is a diagram of a second display screen for explaining the procedure for reducing the number of subjects in a step-by-step manner. FIG. 17 is a first display screen diagram for explaining, in a step-by-step manner, the procedure when it is not possible to reduce the number of subjects in the video display area of the display screen shown in FIG. FIG. 18 is a diagram of a second display screen for explaining, in stages, the procedure to be followed when the number of subjects cannot be reduced. FIG. 19 is a diagram of a third display screen for explaining, in stages, the procedure to be followed when the number of subjects cannot be reduced. FIG. 20 is a first display screen diagram for explaining, in a step-by-step manner, the procedure when tracking of the first subject is lost in the video display area of the display screen shown in FIG. FIG. 21 is a diagram of a second display screen for explaining, in stages, the procedure to be followed when tracking of the first subject is lost. FIG. 22 is a diagram of a third display screen for explaining, in stages, the procedure to be followed when tracking of the first subject is lost. FIG. 23 is a first display screen diagram for explaining, in a step-by-step manner, the procedure when tracking of the second subject is lost in the video display area of the display screen shown in FIG. FIG. 24 is a diagram of a second display screen for explaining, in stages, the procedure to be followed when tracking of the second subject is lost. FIG. 25 is a diagram of a third display screen for explaining, in stages, the procedure to be followed when tracking of the second subject is lost. FIG. 26 is a fourth display screen diagram for explaining, in stages, the procedure to be followed when tracking of the second subject is lost. FIG. 27 is a diagram of a display screen corresponding to FIG. 6, for setting a fail-safe during automatic selection. FIG. 28 is a table showing a list of setting items for setting fail-safes when automatically selected. FIG. 29A is a diagram showing the shape of the display frame of the first subject. FIG. 29B is a diagram showing the shape of the display frame for the second subject. FIG. 29C is a diagram showing the shape of the display frames for a plurality of subjects. FIG. 29D is a diagram showing the shape of the display frame of the video display area. FIG. 30A is a diagram showing an example in which the display frame of the first subject extends beyond the video display area. FIG. 30B is a diagram showing another example in which the display frame of the first subject extends beyond the video display area. FIG. 31 is a flowchart illustrating a method for capturing an image of a single subject. FIG. 32 is a flowchart illustrating a method for capturing images of multiple subjects. FIG. 33 is a diagram showing a list of display items of the tracking status in a table format. FIG. 34 is a diagram showing, in a table format, the relationship between the tally control and the display items of the tracking status. FIG. 35 is an external configuration diagram of an operation terminal that constitutes an imaging system according to the second embodiment of the present disclosure. FIG. 36 is a diagram showing a display screen in the multi-person mode on the operation terminal shown in FIG. FIG. 36 is a diagram showing a display screen in the non-multiple-person mode by the operation terminal shown in FIG.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. The description will be made in the following order.
1. First Embodiment In the first embodiment, a first example in which the present technology is applied to an imaging device, an imaging system, an imaging method, and a storage medium that stores tracking control information (tracking control program) will be described.
In the first embodiment, the imaging system includes a user imaging system including an imaging device, and a cloud imaging system connected to the user imaging system via an internet line.
2. Second Embodiment In the second embodiment, an example will be described in which the configuration of the operation terminal is changed in the imaging device, imaging system, imaging method, and storage medium that stores tracking control information (tracking control program) according to the first embodiment.
3. Other embodiments

1. First embodiment
1 to 34, the imaging device 1 according to the first embodiment of the present disclosure, the user imaging system 100, the cloud imaging system 200, the imaging method, and the storage medium 106 that stores tracking control information will be described.

[Overall configuration of cloud imaging system 200]
FIG. 1 shows an example of the overall system configuration of a user imaging system 100 and a cloud imaging system 200 including an imaging device 1 according to the first embodiment.
1, the cloud imaging system 200 according to the first embodiment is constructed with a user imaging system 100 including an imaging device 1, an operation terminal 2, and a personal computer 3, and a server 120 connected via an internet line 110. The cloud imaging system 200 is also constructed to include a mobile terminal 130 such as a tablet or smartphone.
Here, the user imaging system 100 and the cloud imaging system 200 each correspond to the "imaging system" according to the present technology.

In the cloud imaging system 200, video content can be produced based on video information captured using the imaging device 1 at various events. Examples of various events include Zoom conferences, weddings, receptions, parties, sports, music, and theater. The video content can be stored in one or more of the memory unit 1054 of the imaging device 1, the memory unit of the personal computer 3 (not shown), the memory unit of the server 120 (not shown), and the memory unit of the mobile terminal 130 (not shown). Furthermore, the video content can also be live streamed or broadcast from the user imaging system 100 to the mobile terminal 130 or external devices (not shown) via the Internet line 110.

[Configuration of user imaging system 100]
(1) External Configuration of Imaging Device 1 Fig. 2 shows an example of the external configuration of the imaging device 1 as viewed from the front side. Fig. 3 shows an example of the external configuration of the imaging device 1 as viewed from the back side.
As shown in FIGS. 2 and 3, the imaging device 1 of the user imaging system 100 is a camera that has, for example, a pan tilt zoom auto framing (PTZ) function, a remote control function, and is capable of capturing still images and videos.
Furthermore, the PTZ auto-framing function is a function that automatically tracks a subject by activating the PTZ function of the imaging device 1. The PTZ auto-framing function has a function of moving the orientation of the camera lens 12 in the left-right direction X (pan), a function of moving the orientation of the camera lens 12 in the up-down direction Y (tilt), and a function of zooming in and out of the imaging range (zoom). In other words, the imaging device 1 is also called a panning camera.

As shown in FIG. 2, the imaging device 1 comprises a base body 10 and a camera body 11. On the front of the base body 10 are a power indicator lamp 101 that indicates whether the power is on or not, and an internet indicator lamp 102 that indicates whether the device is connected to the internet line 110.

The camera body 11 is disposed on the base body 10 and configured to be movable in the left-right direction X and the up-down direction Y. The camera body 11 also has a camera lens 12 disposed thereon.
An imaging unit 111 (see FIG. 4) connected to the camera lens 12 is disposed inside the camera body 11. The imaging unit 111 is configured with an image sensor such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor.

As shown in Figure 3, an input/output (interface) unit 105 is provided on the back surface of the base body 10 of the imaging device 1. Although detailed illustrations and explanations are omitted, the input/output unit 105 is provided with multiple connection terminals such as a power connection terminal, an internet connection terminal, an audio input/output terminal, and a display connection terminal. The internet connection terminals include connection terminals to one or more lines selected from an internet line 110, a home network line, a LAN (Local Area Network) line, a satellite communication network, a telephone line, and a fifth-generation mobile communication system (5G) line.

(2) Functional Configuration of the Imaging Device 1 FIG. 4 shows an example of a block configuration for explaining the functions of the imaging device 1.
4 , the imaging device 1 includes an imaging unit 111, a drive unit 112, a central processing unit (CPU) 113, an artificial intelligence (AI) processor 114, a storage unit 115, and an input/output unit 105. In this example, the storage unit 115 includes a read-only memory (ROM) 1151 and a random access memory (RAM) 1152. The imaging unit 111 and other units are connected to each other via a common bus 117.

Furthermore, the imaging device 1 includes an input unit 1051, a display unit 1052, an audio output unit 1053, a storage unit 1054, a communication unit 1055, and a drive unit 1056, which are each connected to the input/output unit 105.

The image capturing unit 111 acquires image information captured through the camera lens 12. This image information includes one or more subjects. In the first embodiment, the subjects are people.
In the present technology, the subject is not limited to a person. For example, the subject may be a living organism such as an animal, bird, fish, insect, flower, tree, or leaf. The subject may also be a car, a motorcycle, an airplane, a ship, or the like.
The driving unit 112 drives the imaging unit 111 to realize a PTZ auto-framing function.

The CPU 113 executes various processes, such as acquiring video information, based on control information (control program) stored in the ROM 1151 or control information stored in the storage unit 1054 and loaded into the RAM 1152. Furthermore, when the CPU 113 executes various processes, the control information, etc. is stored in the RAM 1152 from the storage unit 1054, etc., as appropriate.

In the first embodiment, the imaging device 1 is configured to include an AI processor 114. By including the AI processor 114, the imaging device 1 can realize advanced determination processing, recognition processing, and the like.
For example, the AI processor 114 performs processes such as detection, extraction, face recognition, and personal identification of people who are subjects included in the video information. The AI processor 114 also performs various determination processes regarding the individuals of the detected people, selection processes for subjects to be photographed based on the determination processes, composition setting processes, and the like.

The imaging device 1 may also include an image processing unit (GPU: Graphics Processing Unit), GPU general-purpose computing on graphics processing units (GPGPU), etc.

The input unit 1051 connected to the input/output unit 105 is, for example, various types of operators or operation devices such as a keyboard, a mouse, keys, a dial, a touch panel, a touch pad, a remote controller, etc. When a user operation is detected from the input/output unit 105, a signal corresponding to the input operation is interpreted by the CPU 113.
In the first embodiment, the input/output unit 105 corresponds to, for example, the touch pad 31, mouse 32, and keyboard 33 of the operation terminal 2 or the personal computer 3 (see FIG. 1).

The display unit 1052 is, for example, an external display. Examples of external displays include a liquid crystal display (LCD) and an organic electroluminescence (EL) display.
Various displays are performed on the display unit 1052. Specifically, the display unit 1052 displays images for various image processing, images to be processed, captured video, etc. on the display screen based on instructions from the CPU 113. Furthermore, the display unit 1052 displays various operation menus, icons, messages, etc. as a graphical user interface (GUI) based on instructions from the CPU 113.
In the first embodiment, the display unit 1052 is connected to an external display (not shown). When the display 34 of the personal computer 3 is used as a display device, the imaging device 1 is connected to the personal computer 3 via the communication unit 1055 and further via the Internet line 110 (see FIG. 1).

The audio output unit 1053 is, for example, a speaker. The audio output unit 1053 corresponds to, for example, a speaker (not shown) of the personal computer 3.

A large-capacity memory such as a hard disk or solid-state memory is used for the storage unit 1054. The storage unit 1054 stores video information including filmed or produced video content, management information, etc. The storage unit 1054 also stores various types of control information (control programs).
In the first embodiment, the storage unit 1054 corresponds to, for example, an internal memory (not shown) of the personal computer 3 .

The communication unit 1055 executes communication processing via a transmission path such as the Internet line 110. The communication method is not particularly limited. As long as it has similar functions, one or more communication methods selected from a wired communication method, a wireless communication method, and a bus communication method can be adopted.
As for wireless communication methods, short-range wireless communication methods such as Bluetooth (registered trademark), Wi-Fi (registered trademark) (Wireless Fidelity (registered trademark)), and NFC (registered trademark) (Near Field Communication (registered trademark)) or infrared communication methods can be practically used.
The communication unit 1055 also has a function of distributing video content to destination terminals, such as the server 120 and the mobile terminal 130 .

The storage medium 106 can be attached to the drive unit 1056. For example, various information stored in the storage medium 106 can be stored in the storage unit 1054 of the imaging device 1 through the drive unit 1056. Conversely, various information stored in the storage unit 1054 of the imaging device 1 can be stored in the storage medium 106 through the drive unit 1056.
The storage medium 106 is, for example, a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like, and is portable.

(3) Configuration of Operation Terminal 2 Returning to Fig. 1, the operation terminal 2 is a terminal that remotely controls the imaging device 1. Simply put, the operation terminal 2 is provided on its top surface with an operation lever 21 that controls the direction of the camera lens 12 of the camera body 11, an operation unit 22 that controls the acquisition of video information by the imaging device 1, and a display unit 23 that displays various operating states.
Further, on the rear side of the operation terminal 2, an input/output unit (not shown) that can be connected to one or more selected from the imaging device 1, the personal computer 3, and the Internet line 110 is arranged.

(4) Configuration of Personal Computer 3 As shown in Fig. 1, the personal computer 3 includes at least a touchpad 31, a mouse 32, a keyboard 33, and a display 34, as described above. The basic functions of the personal computer 3 are essentially the same as those of the imaging device 1 shown in Fig. 4. Note that the personal computer 3 may include an imaging unit (camera) capable of capturing still images and videos. Generally, this imaging unit does not include a PTZ auto-framing function.

[Configuration of cloud imaging system 200]
(1) Configuration of Server 120 As shown in Fig. 1, in the first embodiment, video information acquired by the imaging device 1 or the user imaging system 100 is transmitted to the server 120 via the Internet line 110. The server 120 receives the transmitted video information. Of course, the server 120 may transmit various information, such as imaging control information, to the user imaging system 100.
The server 120 may be located, for example, at or near the event venue where the photography is performed, or may be located in a remote location away from the event venue. When the server 120 is located in a remote location, the operation terminal 2 that constitutes the user photography system 100 may also be located in a remote location like the server 120.

The basic functions of the server 120 are essentially the same as those of the imaging device 1 shown in Figure 4. Note that the server 120 generally does not have an imaging unit (camera) capable of capturing still images or videos.

(2) Configuration of the Portable Terminal 130 As shown in FIG. 1, the portable terminal 130 can be connected to the user imaging system 100 via an Internet line 110 .
The basic functions of the mobile terminal 130 are basically the same as those of the imaging device 1 or the personal computer 3 shown in FIG.

[Image capture method and tracking control information]
(1) Setting of the PTZ auto-framing function FIG. 5 shows an example of setting values of the PTZ auto-framing function.
The imaging method and tracking control information applied to the imaging device 1 according to the first embodiment are created based on the setting values of the PTZ auto-framing function shown in Fig. 5. The setting items and their contents are as follows:
In the first embodiment, the subject is set to a person.

1. Automatic Execution of Tracking Function Automatic execution of the tracking function can be used for single or multiple subjects (can be used in common). Automatic execution of the tracking function is a setting for whether or not to automatically transition the tracking function to an execution state when the set main body of the user imaging system 100 including the imaging device 1 is started.
Automatic execution of the tracking function is set, for example, by an ON/OFF button displayed on the display unit 23 of the operation terminal 2 or the display 34 of the personal computer 3. The default value is OFF. It cannot be turned ON until the initial setting is completed.

2. Tally Control Tally control can be used for single and multiple subjects (can be used for both). Tally control is a setting for whether or not to perform tally control depending on the tracking status. Tally control is set using the ON/OFF button. The default setting is ON.

3. Face frame superimposition during application execution Face frame superimposition during application execution can be used for single or multiple subjects (can be used in common). Face frame superimposition during application execution is, for example, a setting for face frame superimposition on Web User Interface (WebUI) streaming. Face frame superimposition during application execution is set using the ON/OFF button. The default setting is ON.

4. Tracking Composition Settings Tracking composition settings include "tracking target size,""verticalplacement," and "horizontal placement."

5. Tracking start position setting The tracking start position setting can be used for single and multiple subjects (can be used in common). The tracking start position setting is the position setting when the shooting method application starts to run. The tracking start position setting is set separately from the preset position on the set main body.

6. Setting the subject detection area The subject detection area setting can be used for both single and multiple subjects (can be used for both). The subject detection area setting specifies the subject detection area within the tracking start position. The default value for the subject detection area setting is the center of the display screen.

7. Tracking target selection Tracking target selection can be used for single and multiple subjects (can be used for both). Tracking target selection is a setting that specifies the selection of the subject to be tracked automatically or manually. The default value for tracking target selection is automatic selection.

8. Personal Identification Personal identification can be used for both single and multiple subjects (can be used in common). Personal identification is a setting for whether or not to enable personal identification (temporary face registration: a function that temporarily stores information about a specific subject, thereby identifying whether the subject is the same during tracking and identifying them as the subject) when manually selected. Personal identification is set using the ON/OFF button. The default setting is ON.

9. Face Registration Face registration can be used for single and multiple subjects (can be used for both). Face registration is a setting for whether to use advance face registration (a function that allows you to store the faces of the targets to be tracked in advance along with their priority). Face registration is set using the ON/OFF button. The default setting is OFF.

10. Tracking timeout time The tracking timeout time can be used for single and multiple subjects (can be used in common). The tracking timeout time is determined when the face disappears from view while tracking the tracking target and the specified timeout time has elapsed. The tracking timeout time can be set, for example, between 0 and 60 seconds. The default value is 0 seconds. When set to the default value, tracking timeout does not occur.

11. Tracking forced reset time The tracking forced reset time can be used for single and multiple subjects (can be used in common). The tracking forced reset time is a setting that specifies the time after which the subject transitions to the "tracking" state and then the state is forcibly transitioned to "searching for target" after the specified time has elapsed. The tracking forced reset time can be set, for example, between 0 and 600 seconds. The default value is 0 seconds. When set to the default value, no forced reset occurs.

12. Tracking Lost Waiting Time The tracking lost waiting time can be used for single and multiple subjects (can be used in common). The tracking lost waiting time is a setting for the time from when the tracking target subject is lost until it is determined that the subject has been lost. The tracking lost waiting time can be set, for example, between 1 second and 60 seconds. The initial value is, for example, 6 seconds.

13. Pan/Tilt Movable Range Limit The pan/tilt movable range limit can be used for single or multiple subjects (can be used commonly). The pan/tilt movable range limit is a setting only for the pan/tilt movable range limit function of the camera body 11.

14. Multiple Person Mode Multiple person mode can be used when there are multiple people, i.e., multiple subjects. Multiple person mode is a setting for whether to use multiple person mode. Multiple person mode is set using the ON/OFF button. The default setting is OFF.

15. Specifying the number of people to track Specifying the number of people to track can be used when multiple people are being tracked. Specifying the number of people to track sets the number of people to track in multi-person mode. The number of people to track can be set, for example, between 2 and 8 people. The default value is 2 people.

16. Number of people waiting time The number of people waiting time can be used when there are multiple people. The number of people waiting time is a setting for whether tracking will start even if the number of detected people does not meet the specified number of people to track. The number of people waiting time can be set, for example, between 1 second and 60 seconds. The initial value is, for example, 0 seconds. 0 seconds is invalid, and tracking will not start until the specified number of people is reached.

(2) At the Start of the Imaging Method In the first embodiment, tracking control information (tracking control program) for realizing the imaging method is stored in the ROM 1151 of the storage unit 115 of the imaging device 1 .
The tracking control information may be stored in the storage medium 106 and may be imported into the imaging device 1 from the storage medium 106 via the drive unit 1056. The tracking control information may also be stored in a storage unit (not shown) of the personal computer 3 and imported from this storage unit into the imaging device 1. The tracking control information may also be stored in a storage unit (not shown) of the server 120 and imported from this storage unit into the imaging device 1.

FIG. 6 shows an example of a display screen displayed on the display 34 of the personal computer 3 (see FIG. 1).
In the user imaging system 100, video information captured by the imaging device 1 is displayed on the display 34 of the personal computer 3. As shown in Fig. 6, the display 34 displays a video display area 341 that displays the video information and a user interface area 342 that executes settings for the PTZ auto-framing function. Furthermore, a subject detection area 343 that tracks multiple subjects H is displayed within the video display area 341.

Fig. 7 shows an example of a display screen when the number of subjects H to be tracked and video information to be acquired is set to 2 in the subject detection area 343 of the display screen. Fig. 8 shows an example of a display screen when the number of subjects H to be tracked and video information to be acquired is set to 3 in the subject detection area 343 of the display screen.
First, the number of subjects H to be tracked is set in the user interface area 342 shown in Fig. 6. Here, the number of subjects H is set to two. Then, of the subjects H, a first subject H1 that is the main subject is identified.
As a result, as shown in FIG. 7, the second subject H2, which plays a supporting role together with the first subject H1, is identified and tracked, and shooting begins, with the first subject H1 at the center.

Here, the faces of a plurality of people are selected as the plurality of subjects H.
The first subject H1 is selected and identified by one or more of the following selection methods.
1. Manual selection method: The user selects the face of the subject H that they wish to track by tapping.
2. Pre-registration method: Selection is made based on face information and priority registered before tracking.
3. Automatic selection method: The AI randomly selects the first face it detects within the subject detection area 343.
This selection method is the same as the method for tracking one subject H.

In contrast to the first subject H1, the second subject H2 is detected only by detecting the entire person or a part of the person's face. Neither the pre-registration method nor the manual selection method is implemented. Multiple people can be set as the second subject H2.

6, the number of subjects H to be tracked is set. Here, the number of subjects H is set to three. Then, of the subjects H, a first subject H1 that is the main subject is identified.
As a result, as shown in Figure 8, with the first subject H1 at the center, the second subjects H21 and H22, which are supporting subjects in order of proximity to the first subject H1, are identified and tracked, and shooting begins.

In other words, the number of people to be tracked from multiple subjects H is set, and once the first subject H1 is identified, the first subject H1, second subjects H21 and second subjects H22 are detected and tracked in descending order of distance from the first subject H1, for the set number of people.

(3) Procedure for Increasing the Number of Subjects H to Be Tracked FIGS. 9 to 11 show an example of a display screen that explains, in a step-by-step manner, the procedure for increasing the number of subjects H in the video display area 341 of the display screen.
The number of subjects H being tracked can be changed manually. The AI does not automatically change the number of subjects.

The procedure for increasing the number of subjects H being tracked from, for example, two to three is as follows.
First, as shown in FIG. 9, a first subject H1 and a second subject H21 are photographed while being tracked within the image display area 341.
When the number of people is increased, the image display area 341 is zoomed out while tracking the first subject H1, as shown in Fig. 10. By zooming out, the subject H first detected in the image display area 341 is detected as the second subject H22, and the second subject H22 becomes the tracking target.
As shown in FIG. 11, the first subject H1, the second subject H21, and the second subject H22 are all tracked, and the angle of view of the display screen is adjusted so that video information can be acquired.

On the other hand, when the number of subjects H being tracked cannot be increased, the procedure is as follows.
12 to 14 show examples of display screens that explain, in a step-by-step manner, the procedure to be followed when the number of subjects H cannot be increased in the video display area 341 of the display screen.
First, as shown in FIG. 12, a first subject H1 and a second subject H21 are photographed while being tracked within the image display area 341.
To increase the number of people, the image display area 341 is zoomed out while tracking the first subject H1, as shown in Fig. 13. If the subject H is not detected within the image display area 341 even after zooming out, the first subject H1 and second subject H21 currently being tracked continue to be tracked after the set number of people waiting time has elapsed.
As shown in FIG. 14, the angle of view of the display screen is adjusted so that the first subject H1 and the second subject H21 are tracked together and video information can be acquired.

(4) Procedure for Reducing the Number of Subjects H to Be Tracked FIGS. 15 and 16 show an example of a display screen that explains, in a step-by-step manner, the procedure for reducing the number of subjects H in the video display area 341 of the display screen.
The number of subjects H being tracked can be changed manually. The AI does not automatically change the number of subjects.

The procedure for reducing the number of subjects H being tracked, for example, from three to two, is as follows.
First, as shown in FIG. 15, in the image display area 341, a first subject H1, a second subject H21, and a second subject H22 are photographed while being tracked.
When the number of people is reduced, the image display area 341 is zoomed in while tracking the first subject H1, as shown in Fig. 16. By zooming in, the second subject H22, which has moved out of the image display area 341, is excluded from the tracking target.
The first subject H1 and the second subject H21 are also tracked, and the angle of view of the display screen is adjusted so that video information can be acquired as needed.

On the other hand, when the number of subjects H being tracked cannot be reduced, the procedure is as follows.
17 to 19 show examples of display screens that explain, in a step-by-step manner, the procedure to be followed when the number of subjects H cannot be reduced in the video display area 341 of the display screen.
First, as shown in FIG. 17, a first subject H1 and a second subject H21 are photographed while being tracked within the video display area 341.
To reduce the number of people, the image display area 341 is zoomed in to the limit of analysis performance while tracking the first subject H1, as shown in Fig. 18. If a supporting subject H is not detected in the image display area 341 even after zooming in, the first subject H1 currently being tracked continues to be the tracking target after the set number of people waiting time has elapsed.
As shown in FIG. 19, the first subject H1 is tracked, and the angle of view of the display screen is adjusted so that video information can be acquired as needed.

(5) Procedure when tracking of the first subject H1 is lost (an event in which the system determines that the subject being tracked has been lost) Figures 20 to 22 show an example of a display screen in the video display area 341 of the display screen that explains the procedure to be followed when tracking of the first subject H1 is lost in stages.

The procedure when tracking of the first subject H1 is lost is as follows.
As shown in FIG. 20, within the video display area 341, a first subject H1, a second subject H21, and a second subject H22 are photographed while being tracked.
As shown in FIG. 21, the first subject H1 moves out of the image display area 341, and tracking of the first subject H1 is lost.
As shown in FIG. 22, after the set time of the tracking-lost waiting time has elapsed, the camera returns to the tracking start position, and a plurality of subjects H within the subject detection area 343 are detected.

On the other hand, the procedure when tracking of the second subject H22 is lost is as follows.
23 to 26 show examples of display screens in the video display area 341 of the display screen that explain, in stages, the procedure to be followed when tracking of the second subject H22 is lost.
As shown in FIG. 23, within the video display area 341, a first subject H1, a second subject H21, and a second subject H22 are photographed while being tracked.
As shown in FIG. 24, the second subject H22 moves out of the image display area 341, and tracking of the second subject H22 is lost.
As shown in FIG. 25, the image display area 341 is zoomed out while tracking the first subject H1.
26, after the set time for the number of people waiting time has elapsed, the first subject H1 and second subject H21 detected in the video display area 341 are set as tracking targets. The first subject H1 and second subject H21 are tracked together, and the angle of view of the display screen is adjusted as necessary so that video information can be acquired.

(6) Fail Safe during automatic selection
FIG. 27 shows an example of a display screen for setting the fail-safe during automatic selection.
27, setting items for fail-safe during automatic selection are displayed in a user interface area 342 displayed on the display 34. By setting these setting items, fail-safe during automatic selection is realized.

FIG. 28 shows a list of exemplary setting items for setting fail-safe during automatic selection.
The setting items are as follows:
1. Detection Area The detection area is a setting item in which the number of subjects H within the subject detection area 343 is set, and tracking starts when the set number of subjects is detected.

2. Tracking Timeout Time The tracking timeout time is a setting item for the set time during which, even if tracking is in progress, if there is a subject H among multiple subjects H that is not facing the camera lens 12, that subject H is excluded from the tracking target.

3. Tracking Forced Reset Time The tracking forced reset time is a setting item for the set time to forcibly restart even during tracking.

4. Tracking Lost Waiting Time The tracking lost waiting time is a setting item for setting the time during which tracking continues without restarting even if the tracking target is lost and the number of detected people falls below the specified number of people to be tracked.

5. Number of People Waiting Time The number of people waiting time is a setting item for setting the time for tracking to start even if the number of people detected is below the specified number of people to track.

(7) Display Frame for First Subject H1, etc. Fig. 29A shows an example of the display frame for the first subject H1. Fig. 29B shows an example of the display frame for the second subject H2. Fig. 29C shows an example of the display frame for multiple subjects H. Fig. 29D shows an example of the display frame for the subject detection area 343.

29A , the display frame of the first subject H1 is formed by, for example, combining a rectangular frame with two lines extending along two opposing sides. The horizontal dimension of the rectangular frame is set to a, and the vertical dimension is set to b.
In this example, the horizontal dimension a and the vertical dimension b are set to the same dimensions, so the rectangular frame is formed as a square frame. The horizontal dimension a and the vertical dimension b can be changed as appropriate depending on the face size of the person.

The distance between the rectangular frame and each of the two lines is set to c. Dimension c is adjusted appropriately depending on the image size of the video displayed on the web user interface. For example, when the resolution is 640 x 360 (VGA) pixels, dimension c is set to 2 pixels. Also, when the resolution is 1280 x 720 (HD) pixels, dimension c is set to 6 pixels.

As shown in Figure 29B, the display frame of the second subject H2 is formed by a rectangular frame. To make it easier to distinguish, the display frame of the second subject H2 has a different color or line type than the display frame of the first subject H1.

As shown in Figure 29C, the display frames for multiple subjects H or detection targets are formed by rectangular frames. To make them distinguishable, the display frames for the first subject H1 and the second subject H2 are each displayed in a different color or line type.

As shown in Figure 29D, the display frame of the subject detection area 343 is formed by a rectangular frame. To make them distinguishable, the display frame of the subject detection area 343 is different in color or line type for each of the display frames for the first subject H1, the second subject H2, and the subject H.

The display frame for the first subject H1, the display frame for the second subject H2, the display frame for subject H, and the display frame for the subject detection area 343 are each formed as a separate layer from the display layer for the video information, and are displayed superimposed on the display layer. The display frame for the first subject H1, the display frame for the second subject H2, the display frame for subject H, and the display frame for the subject detection area 343 are each arranged in this order from top to bottom.

30A shows an example in which the display frame of the first subject H1 extends beyond the image display area 341. FIG. 30B shows another example in which the display frame of the first subject H1 extends beyond the image display area 341.
As shown in FIG. 30A, when two sides of the display frame of the first subject H1 extend beyond the video display area 341, the portion of the first subject H1 that does not extend beyond the display frame is displayed.
Furthermore, as shown in FIG. 30B, when one side of the display frame of the first subject H1 extends beyond the video display area 341, the portion of the first subject H1 that does not extend beyond the display frame is displayed.

Similarly, the display frames for the second subject H2 and subject H are each displayed so that the portions that do not extend beyond the video display area 341 are displayed.

[Method of capturing an image of a single subject H and tracking control information]
FIG. 31 shows an example of a flowchart illustrating a method for capturing an image of a single subject H.
As shown in FIG. 31, the method for capturing an image of a single subject H is as follows.

First, the camera lens 12 moves to a tracking start position where video information including a single subject H is captured (step S11).
When the movement of the camera lens 12 is completed and automatic detection is set to OFF, the process waits for the selection of a tracking target (step S12).

When the tracking target is clicked and selected, subject H is now being manually tracked (step S13). Furthermore, while in the tracking (manual) state, neither a forced tracking reset nor a tracking timeout occurs. Furthermore, the selection of a tracking target can be performed repeatedly.

If the tracking target is lost, the tracking is lost state (step S14). If the tracking target is redetected, the process returns to step S13.

On the other hand, when the movement of the camera lens 12 is completed and automatic detection is set to ON, a target search is in progress (step S15). When face recognition is OFF, a person who continues to be detected within the subject detection area 343 for, for example, one second or more becomes the tracking target. When face recognition is ON, pre-registered people become the detection target. When multiple people are detected, the person to be detected is determined based on priority.
If the tracking target is clicked in step S15, the process moves to step S13. If the tracking target is automatically detected, the tracking (automatic) state is entered (step S16).

If the tracking target is clicked while in the tracking in progress state, the process moves to step S13. Also, after the tracking forced reset time has elapsed or after a tracking timeout, the process returns to step S11. When a restart is requested, the process transitions to the tracking start position moving state regardless of the state.

[Method of capturing images of multiple subjects H and tracking control information]
FIG. 32 shows an example of a flowchart illustrating a method for capturing images of a plurality of subjects H.
As shown in FIG. 32, the method for capturing images of a plurality of subjects H is as follows.

First, the camera lens 12 moves to a tracking start position where video information including a plurality of subjects H is captured (step S21).
When the movement of the camera lens 12 is completed and automatic detection is set to OFF, the process waits for the selection of a tracking target (step S22).

In step S22, when the tracking target is clicked, the first subject H1, which is the main subject, is identified, and the process proceeds to step S23. In step S23, the set number of second subjects H2, which are close to the first subject H1, are detected. The timer for the number of people waiting time starts from this state (identification of the first subject H1), and after the set time has elapsed, the system switches to tracking (manual) (step S24). In step S23, when detection of the second subject H2 is completed and the first subject H1 is manually identified, the system switches to tracking (manual) (step S24). In the tracking (manual) state, neither a tracking forced reset nor a tracking timeout occurs. In step S24, when the first subject H1 is replaced by a click during tracking, all second subjects H2 are re-detected due to the number adjustment.
Furthermore, if the first subject H1 is lost in step S24, the tracking is lost (step S27).If the first subject H1 is detected again after the tracking start position is reached, the process returns to step S24.

In step S24, if tracking of the second subject H2 is lost or the number of tracking targets is changed by the application programming interface (API), the position is changed to the number adjustment detection position (step S25).
When the movement is completed, the target number adjustment is performed (step S26). In step S26, a timer for the number of second subjects H2 waiting time is started after the movement is completed. When the first subject H1 is determined by manual selection, the second subjects H2 for the specified number of people are detected, and once the adjustment of the number of second subjects H2 is completed, the process proceeds to step S24. Even if the number of second subjects H2 for the specified number of people are not detected after the number of people waiting time has elapsed, the process proceeds to step S24.
Also, in step S25, when the first subject H1 is determined by manual selection, once the adjustment of the number of people in the second subject H2 is completed, the process moves to step S24.

In step S24, if the tracking target is lost, the process moves to step S27.
In step S27, if the first subject H1 is detected again after the tracking start position is reached, the process returns to step S23.

On the other hand, when movement of the camera lens 12 is completed and automatic detection is set to ON in step S21, a target search is initiated (step S28). When face recognition is OFF, a person who continues to be detected within the subject detection area 343 for, for example, one second or more becomes the tracking target. When face recognition is ON, pre-registered people become the detection target. When multiple people are detected, the person to be detected is determined based on priority.

In step S28, when the tracking target is automatically detected, the main first subject H1 is identified and the process proceeds to step S23. In step S23, a set number of second subjects H2, which are closer to the first subject H1, are detected. The timer for the number of people waiting time starts from this state (identification of the first subject H1), and after the set time has elapsed, the system switches to tracking (automatic) (step S29). Once detection of the second subject H2 is complete and the first subject H1 is automatically selected, the system switches to tracking (automatic) (step S29).

In step S29, if tracking of the second subject H2 is lost or the number of tracking targets by the API is changed, the process moves to step S25.
When the movement is completed, the target number adjustment is performed (step S26). In step S26, a timer for the number of second subjects H2 waiting time is started after the movement is completed. When the first subject H1 is determined by manual selection, the second subjects H2 for the specified number of people are detected, and once the adjustment of the number of second subjects H2 is completed, the process proceeds to step S29. Even if the number of second subjects H2 for the specified number of people are not detected after the number of people waiting time has elapsed, the process proceeds to step S29.
In step S25, when the first subject H1 is automatically selected and the adjustment of the number of people in the second subject H2 is completed, the process proceeds to step S29. In step S29, when the first subject H1 is changed by clicking during tracking, all people in the second subject H2 are re-detected by adjusting the number of people.

In step S29, if tracking of the first subject H1 is lost, the tracking forced reset time has elapsed, or a tracking timeout occurs, the process returns to step S21. Also, when a restart is requested, the process transitions to "Tracking start position moving" regardless of the state.

[Tracking status display items]
FIG. 33 shows a list of exemplary display items of the tracking status.
As shown in FIG. 33, the Web UI display of the tracking status is not displayed, but the CGI command value is "idle."

When the Web UI display shows "Tracking Lost", the CGI command value will be "missing". This is the display when a manually selected tracking target is determined to be lost. When the number of people currently being detected is set to n and the number of people specified to be tracked is set to m, the tracking status will display "n/m" only when multiple people are being tracked and the device is currently detecting and tracking. When tracking is lost, the number of people detected and the number of people specified to be tracked will definitely be "0/m", so the number of people detected and the number of people specified to be tracked will not be displayed.

When the Web UI display is "Moving to start position," the CGI command value is "preparing." This is the display when the PTZ is moving to the tracking start position immediately before the application starts (while preparing for application operation), while searching for a target, or when transitioning to tracking lost.

When the Web UI display shows "Searching (n/m)", the CGI command value will be "searching". If the first subject H1 is not found, "0/m" will be displayed.

When the Web UI display is "Tracking (n/m)", the CGI command value will be "tracking".

When the Web UI display says "Please select a tracking target," the CGI command value will be "waiting." This is only displayed when tracking target selection is in "manual" mode.

[Relationship between tally control and tracking status display items]
FIG. 34 shows an example of the relationship between tally control and tracking status display items.
As shown in FIG. 34, when the broadcast is on air and the tracking status is not displayed, the tally (indicator lamp 1101 shown in FIG. 2) lights up in red.
When the camera is on air or the application is running and the tracking status is displayed as "Tracking", the tally lights up in red.
When on air or while the application is running, and the tracking status is displayed as "Tracking lost,""Moving to start position,""Detecting," or "Please select tracking target," the tally will flash red.

When the application is running and the tracking status is displayed as "Tracking", the tally lights up green.
When the application is running and the tracking status is displayed as "Tracking lost,""Moving to start position,""Detecting," or "Select tracking target," the tally will flash green.

When the tracking status display is hidden, the tally will turn off.

[Action and effect]
1 to 4, the imaging device 1 according to the first embodiment includes an imaging unit 111 and a storage unit 115. The storage unit 115 is, in particular, a ROM 1151.
The imaging unit 111 acquires video information including a plurality of subjects H in the video display area 341 (see FIG. 6 ). The storage unit 115 stores tracking control information (tracking control program). The tracking control information identifies one first subject H1 from the plurality of subjects H included in the video information, detects a second subject H2 (or second subject H21) that is closest to the first subject H1 from the plurality of subjects H, and tracks the first subject H1 and the second subject H2 to acquire video information.
According to the imaging device 1 configured in this manner, when multiple subjects H are present, it is possible to identify the first subject H1, track this first subject H1 as the appropriate subject, and produce a captured image that includes the second subject H2.

1 to 4, the user imaging system 100 according to the first embodiment includes an imaging unit 111 and a storage unit 115. The storage unit 115 may be replaced with a storage unit built into the personal computer 3.
The imaging unit 111 acquires video information including a plurality of subjects H in the video display area 341 (see FIG. 6 ). The storage unit 115 stores tracking control information (tracking control program). The tracking control information identifies one first subject H1 from the plurality of subjects H included in the video information, detects a second subject H2 (or second subject H21) that is closest to the first subject H1 from the plurality of subjects H, and tracks the first subject H1 and the second subject H2 to acquire video information.
According to the user imaging system 100 configured in this manner, when multiple subjects H are present, it is possible to identify the first subject H1, track this first subject H1 as the appropriate subject, and produce a captured image that includes the second subject H2.

1 to 4, the cloud imaging system 200 according to the first embodiment includes an imaging unit 111 and a storage unit 115. The storage unit 115 may be replaced with a storage unit built into the server 120.
The imaging unit 111 acquires video information including a plurality of subjects H in the video display area 341 (see FIG. 6 ). The storage unit 115 stores tracking control information (tracking control program). The tracking control information identifies one first subject H1 from the plurality of subjects H included in the video information, detects a second subject H2 (or second subject H21) that is closest to the first subject H1 from the plurality of subjects H, and tracks the first subject H1 and the second subject H2 to acquire video information.
According to the cloud imaging system 200 configured in this manner, when multiple subjects H are present, it is possible to identify the first subject H1, track this first subject H1 as the appropriate subject, and produce a captured image that includes the second subject H2.

Furthermore, as shown in Figures 1 to 4, the imaging method according to the first embodiment acquires video information including multiple subjects H within the video display area 341, identifies one first subject H1 from the multiple subjects H included in the video information, detects a second subject H2 (or second subject H21) that is closest to the first subject H1 from the multiple subjects H, and tracks the first subject H1 and the second subject H2 to acquire video information.
According to the imaging method configured in this manner, when there are multiple subjects H, it is possible to identify the first subject H1, track this first subject H1 as the appropriate subject, and produce a captured image including the second subject H2.

Furthermore, as shown in Figures 1 to 4, the storage medium 106 according to the first embodiment non-temporarily stores tracking control information that causes a computer to execute each of the following steps: acquire video information including multiple subjects H within the video display area 341; identify one first subject H1 from the multiple subjects H included in the video information; detect a second subject H2 (or second subject H21) that is closest to the first subject H1 from the multiple subjects H; and track the first subject H1 and the second subject H2 to acquire video information.
According to the storage medium 106 configured in this manner, when there are multiple subjects H, it is possible to identify the first subject H1, track this first subject H1 as the appropriate subject, and produce a photographed video including the second subject H2.

2. Second embodiment
The imaging device 1, user imaging system 100, cloud imaging system 200, imaging method, and storage medium 106 according to the second embodiment of the present disclosure will be described with reference to FIGS.
FIG. 35 shows an example of the external configuration of the operation terminal 4 that constitutes the user imaging system 100 or the cloud imaging system 200.

35 , the user imaging system 100 and the cloud imaging system 200 according to the second embodiment each include an operation terminal 4, which is used in combination with the operation terminal 2 according to the first embodiment or in place of the operation terminal 2. The operation terminal 4 is an infrared (IR) remote controller.
Although a detailed description of the configuration of the operation terminal 4 will be omitted, the operation terminal 4 is provided with an operation unit 41 that switches the number of tracking targets among the multiple subjects H. The operation unit 41 is configured as, for example, a dial type, a switch type, or a lever type. The operation unit 41 makes it possible to switch between, for example, a mode for tracking one subject H and a mode for tracking multiple subjects H.

Fig. 36 shows an example of a display when the multi-person mode is switched ON, and Fig. 37 shows an example of a display when the multi-person mode is switched OFF.
When the multiple person mode is ON, the camera zooms out around the first subject H1 to be tracked until the specified number of second subjects H21 and H22 are detected. Then, as shown in Fig. 36, the first subject H1, the second subject H21, and the second subject H22 are displayed as tracking targets.

On the other hand, when the multiple person mode is OFF, the camera zooms in with the first subject H1, which is the tracking target, at the center. Then, as shown in Figure 37, the first subject H1 is displayed as the tracking target.

It is difficult for a user to know whether the imaging device 1 is in the multi-person mode or the non-multi-person mode just by looking at the movement of the camera body 11 (see FIGS. 2 and 3).
For example, the user can view the tracking video to understand the number of people being tracked. Switching to the multi-person mode can be done at any time by viewing the live screen of the Web UI. Furthermore, switching to the multi-person mode can be done regardless of whether PTZ auto-framing is being performed or not.

All other components are the same or substantially the same as those of the imaging device 1, etc., pertaining to the first embodiment, and therefore redundant explanations will be omitted.

[Action and effect]
According to the imaging device 1, user imaging system 100, cloud imaging system 200, imaging method, and storage medium 106 of the second embodiment, it is possible to obtain the same effects as those obtained by the imaging device 1, user imaging system 100, cloud imaging system 200, imaging method, and storage medium 106 of the first embodiment.

3. Other embodiments
The present technology is not limited to the above-described embodiment, and various modifications are possible without departing from the spirit of the technology.

For example, in the above embodiment, the technology is applied to an imaging device with a PTZ auto-framing function, but the technology may also be applied to an imaging device that does not have the above function. To explain in more detail, the technology can be applied to an imaging device that simply has a PTZ function, an imaging device that does not have a PTZ function, etc.

Furthermore, an imaging system may be constructed that includes imaging devices with PTZ auto-framing functionality and imaging devices without PTZ auto-framing functionality.

An imaging device according to a first embodiment of the present disclosure includes an imaging unit that acquires video information including multiple subjects within a video display area, and a memory unit that stores tracking control information that identifies a first subject from the multiple subjects included in the video information, detects a second subject from the multiple subjects that is closest to the first subject, and tracks the first and second subjects to acquire video information.
According to an imaging device configured in this manner, when multiple subjects are present, it is possible to identify a first subject, track this first subject as the appropriate subject, and produce a captured video that includes a second subject.

An imaging system according to a second embodiment of the present disclosure includes an imaging unit that acquires video information including multiple subjects within a video display area, a display unit that displays the video information, a memory unit that stores tracking control information that identifies a first subject from the multiple subjects included in the video information, detects a second subject from the multiple subjects that is closest to the first subject, and tracks the first and second subjects to acquire video information, a control unit that controls the operation of the imaging unit, acquires the video information based on the tracking control information stored in the memory unit, and displays the video information on a display device, and an operation unit that operates the control unit.
According to an imaging system configured in this manner, when multiple subjects are present, it is possible to identify a first subject, track this first subject as the appropriate subject, and produce a captured image that includes a second subject.

An imaging method according to a third embodiment of the present disclosure acquires video information including multiple subjects within a video display area, identifies one first subject from the multiple subjects included in the video information, detects a second subject from the multiple subjects that is closest to the first subject, and tracks the first and second subjects to acquire video information.
According to the imaging method configured in this manner, when there are multiple subjects, it is possible to identify a first subject, track this first subject as the appropriate subject, and produce a captured video that includes a second subject.

A storage medium according to a fourth embodiment of the present disclosure non-temporarily stores tracking control information that causes a computer to execute each of the following steps: acquire video information including multiple subjects within a video display area; identify a first subject from the multiple subjects included in the video information; detect a second subject from the multiple subjects that is closest to the first subject; and track the first and second subjects to acquire video information.
According to a storage medium configured in this manner, when there are multiple subjects, it is possible to identify a first subject, track this first subject as the appropriate subject, and produce a photographed video including a second subject.

<Configuration of this technology>
The present technology has the following configuration: According to the present technology having the following configuration, in an imaging device, an imaging system, an imaging method, and a storage medium, when multiple subjects are present, it is possible to identify a first subject, and produce a captured video including a second subject while tracking the first subject as an appropriate subject.

(1)
an imaging unit that acquires video information including a plurality of subjects within a video display area;
a storage unit that stores tracking control information for identifying one first subject from the plurality of subjects included in the video information, detecting a second subject from the plurality of subjects that is closest to the first subject, and tracking the first subject and the second subject to acquire the video information; and
An imaging device comprising:
(2)
The imaging device described in (1), wherein the tracking control information identifies the first subject by a user's arbitrary selection from among the plurality of subjects, by matching the plurality of subjects with subject identification information registered before identification, or by initial detection by artificial intelligence from among the plurality of subjects.
(3)
The imaging device according to (1) or (2), wherein the tracking control information is for detecting the second subject by detecting some or all of the plurality of subjects.
(4)
The tracking control information allows the number of second subjects to be set to one or more, and when the number of second subjects is set to two or more, the tracking control information detects the second subjects to be tracked preferentially in order of proximity to the first subject.
(5)
The tracking control information allows the number of second subjects to be set to one or more, and when the number of second subjects is increased, the tracking control information zooms out the image display area while tracking the first subjects, increases the number of second subjects in the order in which they were detected in the image display area, and tracks the increased number of second subjects, adjusting the angle of view so that the image information can be acquired.
(6)
The tracking control information allows the number of second subjects to be set to one or more, and when the number of second subjects is reduced, the tracking control information zooms in on the image display area while tracking the first subjects, reduces the number of second subjects in the order in which they leave the image display area, and tracks the reduced number of second subjects while adjusting the angle of view so that the image information can be acquired.
(7)
The imaging device described in any one of (1) to (3), wherein the tracking control information identifies a new first subject from the plurality of subjects within the image display area when the first subject moves out of the image display area, or tracks the moved-out first subject again.
(8)
The tracking control information allows the number of second subjects to be set to one or more, and when the number of second subjects decreases within the image display area, the tracking control information zooms out the image display area while tracking the first subjects, increases the number of second subjects in the order in which they were detected within the image display area, and tracks the increased number of second subjects, adjusting the angle of view so that the image information can be acquired.
(9)
The imaging device described in any one of (1) to (8), wherein the imaging unit has a pan-tilt-zoom auto-framing function that tracks the first subject and the second subject to acquire the video signal.
(10)
an imaging unit that acquires video information including a plurality of subjects within a video display area;
a display unit that displays the video information;
a storage unit that stores tracking control information for identifying one first subject from the plurality of subjects included in the video information, detecting a second subject from the plurality of subjects that is closest to the first subject, and tracking the first subject and the second subject to acquire the video information; and
a control unit that controls an operation of the imaging unit, acquires the image information based on the tracking control information stored in the storage unit, and displays the image information on the display device;
and an operation unit for operating the control unit.
(11)
The tracking control information is
The imaging system according to (10), wherein an identifier for identifying one or more selected from the image display area, the first subject, and the second subject is displayed on the display unit superimposed on the image information.
(12)
Further comprising a server connectable via a line,
The imaging system according to claim 10 (the (10) or the (11)), wherein the tracking control information is stored in the storage unit from the server.
(13)
Acquire video information including a plurality of subjects within a video display area;
Identifying a first subject from the plurality of subjects included in the video information;
A second object that is closest to the first object is detected from the plurality of objects;
and acquiring the video information by tracking the first subject and the second subject.
(14)
Acquire video information including a plurality of subjects within a video display area;
Identifying a first subject from the plurality of subjects included in the video information;
A second object that is closest to the first object is detected from the plurality of objects;
a storage medium that non-temporarily stores tracking control information (tracking control program) that causes a computer to execute each procedure of tracking the first object and the second object and acquiring the video information;

This application claims priority based on Japanese Patent Application No. 2024-093346, filed on June 7, 2024, with the Japan Patent Office, the entire contents of which are incorporated herein by reference.

It is understood that those skilled in the art will be able to conceive of various modifications, combinations, subcombinations, and variations depending on design requirements and other factors, and that these are within the scope of the appended claims and their equivalents.

Claims (14)

an imaging unit that acquires video information including a plurality of subjects within a video display area;
a storage unit that stores tracking control information for identifying one first subject from the plurality of subjects included in the video information, detecting a second subject from the plurality of subjects that is closest to the first subject, and tracking the first subject and the second subject to acquire the video information; and
An imaging device comprising: 2. The imaging device according to claim 1, wherein the tracking control information identifies the first subject by a user's arbitrary selection from among the plurality of subjects, by matching the plurality of subjects with subject identification information registered before identification, or by initial detection of the plurality of subjects by artificial intelligence. The imaging device according to claim 1 , wherein the tracking control information detects the second subject by detecting some or all of the plurality of subjects. 2. The imaging device according to claim 1, wherein the tracking control information allows the number of the second subjects to be set to one or more, and when the number of the second subjects is set to two or more, the tracking control information detects the second subjects to be tracked preferentially in order of proximity to the first subject. 2. The imaging device according to claim 1, wherein the tracking control information allows the number of second subjects to be set to one or more, and when the number of second subjects is increased, the tracking control information zooms out the image display area while tracking the first subjects, increases the number of second subjects in the order in which they were detected in the image display area, and adjusts the angle of view so that the image information can be acquired by tracking the increased number of second subjects. 2. The imaging device according to claim 1, wherein the tracking control information allows the number of second subjects to be set to one or more, and when the number of second subjects is reduced, the tracking control information zooms in on the image display area while tracking the first subjects, reduces the number of second subjects in the order in which they leave the image display area, and adjusts the angle of view so that the image information can be acquired by tracking the reduced number of second subjects. 2. The imaging device according to claim 1, wherein the tracking control information is configured to identify a new first subject from the plurality of subjects within the image display area when the first subject moves out of the image display area, or to re-track the first subject that has moved out. 2. The imaging device according to claim 1, wherein the tracking control information allows the number of second subjects to be set to one or more, and when the number of second subjects decreases within the image display area, the image display area is zoomed out while tracking the first subjects, the number of second subjects is increased in the order in which they were detected within the image display area, and the angle of view is adjusted so that the image information can be acquired by tracking the increased number of second subjects. The imaging device according to claim 1 , wherein the imaging section has a pan-tilt-zoom auto-framing function for tracking the first subject and the second subject to acquire the video signal. an imaging unit that acquires video information including a plurality of subjects within a video display area;
a display unit that displays the video information;
a storage unit that stores tracking control information for identifying one first subject from the plurality of subjects included in the video information, detecting a second subject from the plurality of subjects that is closest to the first subject, and tracking the first subject and the second subject to acquire the video information; and
a control unit that controls an operation of the imaging unit, acquires the image information based on the tracking control information stored in the storage unit, and displays the image information on the display device;
and an operation unit for operating the control unit. The imaging system according to claim 10 , wherein the tracking control information is displayed on the display unit by superimposing an identifier that identifies one or more selected from the image display area, the first subject, and the second subject on the image information. Further comprising a server connectable via a line,
The imaging system according to claim 10 , wherein the tracking control information is stored in the storage unit from the server. Acquire video information including a plurality of subjects within a video display area;
Identifying a first subject from the plurality of subjects included in the video information;
A second object that is closest to the first object is detected from the plurality of objects;
and acquiring the video information by tracking the first subject and the second subject. Acquire video information including a plurality of subjects within a video display area;
Identifying a first subject from the plurality of subjects included in the video information;
A second object that is closest to the first object is detected from the plurality of objects;
a storage medium that non-temporarily stores tracking control information that causes a computer to execute each procedure of tracking the first object and the second object and acquiring the video information;