JP7754120B2 - Photographing system, photographing system control method, analysis device, and program - Google Patents

Description

This disclosure relates to an imaging system, a control method for an imaging system, an analysis device, and a program.

Technology related to searching for footage captured by a surveillance camera or other imaging device for footage that captures a specific subject, such as a person, object, or animal, is known. For example, Patent Document 1 describes a device that stores image data acquired by a surveillance camera in an image storage unit, performs image analysis on the image data, and generates feature data indicating the characteristics of the target subject upon detection. This device transmits the feature data and metadata, including at least one of the capture time of the image data from which the feature data was generated and the address of the image data in the image storage unit, to a file server accessible from other devices. This allows the metadata to be stored on the file server and used as an index for searching image data.

Japanese Patent Application Laid-Open No. 2021-100201

However, the above-mentioned conventional configuration leaves room for improvement in terms of efficient network utilization and protection of the subject's personal information.

The present disclosure therefore aims to improve technology for searching for specific subjects using video captured by a camera.

In some embodiments, the imaging system includes:
(1) a photographing device for photographing a video consisting of a plurality of frame images;
an analysis device including a first control unit capable of communicating with the image capturing device, which, in response to acquiring the image from the image capturing device, analyzes the image based on a first analysis logic to acquire first feature information indicating features of a subject captured in the image, and stores the image and the first feature information in a storage device in association with each other;
a server device capable of communicating with the analysis device;
a terminal device capable of communicating with the server device;
Equipped with
the server device includes a second control unit that acquires second feature information indicating features of the subject to be searched and transmits the acquired second feature information to the analysis device;
the first control unit of the analysis device, in response to receiving the second feature information from the server device, extracts from the storage device the first feature information whose similarity to the second feature information is greater than a predetermined first threshold, and transmits an extracted video that is a video corresponding to the extracted first feature information to the server device;
the second control unit of the server device transmits the extracted video received from the analysis device to the terminal device;
The terminal device includes a third control unit that causes a display unit to display the extracted video received from the server device.

In this way, upon receiving video from the image capture device, the analysis device analyzes the video and acquires first feature information. Upon receiving second feature information, the analysis device extracts extracted video, which is video similar to the second feature information, and transmits the extracted video to the server device. The server device then integrates the search results from each analysis device and transmits them to the terminal device, which then displays the received video. This makes it easier for users of the terminal device to review the searched video and find specific subjects. Here, the analysis device transmits the video and first feature information to the server device for video whose similarity to the second feature information is greater than the first threshold, rather than for all captured video, thereby minimizing the amount of data transferred over the network. Furthermore, because the server device stores the first feature information acquired from the searched video, rather than for all captured video, the risk of personal information being leaked from the server device, such as the leak of first feature information, can be reduced. This improves the technology for searching for specific subjects using video captured by an image capture device.

In one embodiment,
(2) In the imaging system of (1),
the first control unit of the analysis device, in response to acquiring the video from the imaging device, further stores date and time information indicating a date and time when the video was captured in the storage device in association with the video and the first feature information;
the second control unit of the server device further acquires a date and time range to be searched in addition to the second characteristic information, and further transmits the date and time range to the analysis device in addition to the second characteristic information;
The first control unit of the analysis device
In response to receiving the second feature information and the date and time range from the server device, extracting from the storage device the first feature information of the video captured in the date and time range, the first feature information having a similarity to the second feature information greater than the first threshold;
transmitting to the server device an extracted video that is a video corresponding to the extracted first feature information, the date and time information of the extracted video, and a similarity between the first feature information and the second feature information;
the second control unit of the server device transmits the extracted video, the date and time information of the extracted video, and the similarity received from the analysis device to the terminal device;
The third control unit of the terminal device may cause the display unit to display the extracted video, the date and time information of the extracted video, and the similarity received from the server device.

In this way, the analysis device searches by specifying the date and time range of the video's capture date and time, and by excluding from the search range the period during which the location of the search target was known, it is possible to efficiently search for video containing the search target. Furthermore, the terminal device displays the date and time information and similarity of the extracted video, making it easier for the user to find the subject by checking when the video in which the search target subject appears was captured and the degree of similarity.

In one embodiment,
(3) In the imaging system of (1) or (2),
A plurality of the imaging devices provided at a plurality of locations;
a plurality of analysis devices each capable of communicating with one or more of the imaging devices included in the plurality of imaging devices;
Equipped with
the second control unit of the server device further acquires a geographical range to be searched in addition to the second feature information, and transmits the second feature information to the analysis device that is capable of communicating with the imaging device installed at a location included in the geographical range;
the first control unit of the analysis device that has received the second feature information from the server device extracts, from the storage device, the first feature information of the video whose similarity to the second feature information is greater than the first threshold, and transmits, to the server device, the extracted video corresponding to the extracted first feature information and location information indicating a location of the imaging device that captured the extracted video;
the second control unit of the server device transmits the extracted video and the location information of the extracted video received from the analysis device to the terminal device;
The third control unit of the terminal device may cause the display unit to display the extracted video and the location information of the extracted video received from the server device.

In this way, the terminal device displays location information for the extracted video, making it easier for the user to check where the video that captures the subject they are searching for was shot and find the subject.

In one embodiment,
(4) In any one of the imaging systems (1) to (3),
the third control unit of the terminal device receives, from a user, a designation of a search target image that is an image in which the subject to be searched is reflected;
The second control unit of the server device may acquire, as the second feature information, feature information acquired by analyzing the search target image designated by the user.

In this way, the analysis device searches for video containing the subject to be searched for based on the feature information (second feature information) obtained by analyzing the search target image specified by the user. Therefore, the user can easily search for video containing the subject and check the results by simply specifying the search target image.

In one embodiment,
(5) In any one of the imaging systems (1) to (4),
the second control unit of the server device acquires, in addition to the second feature information, third feature information indicating an appearance feature specific to the subject to be searched, and transmits, in addition to the second feature information, the third feature information to the analysis device;
The first control unit of the analysis device
extracting, from the storage device, the first feature information whose similarity to the second feature information is greater than the first threshold value in response to receiving the second feature information and the third feature information from the server device;
analyzing the image corresponding to the extracted first feature information based on a second analysis logic to obtain fourth feature information indicating external features specific to the subject captured in the image;
The fourth feature information whose similarity to the third feature information is greater than a predetermined second threshold may be extracted, and the video corresponding to the extracted fourth feature information may be transmitted to the server device as the extracted video.

In this way, the analysis device searches for video containing the search target subject based on feature information extracted based on the second analysis logic as well as the first analysis logic, making it possible to search for video that is similar to the search target subject's unique external features.

In one embodiment,
(6) In the imaging system of (5),
the second control unit of the server device transmits, to the analysis device, analysis logic information used in extracting the third feature information, in addition to the second feature information and the third feature information;
The first control unit of the analysis device may use the analysis logic information as the second analysis logic to analyze the video corresponding to the extracted first feature information, and obtain the fourth feature information.

In this way, the server device transmits analysis logic information to be used as the second analysis logic to the analysis device. Therefore, the analysis device can further narrow down and extract videos with similar features specific to the subject being searched from among videos with similar first feature information, which is general feature information.

In one embodiment,
(7) In the imaging system of (5) or (6),
The first control unit of the analysis device
In response to acquiring the video from the photographing device, analyze the video based on the first analysis logic to acquire the first feature information, and store the video, a spatial position in the video of the subject captured in the video, and the first feature information in a storage device in association with each other;
extracting, from the storage device, the first feature information whose similarity to the second feature information is greater than the first threshold value in response to receiving the second feature information and the third feature information from the server device;
The area corresponding to the spatial position of the image corresponding to the extracted first feature information may be analyzed based on the second analysis logic, and feature information indicating the characteristics of the subject reflected in the area of the image may be obtained as the fourth feature information.

In this way, by limiting the analysis target based on the second analysis logic to the area corresponding to the spatial position of the video, the analysis device can reduce the computational costs required for analysis.

A control method for an imaging system according to some embodiments includes:
(8) a photographing device that photographs a video consisting of a plurality of frame images;
an analysis device including a first control unit capable of communicating with the image capturing device, which, in response to acquiring the image from the image capturing device, analyzes the image based on a first analysis logic to acquire first feature information indicating features of a subject captured in the image, and stores the image and the first feature information in a storage device in association with each other;
a server device capable of communicating with the analysis device;
a terminal device capable of communicating with the server device;
A control method for an imaging system comprising:
a second control unit of the server device acquires second feature information indicating features of the subject to be searched, and transmits the acquired second feature information to the analysis device;
the first control unit of the analysis device, in response to receiving the second feature information from the server device, extracts from the storage device the first feature information whose similarity to the second feature information is greater than a predetermined first threshold, and transmits an extracted video that is a video corresponding to the extracted first feature information to the server device;
the second control unit of the server device transmits the extracted video received from the analysis device to the terminal device;
A third control unit of the terminal device causes a display unit to display the extracted video received from the server device.

In this way, the server device integrates the search results from each analysis device and transmits them to the terminal device, which then displays the received video. This makes it easier for users of the terminal device to review the searched video and find specific subjects. Here, the analysis device transmits to the server device not all captured video, but only video whose similarity to the second feature information is greater than the first threshold, along with the first feature information. This minimizes the amount of data transferred over the network. Furthermore, because the server device stores the first feature information obtained from the searched video, rather than all captured video, this reduces the risk of personal information leaks, such as the leakage of the first feature information from the server device. This improves the technology for searching for specific subjects using video captured by a capture device.

In some embodiments, the analysis device
(9) An image capturing device that captures a video consisting of a plurality of frame images, and an analysis device that can communicate with a server device,
In response to acquiring the image from the photographing device, analyze the image based on a first analysis logic to acquire first feature information indicating features of a subject captured in the image, and store the image and the first feature information in a storage device in association with each other;
In response to receiving second feature information indicating features of the subject to be searched from the server device, extracting from the storage device the first feature information whose similarity to the second feature information is greater than a predetermined first threshold, and transmitting an extracted image that is an image corresponding to the extracted first feature information to the server device.
It has a control unit.

In this way, the analysis devices transmit search results to the server device, which can then consolidate the search results from each analysis device and transmit them to the terminal device, thereby presenting the search results to the user. Therefore, even without providing the server device with feature analysis and search functions, simply providing an analysis device makes it possible to search for videos that capture the target subject. Furthermore, users of terminal devices can easily review the searched videos and find specific subjects. Therefore, according to this embodiment, specific subjects can be easily found using videos captured by a camera. Here, the analysis device transmits the videos and first feature information to the server device for videos whose similarity to the second feature information is greater than the first threshold, rather than for all captured videos. This minimizes the amount of data transferred over the network. Furthermore, because the server device stores the first feature information obtained from the searched videos, rather than for all captured videos, the risk of personal information being leaked, such as the leak of the first feature information, from the server device can be reduced. Therefore, technology for searching for specific subjects using videos captured by a camera device can be improved.

In one embodiment,
(10) In the analysis device of (9),
The control unit
In response to acquiring the video from the imaging device, date and time information indicating the date and time when the video was captured is further stored in the storage device in association with the video and the first feature information;
In response to receiving from the server device the date and time range to be searched in addition to the second feature information, extracting from the storage device the first feature information of the video captured in the date and time range, the first feature information having a similarity to the second feature information greater than the first threshold;
An extracted video that is a video corresponding to the extracted first feature information and the date and time information of the extracted video may be transmitted to the server device.

In this way, the analysis device searches by specifying the date and time range of the video's capture date and time, so by excluding from the search range the period during which the location of the search target was known, it is possible to efficiently search for video containing the search target. Furthermore, by displaying the date and time information of the extracted video, the terminal device makes it easier for the user to find the subject by checking when the video in which the search target subject appears was captured.

In one embodiment,
(11) In the analysis device according to (9) or (10),
The control unit
extracting, from the storage device, the first feature information whose similarity to the second feature information is greater than the first threshold value in response to receiving, in addition to the second feature information, third feature information indicating an appearance feature specific to the subject to be searched, from the server device;
analyzing the image corresponding to the extracted first feature information based on a second analysis logic to obtain fourth feature information indicating external features specific to the subject captured in the image;
The fourth feature information whose similarity to the third feature information is greater than a predetermined second threshold may be extracted, and the video corresponding to the extracted fourth feature information may be transmitted to the server device as the extracted video.

In this way, the analysis device searches for video that includes the search target subject based on feature information extracted based on not only the first analysis logic but also the second analysis logic, making it possible to search for video that is similar in appearance to the search target subject.

In one embodiment,
(12) In the analysis device of (11),
The control unit
In response to receiving from the server device the second feature information, the third feature information, and analysis logic information used to extract the third feature information, the analysis logic information may be used as the second analysis logic to analyze the video corresponding to the extracted first feature information, thereby obtaining the fourth feature information.

In this way, the analysis device receives analysis logic information to be used as the second analysis logic from the server device. Therefore, the analysis device can further narrow down and extract videos with similar first feature information (general feature information) to those with similar features specific to the subject being searched for.

In one embodiment,
(13) In the analysis device according to (11) or (12),
The control unit
In response to acquiring the video from the photographing device, analyze the video based on the first analysis logic to acquire the first feature information, and store the video, a spatial position in the video of the subject captured in the video, and the first feature information in a storage device in association with each other;
extracting, in response to receiving the second feature information and the third feature information from the server device, the first feature information whose similarity to the second feature information is greater than the first threshold value from the storage device;
analyzing an area of the image corresponding to the spatial position of the extracted first feature information based on the second analysis logic, and acquiring feature information indicating appearance features specific to the subject reflected in the area of the image as the fourth feature information;
The analysis device according to claim 11.

In this way, by limiting the analysis target based on the second analysis logic to the area corresponding to the spatial position of the video, the analysis device can reduce the computational costs required for analysis.

In some embodiments, the program
(14) A photographing device that photographs a video consisting of a plurality of frame images, and a computer that can communicate with a server device,
In response to acquiring the image from the photographing device, analyze the image based on a first analysis logic to acquire first feature information indicating features of a subject captured in the image, and store the image and the first feature information in a storage device in association with each other;
In response to receiving second feature information indicating features of the subject to be searched from the server device, extracting from the storage device the first feature information whose similarity to the second feature information is greater than a predetermined first threshold, and transmitting an extracted image that is an image corresponding to the extracted first feature information to the server device.
A program for operating an analysis device equipped with a control unit.

In this way, the computer operating as an analysis device by the program transmits search results to the server device, and the server device can present the search results to the user by, for example, integrating the search results from each analysis device (computer) and transmitting them to the terminal device. Therefore, even without providing the server device with feature analysis and search functions, the program allows the computer to operate as the analysis device, making it possible to search for videos containing the search target subject. Furthermore, the user of the terminal device can easily review the searched videos and find a specific subject. Here, the computer transmits the videos and first feature information to the server device for videos whose similarity to the second feature information is greater than the first threshold, rather than all captured videos, thereby minimizing the amount of data transferred over the network. Furthermore, the server device stores the first feature information obtained from the searched videos, rather than all captured videos, thereby reducing the risk of personal information being leaked, such as the leak of the first feature information from the server device. This improves the technology for searching for specific subjects using videos captured by a camera.

One embodiment of the present disclosure can improve technology for searching for specific subjects using video captured by a camera.

FIG. 1 is a diagram illustrating an example of the configuration of an imaging system according to an embodiment of the present disclosure. FIG. 2 is a block diagram showing an example of the configuration of the analysis device shown in FIG. 1 . FIG. 2 is a block diagram showing an example of the configuration of the server device of FIG. 1 . FIG. 2 is a block diagram showing an example of the configuration of the terminal device of FIG. 1 . 2 is a sequence chart showing an example of the operation of the imaging system of FIG. 1 . 2 is a sequence chart showing an example of the operation of the imaging system of FIG. 1 . 2 is a sequence chart showing an example of the operation of the imaging system of FIG. 1 . 2 is a sequence chart showing an example of the operation of the imaging system of FIG. 1 . 2 is a flowchart showing an example of the operation of the analysis device of FIG. 1 .

<Comparative Example>
The device according to the comparative example (claim 1 of Patent Document 1) is "a device including an image storage unit that stores image data acquired by a surveillance camera, an analysis unit that performs image analysis on the image data and generates feature data indicating features of the target subject in response to detection of a target subject that should be detected from among subjects that may be included in the image data, and a transmission unit that transmits metadata including the feature data, the shooting time of the image data for which the feature data was generated, and/or the address of the image data in the image storage unit, to a file server accessible from a device other than the device." According to this device, the metadata can be stored in the file server and used as an index for searching image data (paragraph 0073 of Patent Document 1).

However, in the configuration of the comparative example, feature information for many subjects, including those not targeted by the search, is transmitted over the network and stored on the server device. Therefore, the configuration of the comparative example is wasteful and inefficient in that a large amount of information that is never used is transmitted over the network. Furthermore, by using feature information for subjects such as people, it is possible to track when and where those people were present. Therefore, there is room for improvement in that feature information treated as personal information is aggregated and stored on the server device, increasing the risk of feature information being leaked from the server device and other personal information management issues.

The present disclosure aims to improve the efficient use of networks and the protection of personal information of subjects in a configuration for finding specific subjects using video captured by a camera.

<Embodiment>
Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings. In each drawing, parts having the same configuration or function are denoted by the same reference numerals. In the description of this embodiment, duplicated descriptions of the same parts may be omitted or simplified as appropriate.

(Photography system)
FIG. 1 is a diagram illustrating an example configuration of an image capture system 1 according to an embodiment of the present disclosure. The image capture system 1 searches for a specific subject based on video captured at multiple locations. In this embodiment, the specific subject to be searched for is a specific person, such as a wandering elderly person, a missing child, or a criminal on the run. However, the search target does not have to be a person. For example, the image capture system 1 may search for a lost pet or a stolen car as a specific subject. The image capture system 1 includes image capture devices 10a, 10b, and 10c, analysis devices 20a and 20b, a server device 30, and a terminal device 40. The analysis devices 20a and 20b, the server device 30, and the terminal device 40 are communicably connected to a network N, which may include, for example, the Internet, an intranet, a mobile communication network, or the like.

The camera devices 10a, 10b, and 10c are installed at various locations within the subject and capture video consisting of multiple frame images. Hereinafter, the camera devices 10a, 10b, and 10c are collectively referred to as "camera device 10." The camera device 10 is, for example, a PTZ camera, but the camera may have any configuration and function. In this embodiment, an example is described in which the camera system 1 includes three camera devices 10, but the number of camera devices 10 is arbitrary. The camera devices 10 are installed over a wide area, such as an entire city, ward, town, or village. However, they may also be installed over an even wider area, such as a prefecture or country, or over a smaller area, such as within a specific building or facility. The camera device 10 captures outdoor images, but is not limited to this and may also capture indoor images, for example. As soon as the camera device 10 captures video, it continuously transmits the acquired video to the analysis devices 20a and 20b, which are connected to the camera device 10 in a manner that allows communication.

The analysis devices 20a and 20b continuously acquire video from the image capture device 10, analyze the video based on a first analysis logic in response to the acquisition of the video, and acquire first feature information indicating the characteristics of the subject captured in the video. Furthermore, the analysis devices 20a and 20b associate the video with the first feature information and store it in a storage device (storage unit 22, described below with reference to FIG. 2). The first analysis logic is analysis logic used by the analysis device 20 to continuously analyze the video in response to receiving the video. Details of the first analysis logic will be described below. Hereinafter, the analysis devices 20a and 20b will be collectively referred to as "analysis device 20." The analysis devices 20 are installed near the corresponding image capture device 10. Each analysis device 20 is communicatively connected to one or more image capture devices 10 and acquires video from the connected image capture devices 10. For example, the analysis device 20a is communicatively connected to the image capture device 10a. The analysis device 20b is communicatively connected to the image capture devices 10b and 10c. In this embodiment, an example is described in which the imaging system 1 includes two analysis devices 20, but the number of imaging devices 10 is arbitrary.

The server device 30 is a computer that accepts video search requests from the terminal device 40. The server device 30 also requests the analysis device 20 to perform a video search and transmits the search results to the terminal device 40. The server device 30 may store in advance, for each analysis device 20, information regarding the shooting locations of each imaging device 10 that is communicatively connected to that analysis device 20.

The terminal device 40 is a computer operated by a user. The terminal device 40 accepts search requests from the user for videos that include the subject being searched for, and displays the search results.

In the above configuration, in response to receiving a search request from the terminal device 40, the server device 30 acquires second feature information indicating the features of the subject to be searched, and transmits the acquired second feature information to the analysis device 20. In response to receiving the second feature information from the server device 30, the analysis device 20 extracts first feature information similar to the second feature information from the storage device, and transmits extracted video, which is video corresponding to the extracted first feature information, to the server device 30. The server device 30 transmits the extracted video received from the analysis device 20 to the terminal device 40. The terminal device 40 displays the extracted video received from the server device 30 on a display unit.

In this way, the analysis device 20 continuously receives video from the image capture device 10 and sequentially analyzes the received video based on the first analysis logic to obtain first feature information. In response to receiving second feature information from the server device 30, the analysis device 20 searches for first feature information similar to the second feature information and transmits the video corresponding to the retrieved first feature information to the terminal device 40 via the server device 30. In this way, in the image capture system 1, the first feature information is stored on-site in the analysis device 20 along with the recorded video, and only the video and first feature information extracted for search purposes are sent to the server device 30. Therefore, the image capture system 1 is superior in terms of efficient communication utilization. Furthermore, because only limited information for search purposes is accumulated on the server device 30, the risk of personal information leaks is reduced. Therefore, the image capture system 1 can improve the technology for searching for specific subjects using video captured by the image capture device 10 in terms of efficient network utilization and protection of the subject's personal information.

(Analysis device)
FIG. 2 is a block diagram showing an example configuration of the analysis device 20 in FIG. 1 . The analysis device 20 is one or more computers that can communicate with each other. The analysis device 20 may be, for example, an electronic device equipped with an FPGA (Field-Programmable Gate Array). The analysis device 20 is not limited to these and may be, for example, any general-purpose electronic device such as a PC (Personal Computer), or another dedicated electronic device. As shown in FIG. 2 , the analysis device 20 includes a control unit 21, a storage unit 22, and a communication unit 23.

The control unit 21, which serves as the first control unit, includes one or more processors. In one embodiment, the "processor" may be, but is not limited to, a general-purpose processor or a dedicated processor specialized for a particular process. The control unit 21 is communicatively connected to each component of the analysis device 20 and controls the operation of the entire analysis device 20.

The memory unit 22 includes any memory device, such as a hard disk drive (HDD), a solid state drive (SSD), a read-only memory (ROM), or a random access memory (RAM). The memory unit 22 may function as a main memory device, an auxiliary memory device, or a cache memory, for example. The memory unit 22 stores any information used in the operation of the analysis device 20. For example, the memory unit 22 may store system programs, application programs, and various information received by the communication unit 23. For example, the memory unit 22 may store video continuously received from the image capture device 10 and first feature information resulting from analyzing the video based on first analysis logic. The memory unit 22 is not limited to being built into the analysis device 20, but may also be an external database or an external storage module.

The communication unit 23 includes any communication module that can communicate with other devices, such as the image capture device 10 and the server device 30, using any communication technology. The communication unit 23 may also include a communication control module for controlling communication with other devices, and a storage module for storing communication data, such as identification information, required for communication with other devices.

The functions of the analysis device 20 can be realized by executing a computer program (program) according to this embodiment on a processor included in the control unit 21. That is, the functions of the analysis device 20 can be realized by software. The computer program causes a computer to execute the processing of steps included in the operation of the analysis device 20, thereby causing the computer to realize the functions corresponding to the processing of each step. That is, the computer program is a program that causes a computer to function as the analysis device 20 according to this embodiment. The computer program may be recorded on a computer-readable recording medium. Programs include information used for processing by an electronic computer that is equivalent to a program. For example, data that is not a direct instruction to a computer but has properties that define computer processing falls under the category of "something equivalent to a program."

Some or all of the functions of the analysis device 20 may be implemented by a dedicated circuit included in the control unit 21. In other words, some or all of the functions of the analysis device 20 may be implemented by hardware.

(Server device)
FIG. 3 is a block diagram showing an example configuration of the server device 30 in FIG. 1 . The server device 30 may be one or more computers capable of communicating with each other. The server device 30 may be, for example, a workstation (WS) or a PC. The server device 30 includes a control unit 31, a storage unit 32, and a communication unit 33. The hardware configurations of the control unit 31, storage unit 32, and communication unit 33 as a second control unit are similar to the control unit 21, storage unit 22, and communication unit 23 of the analysis device 20 described with reference to FIG. 2 , and therefore detailed description thereof will be omitted. Some or all of the functions of the server device 30 may be implemented by software or hardware. The storage unit 32 of the server device 30 may pre-store, for each analysis device 20, information regarding the photography locations of each imaging device 10 communicably connected to the analysis device 20.

(Terminal device)
4 is a block diagram showing an example configuration of the terminal device 40 in FIG. 1. The terminal device 40 is one or more computers capable of communicating with each other. The terminal device 40 may be, for example, a PC, a tablet terminal, or a smartphone. The terminal device 40 includes a control unit 41, a storage unit 42, a communication unit 43, an input unit 44, and an output unit 45. The hardware configurations of the control unit 41, storage unit 42, and communication unit 43 as a third control unit are similar to the control unit 21, storage unit 22, and communication unit 23 of the analysis device 20 described with reference to FIG. 2, and therefore detailed description thereof will be omitted.

The input unit 44 includes one or more input interfaces that accept user input operations and acquire input information based on the user operations. For example, the input unit 44 may be, but is not limited to, physical keys, capacitive keys, a pointing device, a touchscreen integrated with the display of the output unit 45, or a microphone that accepts voice input.

The output unit 45 includes one or more output interfaces that output information to the user and notify the user. For example, the output unit 45 may be, but is not limited to, a display that outputs information as an image or a speaker that outputs information as audio. Such a display may be, for example, a liquid crystal panel display or an organic EL (Electro Luminescence) display. At least one of the input unit 44 and the output unit 45 may be configured integrally with the terminal device 40 or provided separately.

Some or all of the functions of the terminal device 40 may be implemented by software or hardware.

(Operation example 1 of the imaging system)
5A and 5B are sequence charts showing an example of the operation of the imaging system 1 of Fig. 1. The operation of each device described with reference to Fig. 5A and 5B may correspond to at least a part of a control method for the imaging system 1. The operation of each step in Fig. 5A and 5B may be executed based on control by at least one of the imaging device 10, the control unit 21 of the analysis device 20, the control unit 31 of the server device 30, and the control unit 41 of the terminal device 40.

In step S1 of FIG. 5A, each image capture device 10 begins capturing video. Specifically, the image capture device 10 begins the process of continuously acquiring frame images at a predetermined frame rate and generating video, which is a moving image.

In step S2, each imaging device 10 transmits the captured image to the corresponding analysis device 20. Specifically, as soon as the imaging device 10 captures image through imaging, it continuously transmits the image to the communicatively connected analysis device 20. As a result, while the imaging system 1 is operating, the analysis device 20 continuously receives image from the corresponding imaging device 10. However, the analysis device 20 may also intermittently (for example, every second) perform the process of extracting first feature information through image analysis in step S3, which will be described later.

In step S3, upon receiving video from the communicatively connected image capturing device 10, the control unit 21 of the analysis device 20 analyzes the video based on a first analysis logic set in advance to obtain first feature information indicating the features of the subjects appearing in the video. The first analysis logic is analysis logic for obtaining first feature information indicating the features of the subjects appearing in the video for all of the received video. For example, when obtaining the features of a person as the subject, the control unit 21 may obtain, for example, the following information as the first feature information:
Information for identifying a face: For example, the contour of the face, and the size, shape, and relative position of facial features such as the eyes, nose, and mouth relative to the contour.
- Skin color of the face.
- Hair information: e.g. hairstyle, hair color.
- Information about each part of the body: for example, height, body shape, build, relative lengths of each part such as hands and feet.
- Information about clothing: For example, the shape and color of the clothing on the upper body, the general shape and color of the clothing on the lower body, the general shape and color of the shoes.
Information about personal belongings: for example, whether or not the person has glasses, a hat, a bag, a walking stick, etc., and the general shape of each.

The control unit 21 may use a learning model previously acquired through machine learning as the first analysis logic for acquiring such first feature information. The first analysis logic may be provided, for example, as parameters of a neural network.

In step S4, the control unit 21 of the analysis device 20 associates the video acquired from the image capture device 10, the first feature information of the video, and date and time information indicating the date and time the video was captured, and stores them in a storage device such as the memory unit 22. The control unit 21 repeatedly performs the processes of steps S2 to S4. That is, the image capture device 10 and the analysis device 20 perform the processes of steps S2 to S4 in parallel with the processes of steps S5 to S14. The analysis device 20 continuously acquires video from the image capture device 10 and records the acquired video. Alternatively, the control unit 21 may continuously analyze the video based on the first analysis logic as it is acquired sequentially, and continuously output the first feature information and date and time information to the memory unit 22. Alternatively, the control unit 21 may intermittently perform analysis of the video based on the first analysis logic, for example at intervals of about one second, and output the first feature information and date and time information to the memory unit 22. The control unit 21 may also delete from the storage unit 22 any video whose storage period has elapsed, as well as the first feature information and date and time information corresponding to that video, based on the storage capacity of the storage unit 22, etc.

In step S5, the control unit 41 of the terminal device 40 acquires a search target image in response to a user instruction. A search target image is an image that includes the subject to be searched for. For example, when searching for a missing person as the subject, the control unit 41 may accept a specification from the user of an image that includes the missing person. The control unit 41 may acquire, as the search target image, one or more still images that include the missing person, or a moving image that includes the missing person.

In step S6, the control unit 41 of the terminal device 40 accepts from the user a date and time range, which is the range of shooting dates and times for which the subject is to be searched, and a geographical range of the shooting location. For example, the control unit 41 may display an input field for the start date and time and end date and time for the search on the display unit of the output unit 45 and accept the date and time range specification. For example, the control unit 41 may display a map including the installation location of the shooting device 10 on the display unit of the output unit 45, and obtain the range selected by the user on the map as the geographical range of the shooting location.

In step S7, the control unit 41 of the terminal device 40 transmits the search target image acquired in step S5 and the date and time range and geographic range acquired in step S6 to the server device 30, requesting a video search.

In step S8, the control unit 31 of the server device 30 analyzes the search target image received from the terminal device 40 in step S7 and acquires second feature information indicating the features of the subject of the search target. The control unit 31 may acquire feature information of the same type as the first feature information as the second feature information using the same analysis logic as the first analysis logic used by the analysis device 20. Note that in this embodiment, the server device 30 analyzes the search target image and acquires the second feature information. However, the server device 30 may also acquire feature information obtained by another device analyzing the search target image using the same analysis logic as the second feature information. For example, the server device 30 may receive, from the terminal device 40, feature information obtained by the terminal device 40 or another device analyzing the search target image, and acquire that feature information as the second feature information.

In step S9 of FIG. 5B, the control unit 31 of the server device 30 transmits the date and time range acquired in step S7 and the second feature information acquired in step S8 to the analysis device 20 communicatively connected to the image capture devices 10 included in the geographical range acquired in step S7, and requests a video search. As described above, the storage unit 32 of the server device 30 pre-stores, for each analysis device 20, information regarding the image capture locations of each image capture device 10 communicatively connected to that analysis device 20. Therefore, the server device 30 may identify the image capture devices 10 included in the geographical range acquired in step S7 by referencing the image capture locations of each image capture device 10. If the server device 30 does not retain information on the image capture locations of each image capture device 10, the server device 30 may transmit the date and time range and the second feature information to all analysis devices 20 and request a video search.

In step S10, the control unit 21 of the analysis device 20 that has received a request to search for video extracts from the storage unit 22 first feature information that is similar to the received second feature information and date/time range. Specifically, for example, the control unit 21 may evaluate the similarity between each piece of first feature information for the specified date/time range stored in the storage unit 22 and the second feature information, and extract first feature information whose degree of similarity (similarity) is greater than a predetermined first threshold. Note that if the search request from the terminal device 40 does not include a specified date/time range, the control unit 21 may evaluate the similarity between all pieces of first feature information stored in the storage unit 22 and the second feature information, and extract first feature information that is similar to the second feature information.

In step S11, the control unit 21 of the analysis device 20 acquires, from the memory unit 22, the video corresponding to the first feature information extracted in step S10 as extracted video.

In step S12, the control unit 21 of the analysis device 20 transmits the extracted video acquired in step S11 to the server device 30. The control unit 21 may also transmit to the server device 30 the similarity degree acquired for the extracted video in step S10 and information regarding the date and time the extracted video was captured.

In step S13, the control unit 31 of the server device 30 transmits the extracted video received from the analysis device 20 in step S12 to the terminal device 40 as a video search result. The control unit 31 may also transmit to the terminal device 40 the similarity degree acquired in step S10 for the extracted video, information related to the shooting date and time of the extracted video, and the shooting location (location information) of the imaging device 10.

In step S14, the control unit 41 of the terminal device 40 displays the extracted video received from the server device 30 in step S13 as a search result on the display unit of the output unit 45. The user can check the search results for the search target subject that appears in the search target image by referring to the display unit of the output unit 45. The control unit 41 may also display the similarity, shooting date and time, and shooting location (location information) on the display unit of the output unit 45. For example, the control unit 41 may display the start date and time and end date and time of shooting. The control unit 41 may also display the shooting location on a map. This allows the user to check when and where the search target subject was located.

(Operation example 2 of the imaging system)
6A and 6B are sequence charts showing an example of the operation of the imaging system 1 of Fig. 1. The operation of each device described with reference to Fig. 6A and 6B may correspond to at least a part of a control method for the imaging system 1. The operation of each step in Fig. 6A and 6B may be executed based on control by at least one of the imaging device 10, the control unit 21 of the analysis device 20, the control unit 31 of the server device 30, and the control unit 41 of the terminal device 40.

In step S21 of FIG. 6A, each image capture device 10 begins capturing video. The image capture device 10 can perform the process of step S21 in the same manner as step S1 of FIG. 5A.

In step S22, each image capture device 10 transmits the captured video to the corresponding analysis device 20. The image capture device 10 can perform the process of step S21 in the same manner as step S2 in Figure 5A.

In step S23, upon acquiring video from the communicatively connected imaging device 10, the control unit 21 of the analysis device 20 analyzes the video based on a pre-set first analysis logic and acquires first feature information indicating the characteristics of the subject appearing in the video. The control unit 21 can acquire the first acquired information by processing similar to step S3 in FIG. 5A. Furthermore, in parallel with analyzing the video to acquire the first feature information, the control unit 21 acquires information regarding the spatial position of the subject appearing in the video. Specifically, the control unit 21 may acquire the angle of view and coordinates of the area occupied by the subject in the video as information regarding the spatial position of the subject.

In step S24, the control unit 21 of the analysis device 20 associates the video acquired from the image capture device 10, the first feature information of the video, date and time information indicating the date and time the video was captured, and information regarding the spatial position of the subject, and stores them in a storage device such as the memory unit 22. The control unit 21 repeatedly performs the processes of steps S22 to S24. That is, the image capture device 10 and the analysis device 20 perform the processes of steps S22 to S24 in parallel with the processes of steps S25 to S36. The analysis device 20 constantly acquires video from the image capture device 10 and records the acquired video. The control unit 21 may continuously analyze the video as it is acquired sequentially, and continuously output the first feature information, date and time information, and information regarding the spatial position to the memory unit 22. Alternatively, the control unit 21 may perform analysis of the video based on the first analysis logic intermittently, for example at intervals of about one second, and output the first feature information, date and time information, and information regarding the spatial position to the storage unit 22. Note that the control unit 21 may delete from the storage unit 22 video that has passed a storage period that is preset according to the storage capacity of the storage unit 22, as well as the first feature information, date and time information, and information regarding the spatial position corresponding to that video.

In step S25, the control unit 41 of the terminal device 40 acquires the search target image in response to a user instruction. The control unit 41 can perform the processing of step S25 in the same manner as step S5 in FIG. 5A.

In step S26, the control unit 41 of the terminal device 40 accepts from the user a date and time range, which is the range of shooting dates and times for which the subject is to be searched, and a geographical range of the shooting location. The control unit 41 can perform the process of step S26 in the same manner as step S6 in FIG. 5A.

In step S27, the control unit 41 of the terminal device 40 transmits the search target image acquired in step S25 and the date and time range and geographic range acquired in step S26 to the server device 30, requesting a video search. The control unit 41 can perform the process of step S27 in the same manner as step S7 in FIG. 5A.

In step S28, the control unit 31 of the server device 30 analyzes the search target image received from the terminal device 40 in step S7 to acquire second feature information indicating features of the search target subject. The control unit 31 may acquire feature information of the same type as the first feature information as the second feature information using the same analysis logic as the first analysis logic used by the analysis device 20. Furthermore, the control unit 31 analyzes the search target image received from the terminal device 40 in step S7 based on the second analysis logic to acquire third feature information indicating appearance features unique to the search target subject. The second analysis logic is analysis logic optimized for acquiring feature information indicating more detailed features of the subject that are not extracted by the first analysis logic. The feature information acquired by the second analysis logic is, for example, the following information.
Detailed information about clothing: For example, detailed shapes, colors, patterns, etc. of clothing and shoes. For example, whether or not a specific brand logo or character is displayed.
Detailed information about belongings: For example, the detailed shape, color, and pattern of glasses, hats, bags, walking sticks, etc. For example, the presence or absence of a specific brand logo or character display, etc.
-Walking style, gestures, etc.: For example, walking habits such as dragging one leg, walking posture, etc.

The server device 30 may use a learning model previously acquired through machine learning as the second analysis logic for acquiring such more detailed feature information. The second analysis logic may be provided, for example, as parameters of a neural network. In this embodiment, the server device 30 analyzes the search target image to acquire the second and third feature information. However, the server device 30 may also acquire, from the terminal device 40, feature information obtained by analyzing the search target image using the same analysis logic by the terminal device 40 or another device, as the second and third feature information. The server device 30 may also receive a designation of analysis logic from the terminal device 40 and use the designated analysis logic as the second analysis logic to analyze the search target image. For example, when searching for a missing person dragging one leg, it is effective to perform a video search focusing on the gait. Therefore, the server device 30 may receive a designation of analysis logic from the terminal device 40 that performs analysis focusing on the gait, and use that analysis logic as the second analysis logic to analyze the search target image. The server device 30 may store multiple analysis logics that can be used as the second analysis logic in the storage unit 22 or the like, and read out a specified analysis logic to use as the second analysis logic. Alternatively, the server device 30 may receive analysis logic from another device such as the terminal device 40 and use that analysis logic as the second analysis logic.

In step S29, the control unit 31 of the server device 30 transmits the date and time range acquired in step S27, the second and third feature information acquired in step S28, and the analysis logic information used to acquire the third feature information to the analysis device 20 communicatively connected to the image capture devices 10 included in the geographical range acquired in step S27, and requests a video search. As in step S9 of FIG. 5B , the control unit 31 may identify the image capture devices 10 included in the geographical range acquired in step S27 by referring to information about the image capture locations of each image capture device 10 pre-stored in the memory unit 32. If the server device 30 does not hold information about the image capture locations of each image capture device 10, the server device 30 may transmit the date and time range, the second and third feature information, and the analysis logic to all analysis devices 20 and request a video search.

In step S30 of FIG. 6B, the control unit 21 of the analysis device 20 that has been requested to search for video extracts from the storage unit 22 first feature information that is similar to the received second feature information and date/time range. Specifically, for example, the control unit 21 may evaluate the similarity between each piece of first feature information for the specified date/time range stored in the storage unit 22 and the second feature information, and extract first feature information whose similarity is greater than a predetermined first threshold. The control unit 21 may perform the process of step S30 in the same manner as step S10 of FIG. 5B.

In step S31, the control unit 21 of the analysis device 20 acquires information regarding the spatial position associated with the video corresponding to the first feature information extracted in step S30. The control unit 21 analyzes the area of the video (frame image) corresponding to that spatial position using the analysis logic information (second analysis logic) received in step S29, and acquires fourth feature information related to the subject's unique appearance features. In this way, in this embodiment, the area to be analyzed based on the second analysis logic is limited using information regarding the spatial position, allowing the control unit 21 to perform analysis more efficiently than when analyzing the entire area of the video (frame image).

In step S32, the control unit 21 of the analysis device 20 extracts fourth feature information that is similar to the third feature information acquired in step S29. Specifically, for example, the control unit 21 may evaluate the similarity between each piece of fourth feature information acquired in step S31 and the third feature information, and extract fourth feature information whose degree of similarity (similarity) is greater than a predetermined second threshold.

In step S33, the control unit 21 of the analysis device 20 acquires, from the memory unit 22, the video corresponding to the fourth feature information extracted in step S32 as extracted video.

In step S34, the control unit 21 of the analysis device 20 transmits the extracted video acquired in step S33 to the server device 30. The control unit 21 may also transmit to the server device 30 the similarity degree acquired for the extracted video in steps S30 and S32, as well as information regarding the shooting date and time of the extracted video.

In step S35, the control unit 31 of the server device 30 transmits the extracted video received from the analysis device 20 in step S34 to the terminal device 40 as a video search result. The control unit 31 may also transmit to the terminal device 40 the similarity degree acquired in steps S30 and S32 for the extracted video, information related to the shooting date and time of the extracted video, and the shooting location (location information) of the imaging device 10.

In step S36, the control unit 41 of the terminal device 40 displays the extracted video received from the server device 30 in step S35 on the display unit of the output unit 45 as a search result. The user can check the search results for the search target subject that appears in the search target image by referring to the display unit of the output unit 45. The control unit 41 may also display the similarity, shooting date and time, and shooting location (location information) obtained in steps S30 and S32 for the extracted video on the display unit of the output unit 45. This allows the user to check when and where the search target subject was located.

(Example of analysis device operation)
7 is a flowchart showing an example of the operation of the analysis device 20 of FIG. 1. The operation of the analysis device 20 described with reference to FIG. 7 may correspond to at least a part of a control method for the analysis device 20. The operation of each step in FIG. 7 may be performed under control of the control unit 21 of the analysis device 20. FIG. 7 shows an example of processing performed by the analysis device 20 in the above-mentioned operation example 2 of the imaging system. The control unit 21 performs steps S42 and S43 every time it receives video from the imaging device 10, and performs steps S45 to S49 every time it receives a video search request from the server device 30, but the control unit 21 may perform these processes in parallel.

In step S41, the control unit 21 determines whether or not video has been received from the imaging device 10. If video has been received (YES in step S41), the control unit 21 proceeds to step S42; if not (NO in step S41), the control unit 21 proceeds to step S44.

In step S42, upon receiving video from the image capture device 10, the control unit 21 analyzes the video based on a first analysis logic that has been set in advance, and obtains first feature information that indicates the characteristics of the subject captured in the video, as well as information regarding the spatial position of the subject. The control unit 21 can perform the process of step S42 in the same manner as step S23 in FIG. 6A.

In step S43, the control unit 21 associates the image acquired from the image capture device 10, the first feature information of the image, date and time information indicating the date and time the image was captured, and information regarding the spatial position of the subject, and stores them in a storage device such as the memory unit 22. The control unit 21 can perform the process of step S43 in the same way as step S24 in FIG. 6A.

In step S44, the control unit 21 determines whether a video search request has been received from the server device 30. The search request may include the date and time range for the video search, the second feature information, the third feature information, and analysis logic. If the control unit 21 has received a search request (YES in step S44), it proceeds to step S45; otherwise (NO in step S44), it proceeds to step S50.

In step S45, the control unit 21 extracts from the storage unit 22 first feature information that is similar to the second feature information and date/time range received in step S44. Specifically, for example, the control unit 21 may evaluate the similarity between each piece of first feature information for the specified date/time range stored in the storage unit 22 and the second feature information, and extract first feature information whose similarity is greater than a predetermined first threshold. The control unit 21 may perform the process of step S30 in the same manner as step S10 in FIG. 5B and step S30 in FIG. 6B.

In step S46, the control unit 21 acquires information relating to the spatial position associated with the video corresponding to the first feature information extracted in step S45. The control unit 21 analyzes the area of the video (frame image) corresponding to that spatial position using the analysis logic information (second analysis logic) received in step S44, and acquires fourth feature information relating to the external features unique to the subject. The control unit 21 can perform the process of step S46 in the same way as step S31 in FIG. 6B.

In step S47, the control unit 21 extracts fourth feature information that is similar to the third feature information acquired in step S44. Specifically, for example, the control unit 21 may evaluate the similarity between each piece of fourth feature information acquired in step S46 and the third feature information, and extract fourth feature information whose degree of similarity (similarity) is greater than a predetermined second threshold. The control unit 21 may perform the process of step S47 in the same manner as step S32 in FIG. 6B.

In step S48, the control unit 21 obtains the video corresponding to the fourth feature information extracted in step S47 from the memory unit 22 as the extracted video.

In step S49, the control unit 21 transmits the extracted video acquired in step S48 to the server device 30. The control unit 21 may also transmit to the server device 30 the similarity degree acquired for the extracted video in steps S45 and S47, as well as information regarding the shooting date and time of the extracted video.

In step S50, the control unit 21 determines whether or not to end the processing of the flowchart. The control unit 21 may end the processing of the flowchart, for example, when instructed to end the processing by the user or an external device. If the processing is to be ended (YES in step S50), the control unit 21 ends the processing of the flowchart; if not (NO in step S50), the control unit 21 returns to step S41.

As described above, the photography system 1 comprises a photography device 10, an analysis device 20, a server device 30, and a terminal device 40. The photography device 10 captures video consisting of a plurality of frame images. The analysis device 20 is capable of communicating with the photography device 10. Upon acquiring video from the photography device 10, the analysis device 20 analyzes the video based on a first analysis logic to acquire first feature information indicating the features of the subject appearing in the video. The analysis device 20 associates the video with the first feature information and stores them in a storage device (e.g., storage unit 22). The server device 30 is capable of communicating with the analysis device 20. The terminal device 40 is capable of communicating with the server device 30. In this configuration, the server device 30 acquires second feature information indicating the features of the subject to be searched for, and transmits the acquired second feature information to the analysis device 20. In response to receiving second feature information from server device 30, analysis device 20 extracts first feature information similar to the second feature information from the storage device and transmits extracted video, which is video corresponding to the extracted first feature information, to server device 30. Server device 30 transmits the extracted video received from analysis device 20 to terminal device 40. Terminal device 40 displays the extracted video received from server device 30 on the display unit of output unit 45.

In this way, the analysis device 20 continuously analyzes the video received from the image capture device 10 and acquires the first feature information. In response to a search request including the second feature information, the analysis device 20 extracts video similar to the second feature information and transmits the corresponding video as extracted video to the terminal device 40 via the server device 30. The terminal device 40 displays the received video. Therefore, according to this embodiment, simply providing the analysis device 20 enables a search for video that includes a target subject, making it easy for the user of the terminal device 40 to review the searched video and find a specific subject. Here, the analysis device 20 transmits the video and first feature information to the server device 30 for video whose similarity to the second feature information is greater than the first threshold, rather than for all captured video, thereby minimizing the amount of data transferred over the network. Furthermore, because the server device 30 stores the first feature information acquired from the searched video, rather than for all captured video, the risk of personal information being leaked, such as the leakage of the first feature information from the server device 30, is reduced. This makes it possible to improve the technology for searching for specific subjects using video captured by the image capture device 10, for example, in terms of efficient network use and personal information protection.

In addition, in response to acquiring video from the imaging device 10, the analysis device 20 may further store date and time information indicating the date and time when the video was captured in the storage device, in association with the video and the first feature information. The server device 30 may further acquire a date and time range for the search in addition to the second feature information, and may further transmit the date and time range to the analysis device 20 in addition to the second feature information. In response to receiving the second feature information and the date and time range from the server device 30, the analysis device 20 may extract from the storage device first feature information of video captured in a date and time range similar to the second feature information. The analysis device 20 may transmit to the server device 30 extracted video, which is video corresponding to the extracted first feature information, and the date and time information of the extracted video. The server device 30 may transmit the extracted video and the date and time information of the extracted video received from the analysis device 20 to the terminal device 40. The terminal device 40 may display the extracted video and the date and time information of the extracted video received from the server device 30 on the display unit of the output unit 45.

In this way, the analysis device 20 searches by specifying the date and time range of the video's capture date and time, and by excluding from the search range the period during which the location of the search target was known, it is possible to efficiently search for video containing the search target. In addition, the terminal device 40 displays the date and time information of the extracted video. This makes it easy for the user to find the subject by checking when the video in which the search target subject appears was captured.

The photography system 1 may also include multiple photography devices 10 installed at multiple locations and multiple analysis devices 20 capable of communicating with one or more of the photography devices 10. The server device 30 may further acquire a geographical range to be searched in addition to the second feature information, and transmit the second feature information to analysis devices 20 capable of communicating with photography devices 10 installed at locations included in that geographical range. Upon receiving the second feature information from the server device 30, the analysis device 20 may extract first feature information of video similar to the second feature information from a storage device, and transmit the extracted video corresponding to the extracted first feature information and location information indicating the location of the photography device 10 that captured the extracted video to the server device 30. The server device 30 may transmit the extracted video and location information for the extracted video received from the analysis device 20 to the terminal device 40. The terminal device 40 may display the extracted video and location information for the extracted video received from the server device 30 on the display unit of the output unit 45.

In this way, the terminal device 40 displays location information for the extracted video. This allows the user to check where the video that captures the subject being searched for was shot, making it easier for the user to find the subject.

The terminal device 40 may also accept from the user a designation of a search target image that is an image that includes the subject to be searched. The server device 30 may analyze the search target image designated by the user and acquire, as the second feature information, feature information obtained.

In this way, the analysis device 20 searches for video images that include the subject to be searched for based on the feature information (second feature information) obtained by analyzing the search target image specified by the user. Therefore, the user can easily search for video images that include the subject and check the results by simply specifying the search target image. As mentioned above, the analysis of the search target image may be performed by the server device 30 or another device such as the terminal device 40.

Furthermore, in addition to the second feature information, the server device 30 may acquire third feature information indicating appearance features specific to the subject being searched, and transmit the third feature information to the analysis device 20 in addition to the second feature information. In response to receiving the second feature information and the third feature information from the server device 30, the analysis device 20 may extract first feature information similar to the second feature information from the storage device. The analysis device 20 may analyze the video corresponding to the extracted first feature information based on the second analysis logic, and acquire fourth feature information indicating appearance features specific to the subject appearing in the video. The analysis device 20 may extract fourth feature information similar to the third feature information, and transmit the video corresponding to the extracted fourth feature information to the server device 30 as extracted video.

In this way, the analysis device 20 searches for video containing the search target subject based on feature information extracted based on not only the first analysis logic but also the second analysis logic, making it possible to search for video that is similar in appearance to the search target subject.

Furthermore, in addition to the second feature information and the third feature information, the server device 30 may transmit to the analysis device 20 the analysis logic information used to extract the third feature information. The analysis device 20 may analyze the video corresponding to the extracted first feature information using the received analysis logic information as the second analysis logic, and obtain the fourth feature information.

In this way, the server device 30 transmits analysis logic information to be used as the second analysis logic to the analysis device 20. Therefore, the analysis device 20 can further narrow down and extract videos with similar features specific to the subject being searched from among videos with similar first feature information, which is general feature information.

Furthermore, in response to acquiring video from the imaging device 10, the analysis device 20 may analyze the video based on the first analysis logic to acquire first feature information, and may store the video, the spatial position in the video of the subject appearing in the video, and the first feature information in association with each other in the storage device. In response to receiving second feature information and third feature information from the server device 30, the analysis device 20 may extract first feature information similar to the second feature information from the storage device. The analysis device 20 may analyze an area corresponding to the spatial position in the video corresponding to the extracted first feature information based on the second analysis logic, and acquire feature information indicating the appearance characteristics unique to the subject appearing in that area of the video as fourth feature information.

In this way, by limiting the analysis target based on the second analysis logic to the area corresponding to the spatial position of the video, the analysis device 20 can reduce the computational costs required for analysis.

The present disclosure is not limited to the above-described embodiments. For example, multiple blocks shown in the block diagrams may be integrated, or one block may be divided. Multiple steps shown in the flowcharts may be executed in parallel or in a different order, depending on the processing capabilities of the device executing each step, or as needed, instead of being executed chronologically as described. Other modifications are possible without departing from the spirit of the present disclosure.

REFERENCE SIGNS LIST 1 Photography system 10 Photography device 20 Analysis device 21 Control unit 22 Storage unit 23 Communication unit 30 Server device 31 Control unit 32 Storage unit 33 Communication unit 40 Terminal device 41 Control unit 42 Storage unit 43 Communication unit 44 Input unit 45 Output unit N Network

Claims (14)

an imaging device that captures a video consisting of a plurality of frame images;
an analysis device including a first control unit capable of communicating with the image capturing device, which, in response to acquiring the image from the image capturing device, analyzes the image based on a first analysis logic to acquire first feature information indicating features of a subject captured in the image, and stores the image and the first feature information in a storage device in association with each other;
a server device capable of communicating with the analysis device;
a terminal device capable of communicating with the server device;
Equipped with
the server device includes a second control unit that acquires second feature information indicating features of the subject to be searched and transmits the acquired second feature information to the analysis device;
the first control unit of the analysis device, in response to receiving the second feature information from the server device, extracts from the storage device the first feature information whose similarity to the second feature information is greater than a predetermined first threshold, and transmits an extracted video that is a video corresponding to the extracted first feature information to the server device;
the second control unit of the server device transmits the extracted video received from the analysis device to the terminal device;
the terminal device includes a third control unit that causes the extracted video received from the server device to be displayed on a display unit.
Shooting system. the first control unit of the analysis device, in response to acquiring the video from the imaging device, further stores date and time information indicating a date and time when the video was captured in the storage device in association with the video and the first feature information;
the second control unit of the server device further acquires a date and time range to be searched in addition to the second characteristic information, and further transmits the date and time range to the analysis device in addition to the second characteristic information;
The first control unit of the analysis device
In response to receiving the second feature information and the date and time range from the server device, extracting from the storage device the first feature information of the video captured in the date and time range, the first feature information having a similarity to the second feature information greater than the first threshold;
transmitting to the server device an extracted video that is a video corresponding to the extracted first feature information, the date and time information of the extracted video, and a similarity between the first feature information and the second feature information;
the second control unit of the server device transmits the extracted video, the date and time information of the extracted video, and the similarity received from the analysis device to the terminal device;
the third control unit of the terminal device causes the display unit to display the extracted video, the date and time information of the extracted video, and the similarity received from the server device.
The imaging system according to claim 1 . A plurality of the imaging devices provided at a plurality of locations;
a plurality of analysis devices each capable of communicating with one or more of the imaging devices included in the plurality of imaging devices;
Equipped with
the second control unit of the server device further acquires a geographical range to be searched in addition to the second feature information, and transmits the second feature information to the analysis device that is capable of communicating with the imaging device installed at a location included in the geographical range;
the first control unit of the analysis device that has received the second feature information from the server device extracts, from the storage device, the first feature information of the video whose similarity to the second feature information is greater than the first threshold, and transmits, to the server device, the extracted video corresponding to the extracted first feature information and location information indicating a location of the imaging device that captured the extracted video;
the second control unit of the server device transmits the extracted video and the location information of the extracted video received from the analysis device to the terminal device;
the third control unit of the terminal device causes the display unit to display the extracted video and the location information of the extracted video received from the server device.
The imaging system according to claim 1 . the third control unit of the terminal device receives, from a user, a designation of a search target image that is an image in which the subject to be searched is reflected;
the second control unit of the server device analyzes the search target image designated by the user and acquires, as the second feature information, feature information acquired by analyzing the search target image designated by the user;
The imaging system according to claim 1 . the second control unit of the server device acquires, in addition to the second feature information, third feature information indicating an appearance feature specific to the subject to be searched, and transmits, in addition to the second feature information, the third feature information to the analysis device;
The first control unit of the analysis device
extracting, from the storage device, the first feature information whose similarity to the second feature information is greater than the first threshold value in response to receiving the second feature information and the third feature information from the server device;
analyzing the image corresponding to the extracted first feature information based on a second analysis logic to obtain fourth feature information indicating external features specific to the subject captured in the image;
The photographing system according to claim 1, further comprising: extracting the fourth feature information whose similarity to the third feature information is greater than a predetermined second threshold; and transmitting the image corresponding to the extracted fourth feature information as the extracted image to the server device. the second control unit of the server device transmits, to the analysis device, analysis logic information used in extracting the third feature information, in addition to the second feature information and the third feature information;
the first control unit of the analysis device analyzes the video corresponding to the extracted first feature information by using the analysis logic information as the second analysis logic, and acquires the fourth feature information.
The imaging system according to claim 5 . The first control unit of the analysis device
In response to acquiring the video from the photographing device, analyze the video based on the first analysis logic to acquire the first feature information, and store the video, a spatial position in the video of the subject captured in the video, and the first feature information in a storage device in association with each other;
extracting, from the storage device, the first feature information whose similarity to the second feature information is greater than the first threshold value in response to receiving the second feature information and the third feature information from the server device;
analyzing an area of the image corresponding to the spatial position of the extracted first feature information based on the second analysis logic, and acquiring feature information indicating a feature of the subject reflected in the area of the image as the fourth feature information;
The imaging system according to claim 5 . an imaging device that captures a video consisting of a plurality of frame images;
an analysis device including a first control unit capable of communicating with the image capturing device, which, in response to acquiring the image from the image capturing device, analyzes the image based on a first analysis logic to acquire first feature information indicating features of a subject captured in the image, and stores the image and the first feature information in a storage device in association with each other;
a server device capable of communicating with the analysis device;
a terminal device capable of communicating with the server device;
A control method for an imaging system comprising:
a second control unit of the server device acquires second feature information indicating features of the subject to be searched, and transmits the acquired second feature information to the analysis device;
the first control unit of the analysis device, in response to receiving the second feature information from the server device, extracts from the storage device the first feature information whose similarity to the second feature information is greater than a predetermined first threshold, and transmits an extracted video that is a video corresponding to the extracted first feature information to the server device;
the second control unit of the server device transmits the extracted video received from the analysis device to the terminal device;
a third control unit of the terminal device causing a display unit to display the extracted video received from the server device;
A method for controlling the imaging system. An imaging device that captures a video consisting of a plurality of frame images, and an analysis device that can communicate with a server device,
In response to acquiring the image from the photographing device, analyze the image based on a first analysis logic to acquire first feature information indicating features of a subject captured in the image, and store the image and the first feature information in a storage device in association with each other;
In response to receiving second feature information indicating features of the subject to be searched from the server device, extracting from the storage device the first feature information whose similarity to the second feature information is greater than a predetermined first threshold, and transmitting an extracted image that is an image corresponding to the extracted first feature information to the server device.
An analysis device having a control unit. The control unit
In response to acquiring the video from the imaging device, date and time information indicating the date and time when the video was captured is further stored in the storage device in association with the video and the first feature information;
In response to receiving from the server device the date and time range to be searched in addition to the second feature information, extracting from the storage device the first feature information of the video captured in the date and time range, the first feature information having a similarity to the second feature information greater than the first threshold;
transmitting an extracted video corresponding to the extracted first feature information and the date and time information of the extracted video to the server device;
The analysis device according to claim 9. The control unit
extracting, from the storage device, the first feature information whose similarity to the second feature information is greater than the first threshold value in response to receiving, in addition to the second feature information, third feature information indicating an appearance feature specific to the subject to be searched, in addition to the second feature information;
analyzing the image corresponding to the extracted first feature information based on a second analysis logic to obtain fourth feature information indicating external features specific to the subject captured in the image;
The analysis device according to claim 9, further comprising: extracting the fourth feature information whose similarity to the third feature information is greater than a predetermined second threshold; and transmitting the video corresponding to the extracted fourth feature information to the server device as the extracted video. The control unit
and in response to receiving, from the server device, analysis logic information used in extracting the third feature information in addition to the second feature information and the third feature information, analyzing the video corresponding to the extracted first feature information using the analysis logic information as the second analysis logic to acquire the fourth feature information.
The analysis device according to claim 11. The control unit
In response to acquiring the video from the photographing device, analyze the video based on the first analysis logic to acquire the first feature information, and store the video, a spatial position in the video of the subject captured in the video, and the first feature information in a storage device in association with each other;
extracting, in response to receiving the second feature information and the third feature information from the server device, the first feature information whose similarity to the second feature information is greater than the first threshold value from the storage device;
analyzing an area of the image corresponding to the spatial position of the extracted first feature information based on the second analysis logic, and acquiring feature information indicating appearance features specific to the subject reflected in the area of the image as the fourth feature information;
The analysis device according to claim 11. A photographing device that photographs a video consisting of a plurality of frame images, and a computer that can communicate with a server device,
In response to acquiring the image from the photographing device, analyze the image based on a first analysis logic to acquire first feature information indicating features of a subject captured in the image, and store the image and the first feature information in a storage device in association with each other;
In response to receiving second feature information indicating features of the subject to be searched from the server device, extracting from the storage device the first feature information whose similarity to the second feature information is greater than a predetermined first threshold, and transmitting an extracted image that is an image corresponding to the extracted first feature information to the server device.
A program for operating an analysis device equipped with a control unit.