JP7559511B2 - WORK ESTIMATION DEVICE, CONTROL METHOD FOR WORK ESTIMATION DEVICE, INFORMATION PROCESSING PROGRAM, AND RECORDING MEDIUM - Google Patents

Description

The present invention relates to a task estimation device that estimates, from an image of a workplace, the task being performed by a worker captured in the image.

Conventionally, there are known task estimation devices and methods for estimating the content of tasks performed by workers at production sites and the like. For example, Patent Document 1 listed below discloses a task recognition method for identifying which of a set task types the task being performed by an imaged worker corresponds to, based on the body parts and objects of the worker detected from camera footage.

International Publication No. WO2018/087844

However, because the above task recognition method identifies task types on a rule-based basis, as the number of types of tasks performed by workers increases, there is a problem in that it is cumbersome to manually prepare in advance the rules required to identify task types.

One aspect of the present invention aims to realize a task estimation device that can estimate the tasks being performed by a worker without the need for complicated rule development, even if the types of tasks performed by the worker increase.

In order to solve the above problem, a task estimation device according to one aspect of the present invention is a task estimation device that estimates the content of a task being performed by a worker captured in a captured image from a captured image of a workplace, and includes: a detection unit that detects the worker captured in the captured image and the task object captured in the captured image from the captured image; a determination unit that determines the presence or absence of an interaction between the worker captured in the captured image and the task object captured in the captured image from the captured image or the feature information using a determination model that is a trained model that receives the captured image or feature information generated from the captured image as an input and outputs the presence or absence of an interaction between the worker captured in the captured image and the task object captured in the captured image, based on the captured image or the feature information; and an estimation unit that estimates the content of the task being performed by the worker on the task object for a combination of the worker and the task object determined by the determination unit to have an interaction.

According to the above configuration, the work estimation device uses the judgment model to judge whether or not there is an interaction between the worker and the work object detected from the captured image. Then, for a combination of the worker and the work object for which it is judged that there is an interaction, the work estimation device estimates the content of the work being performed by the worker on the work object.

In other words, the task estimation device determines whether or not there is interaction between the worker and the task object detected from the captured image on a model-based basis, rather than a rule-based basis.

The work estimation device thus has the effect of being able to estimate the work being performed by a worker, even if the number of types of work being performed by the worker increases, without the need to prepare complicated rules for estimating the content of the work.

The task estimation device also estimates the content of the task that the worker is performing on the task object for a combination of the worker and the task object that is determined to have an interaction based on a model.

In other words, the task estimation device estimates the content of the task being performed by the worker only for combinations of the worker and the task object for which it is determined that an interaction exists, rather than for all combinations of the worker and the task object captured in the captured image.

Therefore, the task estimation device has the effect of reducing the amount of calculation required to estimate the task that the worker captured in the captured image is performing on the task object captured in the captured image.

In a task estimation device according to one aspect of the present invention, the detection unit may detect the worker captured in the captured image and the task object captured in the captured image from the captured image using a detection model that is a trained model that receives the captured image as an input and outputs the worker captured in the captured image and the task object captured in the captured image.

According to the above configuration, the task estimation device uses the detection model to detect from the captured image the worker captured in the captured image and the task object captured in the captured image.

In other words, the task estimation device detects the worker captured in the captured image and the task object captured in the captured image from the captured image in a model-based manner, rather than a rule-based manner.

The work estimation device therefore has the effect of being able to detect the worker captured in the captured image and the work object captured in the captured image from the captured image without the need for complicated rule development.

In the work estimation device according to one aspect of the present invention, the estimation unit may use an estimation model that is a trained model that receives the captured image or feature information generated from the captured image as an input and outputs the content of the work that the worker captured in the captured image is performing on the work object captured in the captured image, for a combination of the worker and the work object for which an interaction is determined to exist by the determination unit, to estimate the content of the work that the worker is performing on the work object from the captured image or the feature information.

According to the above configuration, the task estimation device uses the estimation model to estimate the content of the task that the worker is performing on the task object for a combination of the worker and the task object for which it is determined that an interaction exists.

In other words, the task estimation device estimates the content of the task that the worker is performing on the task object based on a model rather than a rule base, for the worker and the task object detected from the captured image.

The work estimation device thus has the effect of being able to estimate the work being performed by a worker, even if the number of types of work being performed by the worker increases, without the need to prepare complicated rules for estimating the content of the work.

The work estimation device according to one aspect of the present invention may operate in a normal mode or a transfer learning mode, and in the normal mode, the judgment unit may use the judgment model constructed by learning from a dataset collected in a certain factory to judge whether or not there is an interaction between the worker and the work object for the captured image taken in the workplace of the certain factory, and in the transfer learning mode, the judgment unit may use the judgment model constructed by learning from a dataset collected in a certain factory to judge whether or not there is an interaction between the worker and the work object for the captured image taken in the workplace of another factory different from the certain factory.

According to the above configuration, the work estimation device operates in a normal mode or a transfer learning mode. In the normal mode, the factory where the captured image, for which the presence or absence of an interaction between the worker and the work object is determined, was taken and the factory where the dataset for constructing the judgment model was collected are the same factory. In the transfer learning mode, the factory where the captured image, for which the presence or absence of an interaction between the worker and the work object is determined, was taken and the factory where the dataset for constructing the judgment model was collected are not the same factory.

Here, the developer of the task estimation device confirmed that the presence or absence of interaction between the worker captured in the captured image and the task object captured in the captured image can be determined sufficiently independently of the influence of the environment in which the captured image was captured.

In other words, the judgment model for determining whether or not an interaction exists can be constructed by learning from a data set collected in one factory, and can be used to determine whether or not an interaction exists in the images captured in another factory.

Therefore, in the transfer learning mode, the task estimation device has the effect of being able to determine the presence or absence of an interaction for the image captured in a certain factory by using the determination model constructed by learning from a dataset collected in another factory.

The work estimation device according to one aspect of the present invention may further include an acquisition unit that acquires, from a control device that controls the operation of equipment installed in the workplace, process information generated from operation results indicating the content and results of the operation performed by the equipment, and a generation unit that generates support information, which is information that can be used to support the efficient execution of the work by the worker, from the content of the work of the worker estimated by the estimation unit and the process information acquired by the acquisition unit.

According to the above configuration, the work estimation device acquires from the control device process information generated from operation results indicating the content and results of operations performed by equipment installed in the workplace. The work estimation device then generates support information, which is information that can be used to support the efficient execution of the work by the worker, from the content of the work of the worker estimated from the captured image and the process information.

Therefore, the work estimation device has the effect of generating support information that can be used to support the worker in efficiently performing the work by combining the content of the work of the worker estimated from the captured image with the process information.

In order to solve the above problem, a control method according to one aspect of the present invention is a control method for a work estimation device that estimates the content of work being performed by a worker captured in a captured image of a workplace from the captured image, and includes a detection step of detecting the worker captured in the captured image and the work object captured in the captured image from the captured image, a determination step of determining the presence or absence of an interaction between the worker captured in the captured image and the work object captured in the captured image from the captured image or the feature information using a determination model that is a trained model that receives as input the captured image or feature information generated from the captured image and outputs the presence or absence of an interaction between the worker captured in the captured image and the work object captured in the captured image, from the captured image or the feature information, and an estimation step of estimating the content of the work being performed by the worker on the work object for a combination of the worker and the work object determined to have an interaction by the determination step.

According to the above configuration, the control method uses the judgment model to judge whether or not there is an interaction between the worker and the work object detected from the captured image. Then, for a combination of the worker and the work object for which it is judged that there is an interaction, the control method estimates the content of the work being performed by the worker on the work object.

In other words, the control method determines whether or not there is interaction between the worker and the work object detected from the captured image on a model-based basis, rather than a rule-based basis.

Therefore, the control method has the effect of being able to estimate the work being performed by a worker without the need to prepare complicated rules for estimating the content of the work, even if the number of types of work performed by the worker increases.

The control method also estimates the content of the work being performed by the worker on the work object for a combination of the worker and the work object that is determined to have an interaction based on a model.

In other words, the control method estimates the work being performed by the worker only for combinations of the worker and the work object for which it is determined that an interaction exists, rather than for all combinations of the worker and the work object captured in the captured image.

The control method thus has the effect of reducing the amount of calculation required to estimate the work being performed by the worker captured in the captured image on the work object captured in the captured image.

According to one aspect of the present invention, it is possible to estimate the work being performed by a worker without the need for complicated rule development, even if the number of types of work performed by the worker increases.

1 is a block diagram showing a configuration of a main part of an information processing device according to a first embodiment of the present invention; FIG. 2 is a diagram showing an overall outline of a control system including the information processing device of FIG. 1 . FIG. 1 is a diagram for explaining a problem that a general HOI detection algorithm has. FIG. 13 is a diagram illustrating an example of an analysis unit realized as a neural network. 2 is a diagram illustrating an overview of a process executed by the information processing apparatus in FIG. 1 in a normal mode. FIG. 2 is a diagram illustrating an overview of a process executed by the information processing device in FIG. 1 in a transfer learning mode. FIG. 2 is a flow diagram illustrating an overview of an analysis process executed by the information processing device of FIG. 1 .

[Embodiment 1]
Hereinafter, an embodiment according to one aspect of the present invention (hereinafter also referred to as "the present embodiment") will be described with reference to the drawings. Note that the same or corresponding parts in the drawings are given the same reference numerals and their description will not be repeated. In the present embodiment, an information processing device 10 will be described as a typical example of "an operation estimation device that estimates, from an image Im obtained by capturing an image of a workplace Ar, the content of an operation Op being performed by a worker Hu captured in an image Im."

In the following description, "n", "m", "p", "q", "x", and "y" each represent an integer greater than or equal to 1, with "p" and "q" being different integers and "x" and "y" being different integers.

§1. Application Example In order to facilitate understanding of the information processing device 10 (task estimation device) according to one aspect of the present invention, first, an environment in which the information processing device 10 is applied will be described. For example, the information processing device 10 uses the analysis results of the captured image Im to generate information (support information) that contributes to the efficiency of a task Op performed by a worker Hu on a task object Ob. First, a worker Hu, a task object Ob, a task Op, a work site Ar, and a factory Fa including one or more work sites Ar will be described.

(About the factory and work process)
In a factory Fa, which is a production site, for example, various products are produced through one or more work processes Pr(1), Pr(2), Pr(3), ..., Pr(n). Each of the multiple work processes Pr(1), Pr(2), Pr(3), ..., Pr(n) carried out in the factory Fa is, for example, a "painting" process, a "main work assembly" process, a "installation of the main work into the main body" process, and an "inspection" process.

When it is necessary to distinguish between multiple factories Fa, they will be distinguished by adding suffixes such as "(A)", "(B)", "(C)", ..., "(Z)" to the reference numerals. For example, they will be distinguished by notation such as "Factory Fa(A)", "Factory Fa(B)", "Factory Fa(X)", ..., "Factory Fa(Z)". When there is no particular need to distinguish between each of the multiple factories Fa, they will simply be referred to as "Factory Fa".

In addition, when it is necessary to distinguish between multiple work processes Pr, they are distinguished by adding suffixes such as "(1)", "(2)", "(3)", ..., "(n)" to the reference numeral. For example, they are distinguished by notation such as "work process Pr(1)", "work process Pr(2)", "work process Pr(3)", ..., "work process Pr(n)". When there is no particular need to distinguish between multiple work processes Pr, they are simply referred to as "work process Pr".

(About the work)
In a work process Pr, a worker Hu performs one or more operations Op(1), Op(2), Op(3), ... Op(n). For example, in a work process Pr(x), a worker Hu performs operations Op(x1), Op(x2), and Op(x3). For example, when a work process Pr(m) is a "painting" process, the operation Op(x1) may be "holding" the work object Ob, the operation Op(x2) may be "painting" the work object Ob, and the operation Op(x3) may be "transporting" the work object Ob.

When it is necessary to distinguish between multiple work Ops, they will be distinguished by adding suffixes such as "(1)", "(2)", "(3)", ..., "(n)" to the reference numerals. When there is no particular need to distinguish between them, they will simply be referred to as "work Ops."

(About the workplace)
The factory Fa includes one or more work sites Ar(1), Ar(2), Ar(3), ..., Ar(n). Each of the one or more work sites Ar(1), Ar(2), Ar(3), ..., Ar(n) corresponds to one or more work processes Pr(1), Pr(2), Pr(3), ..., Pr(n). In other words, the work site Ar(m) is an area where a worker Hu performs an operation Op(m) included in the work process Pr(m).

When it is necessary to distinguish between multiple workplaces Ar, they will be distinguished by adding suffixes such as "(1)", "(2)", "(3)", ..., "(n)" to the reference numerals, and when there is no particular need to distinguish between them, they will simply be referred to as "workplace Ar".

(About the work object)
In the workplace Ar, one or more work objects Ob that are the targets of work Op performed by a worker Hu are placed. For example, in the workplace Ar(x), work objects Ob(x1), Ob(x2), Ob(x3), ..., Ob(xn) are placed.

The work object Ob is an entity other than the worker Hu, for example, a so-called "workpiece," or, for example, a tool used when performing the work Op.

When it is necessary to distinguish between multiple work objects Ob, the reference numerals are suffixed with letters such as "(1)", "(2)", "(3)", ..., "(n)" to distinguish between them. When there is no particular need to distinguish between them, they are simply referred to as "work objects Ob".

(About the workers)
In the factory Fa, there are one or more workers Hu who perform the work Op included in the work process Pr, for example, workers Hu(1), Hu(2), Hu(3), ..., Hu(n). The workers Hu may be identified, for example, by a worker ID attached to the top of the hat worn by the worker Hu. Specifically, the workers Hu(1) and Hu(2) in the factory Fa may be identified by the worker ID(1) attached to the hat worn by the worker Hu(1) and the worker ID(2) attached to the hat worn by the worker Hu(2). In addition, a certain worker Hu(1) may be regarded as a work object Ob for another worker Hu(2).

When it is necessary to distinguish between multiple workers Hu, the reference numbers are supplemented with suffixes such as "(1)", "(2)", "(3)", ..., "(n)" to distinguish between them. When there is no particular need to distinguish between them, they are simply referred to as "workers Hu."

(About equipment and operation)
The factory Fa includes one or more pieces of equipment 40(1), 40(2), 40(3), ... 40(n). When performing an operation Op, a worker Hu may use the equipment 40, and specifically, the worker Hu may perform the operation Op by using an operation Ac executed by the equipment 40. For example, when the worker Hu performs an operation Op(x) included in a work process Pr(x), the worker Hu may use the equipment 40(x), and specifically, the worker Hu may perform the operation Op(x) by using an operation Ac(x) executed by the equipment 40(x).

When it is necessary to distinguish between multiple devices 40, the devices 40 are distinguished by adding suffixes such as "(1)", "(2)", "(3)", ..., "(n)" to the reference numerals, and when there is no particular need to distinguish between the devices 40, they are simply referred to as "devices 40".

Similarly, when it is necessary to distinguish between multiple operations Ac, they are distinguished by adding suffixes such as "(1)", "(2)", "(3)", ..., "(n)" to the reference numerals, and when there is no particular need to distinguish between them, they are simply referred to as "operations Ac".

(Overview of the system including the information processing device 10)
In the environment described above, the information processing device 10 estimates, from a captured image Im of the workplace Ar, the content of the operation Op being performed by the worker Hu captured in the captured image Im.

When distinguishing between the factories Fa in which each of the multiple captured images Im was captured, the multiple captured images Im are described as follows to distinguish between them. For example, an image Im captured of the work area Ar of factory Fa(A) is described as captured image Im(A), and an image Im captured of the work area Ar of factory Fa(B) is described as captured image Im(B) to distinguish between the factories Fa in which the captured images Im were captured. When there is no need to distinguish between the factories Fa in which each of the multiple captured images Im was captured, they are simply referred to as "captured images Im".

First, the following describes an overview of the control system 1 including the information processing device 10 using FIG. 2.

Figure 2 shows an overview of the control system 1 including the information processing device 10.

The ceiling camera 30 is installed, for example, on the ceiling of the work area Ar, and generates a captured image Im looking down diagonally on the work area Ar. However, it is not necessary to install a ceiling camera 30 for each work area Ar. One ceiling camera 30, which is a wide-area imaging camera, may be installed on the ceiling of the factory Fa, and the entire factory Fa may be captured from above by the single ceiling camera 30. In other words, the single ceiling camera 30 may overlook the entire factory Fa, including one or more work areas Ar, and generate a captured image Im that captures the entire factory Fa. The ceiling camera 30 outputs the generated captured image Im to the information processing device 10.

The information processing device 10 is included in the control system 1 and is communicatively connected to the ceiling camera 30 via, for example, a USB (Universal Serial Bus) cable. The information processing device 10 acquires an image Im of the workplace Ar from the ceiling camera 30. The information processing device 10 performs image analysis on the image Im acquired from the ceiling camera 30 and estimates the content of the work Op being performed by the worker Hu captured in the image Im. The information processing device 10 outputs the estimated content of the work Op to the outside.

The information processing device 10 also acquires process information generated from an operation result La from a PLC (Programmable Logic Controller) 20, the operation result La indicating the content and result of an operation Ac executed by a device 40 whose operation Ac is controlled by the PLC 20.

(Master-slave control system)
In the control system 1, the devices 40 used to perform the work process Pr in the factory Fa are controlled by the PLC 20 as a line controller. That is, the control system 1 is constructed as a master-slave control system in which the PLC 20 is the master and each of the devices 40 is a slave, and each of the devices 40 is communicatively connected to the PLC 20 via a network (control network 50). The PLC 20 is called a "master" in the sense that it manages data transmission via the control network 50. The terms "master" and "slave" are defined with a focus on the control function of data transmission on the control network 50, and there is no particular limitation on the type of information transmitted and received between the devices.

PLC 20 is a control device (controller) that controls the entire control system 1, and is connected to each of the multiple devices 40 so as to be able to communicate with them. PLC 20 acquires information from each of the multiple devices 40 as input devices (measurement devices) as input data. PLC 20 executes arithmetic processing using the acquired input data according to a user program previously installed. PLC 20 executes the arithmetic processing to determine the control content for the control system 1, for example, determines the control content for each of the multiple devices 40 as output devices such as actuators, and outputs control data corresponding to the control content to each of the multiple devices 40. PLC 20 repeatedly acquires input data from each of the multiple devices 40 and acquires control data for each of the multiple devices 40 at a predetermined period (control period). For example, a display unit and an operation unit (not shown) may be connected to PLC 20. The display unit is composed of a liquid crystal panel capable of displaying images, and the operation unit is typically composed of a touch panel, a keyboard, a mouse, etc.

The device 40 is a slave in the control system 1 as a master-slave control system with the PLC 20 as the master. The device 40 is an input device that repeatedly transmits input data to the PLC 20 at a predetermined control period, or an output device that repeatedly receives control data from the PLC 20 at a predetermined control period and operates according to the received control data. The device 40 may be, for example, a sensor (e.g., a photoelectric sensor) as an input device that transmits detection results, etc., as input data to the PLC 20, a barcode reader that transmits read results, or an inspection machine (tester) that transmits inspection results. The device 40 may also be a PT (Programmable Terminal) to which multiple input devices are connected. Furthermore, the device 40 may be a robot, etc., as an output device that performs screw tightening, picking, etc.

The control network 50 transmits various data received by the PLC 20 or transmitted by the PLC 20, and typically may use various types of industrial Ethernet (registered trademark), and may also be referred to as a field network. Known industrial Ethernet (registered trademark) include, for example, EtherCAT (registered trademark), Profinet IRT, MECHATROLINK (registered trademark)-III, Powerlink, SERCOS (registered trademark)-III, and CIP Motion, and any of these may be adopted. Furthermore, a field network other than the industrial Ethernet (registered trademark) may also be used. For example, if motion control is not performed, DeviceNet, CompoNet/IP (registered trademark), etc. may be used.

In this embodiment, a control system 1 is described in which data frames are transferred sequentially over a control network 50, thereby transmitting and receiving data between a PLC 20 (master) and a device 40 (slave). That is, data frames are transferred sequentially over the control network 50 at a predetermined control period, thereby repeatedly transmitting and receiving data between the PLC 20 and the device 40 at each control period. Data may be transmitted and received between multiple devices 40, i.e., multiple slaves, by transferring data frames sequentially over the control network 50.

The information processing device 10 is included in the control system 1, but the information processing device 10 does not have to be a slave in the control system 1.

(Regarding process information)
In the control system 1, which is a master-slave control system in which the device 40 is a slave, the PLC 20, which is the master, receives an operation result La from the device 40, which is a slave, for example, repeatedly at a predetermined control period. The operation result La is information indicating "the content and result of the operation Ac performed by the device 40." That is, the device 40 transmits the operation result La, which indicates the content and result of the operation Ac actually performed when performing the work process Pr, to the PLC 20 at a predetermined period. For example, the device 40(x) transmits the operation result La(x), which indicates the content and result of the operation Ac(x) performed when performing the work process Pr(x), to the PLC 20 at a control period.

The PLC 20 acquires, for example, the measurement result, which is the result of the measurement operation performed by the equipment 40 as an input device (measuring device), as the operation result La of the equipment 40. Furthermore, if the equipment 40 is an inspection machine, the PLC 20 acquires the result of the inspection operation performed by the equipment 40, for example, the inspection result, such as "whether the inspection criteria were met or not met," as the operation result La of the equipment 40. Furthermore, the PLC 20 acquires, for example, the result of the output operation performed by the equipment 40 as an output device, as the operation result La of the equipment 40. If the equipment 40 is a robot that performs screw tightening, picking, etc., the PLC 20 acquires, for example, the operation result La of the equipment 40, such as the number of screw tightenings, the picking result (picking success or picking error), etc.

The PLC 20 repeatedly receives operation results La from the equipment 40 at a predetermined cycle, the operation results La indicating the content and results of the operations Ac actually performed by the equipment 40 when carrying out the work process Pr, and transmits (i.e., transfers) the received operation results La to the information processing device 10 as process information. The PLC 20 also transmits information generated using the operation results La repeatedly received at a predetermined cycle from the equipment 40 to the information processing device 10 as process information.

Furthermore, the PLC 20 may transmit the operation result La repeatedly received from the device 40 at a predetermined cycle to the outside of the control system 1 as process information. For example, the PLC 20 may transmit the operation result La repeatedly received from the device 40 at a predetermined cycle to an in-house LAN (Local Area Network) shown in FIG. 2 connected to a MES (Manufacturing Execution System) or the like as process information.

The information processing device 10 identifies the operation start time Tms, operation completion time Tme, and operation period Da of the operation Ac performed by the equipment 40 when carrying out the work process Pr from the process information (operation result La) acquired from the PLC 20. Then, the information processing device 10 performs various judgments on the operation Ac using the identified operation start time Tms, operation completion time Tme, operation period Da, and a predetermined operation standard Sa corresponding to the operation Ac.

However, the above-mentioned judgment on the operation result La (particularly, operation Ac) included in the process information may be performed by PLC 20, and PLC 20 may transmit the result of the above-mentioned judgment to information processing device 10 by including it in the process information or instead of the process information.

In the above description, the operation start time Tms is the time when the device 40 used in the work process Pr starts to perform the operation Ac when carrying out the work process Pr, and the operation completion time Tme is the time when the operation Ac is completed. The operation period Da is the period from the operation start time Tms to the operation completion time Tme.

(For systems and devices other than master-slave control systems)
2 shows an in-house LAN system, other network systems, etc., in addition to the control system 1 as a master-slave control system. The in-house LAN is connected to a process information DB (Database) also called MES. The process information DB stores information indicating "standard operations that should be performed by the equipment 40 used to perform the work process Pr" as an operation standard Sa.

In the example shown in FIG. 2, an event management device 60 that monitors and manages various events that occur in the factory Fa is connected to the process information DB as the MES via an in-house LAN. However, it is not essential that the event management device 60 is connected to the process information DB via an in-house LAN, and the event management device 60 does not have to be provided.

Furthermore, the process information DB is connected to the PLC 20 via an in-house LAN. Although not shown, the process information DB and the information processing device 10 may be connected. In addition to the MES, the in-house LAN may also be connected to an ERP (Enterprise Resources Planning), WMS (Warehouse Management System), etc. (not shown).

In FIG. 2, the process information DB is connected to a video storage server, etc., via an "other network" different from the control network 50 and the in-house LAN. The information processing device 10 is connected to the video storage server, etc., via the other network, and the captured image Im transmitted from the information processing device 10, the contents of the work Op estimated by the information processing device 10, and support information are stored in the video storage server, etc. In addition, an external device 70 realized by a PC (Personal Computer) or the like is connected to the video storage server, etc., and the external device 70 displays, for example, the captured image Im, the contents of the work Op estimated by the information processing device 10, and support information, etc., and performs visualization of process information, etc. In other words, the external device 70 displays a list of information necessary for improving the work process Pr, and displays information indicating the work process Pr that is a bottleneck, the date and time of an error that occurred in the work process Pr, etc., in association with the corresponding captured image Im.

As described above, the ceiling camera 30 captures an image of the work area Ar to generate a captured image Im, and transmits the generated captured image Im to the information processing device 10 via a communication cable, such as a USB (Universal Serial Bus) cable.

The information processing device 10 is realized by, for example, a PC or the like, and is a data extraction device that combines process information acquired from the PLC 20 and captured images Im acquired from the ceiling camera 30 to enable efficient use of both. The information processing device 10 visualizes the work process Pr by combining process information including "operation results La showing the contents and results of actual operations Ac of multiple devices 40 in the factory Fa" with "captured images Im" and "contents of work Op estimated from captured images Im". The information processing device 10 visualizes the work process Pr by combining process information acquired from the PLC 20, captured images Im acquired from the ceiling camera 30, and contents of work Op estimated from captured images Im, and also analyzes the work process Pr efficiently and precisely.

For example, the information processing device 10, in addition to visualizing the process information, extracts the work process Pr that is a bottleneck, and combines an image Im capturing an image of the implementation status of the extracted work process Pr with the process information of the extracted work process Pr. Therefore, for example, the user can easily identify the cause of the bottleneck and the cause of the defect for the work process Pr that is a bottleneck, and can efficiently perform improvement work for the work process Pr. In addition, the information processing device 10 can also be used for traceability when a defect occurs.

In addition, the information processing device 10 can improve the accuracy of both the analysis of the captured image Im and the analysis of the process information by combining the captured image Im with the process information of each of the multiple devices 40. For example, the information processing device 10 uses the process information (operation result La) acquired from the PLC 20 to identify the operation start time Tms at which the device 40 starts the operation Ac and the operation completion time Tme at which the operation Ac is completed. The information processing device 10 can perform an analysis of the captured image Im captured during the operation period Da, which is the period from the operation start time Tms to the operation completion time Tme, to associate the operation Ac of the device 40 during the operation period Da with the work Op of the worker Hu with high accuracy.

Furthermore, the information processing device 10 acquires an action standard Sa indicating "standard actions that should be performed by the equipment 40 used to perform the work process Pr" from the process information DB. Then, the information processing device 10 uses the acquired action standard Sa to perform a judgment on "actual actions Ac performed by the equipment 40 used to perform the work process Pr". The information processing device 10 may perform an analysis on the imaging data Id capturing an image of the implementation status of the work process Pr corresponding to the actions Ac that are determined to be different from the standard actions indicated by the action standard Sa.

(Estimation of work content)
The information processing device 10 analyzes the captured image Im to estimate the content of the work Op performed by the worker Hu on the work object Ob. The work Op performed by the worker Hu in the work area Ar is generally a work Op performed on the work object Ob. Therefore, the work Op can be regarded as an interaction between the worker Hu and the work object Ob for a combination (pair) of the worker Hu and the work object Ob.

Here, conventionally, methods have been studied for detecting a HOI (Human Object Interaction), which is an interaction between a human captured in a captured image Im and an object captured in the captured image Im, from a captured image Im. For example, HOI detection is known that localizes both a human and an object and identifies the interaction between the two. An example of an algorithm for such HOI detection is iCAN (Instance-Centric Attention Network for Human-Object Interaction Detection).

The information processing device 10 can use an HOI detection algorithm such as iCAN to estimate the content of the work Op that the worker Hu is performing on the work object Ob. That is, the information processing device 10 can estimate the content of the work Op that the worker Hu is performing on the work object Ob for the "pair of the worker Hu and the work object Ob" captured in the captured image Im by using an HOI detection algorithm that detects the HOI.

However, general HOI detection algorithms such as iCAN consider all possible pairs of all people and all objects detected in the captured image Im, and predict the content of the interaction between each pair based on the appearance features and rough spatial information, etc. For this reason, general HOI detection algorithms have problems as described using Figure 3.

(Problems with General HOI Detection Algorithms)
Fig. 3 is a diagram for explaining the problems with a general HOI detection algorithm such as iCAN. The captured image Im shown in Fig. 3 is a captured image Im capturing a situation in which two brothers (workers Hu(1) and Hu(2)) are on the left and three cows (work objects Ob(1), OB(2), and OB(3)) are on the right, separated by a fence. In the captured image Im shown in Fig. 3, the younger brother is looking at the cow in the foreground, and the older brother is offering food to the second cow from the front.

In the example shown in FIG. 3, two people (workers Hu(1) and Hu(2)) and four objects (work objects Ob(1), OB(2), OB(3), and OB(4)) are detected from the captured image Im. Note that the work object Ob(4) is an object that has been erroneously detected from the captured image Im.

As described above, a typical HOI detection algorithm considers all possible pairs of all humans and all objects detected from the captured image Im, classifies the HOI, which is the interaction between each pair, and predicts (estimates) the content of the HOI. Therefore, for the captured image Im shown in FIG. 3, a typical HOI detection algorithm considers a total of eight pairs of two humans and four objects, and estimates the content of the HOI, which is the interaction between the human and the object, for each of the eight considered pairs.

However, in the captured image Im shown in FIG. 3, there are only two pairs in which there is actual interaction between a human and an object: "pair of worker Hu(1) and work object Ob(2)" and "pair of worker Hu(2) and work object Ob(3)." Although there is no need to estimate the HOI content for the other six pairs, a typical HOI detection algorithm attempts to estimate the HOI content for these six pairs as well. Therefore, a typical HOI detection algorithm has the problem that it takes a long time to detect the HOI from the captured image Im.

In addition, general HOI detection algorithms have the problem of inferring the HOI content even for "human-object pairs" where there is no actual interaction between the two, resulting in reduced estimation accuracy. For example, general HOI detection algorithms infer the HOI content for pairs other than the "pair of worker Hu(1) and work object Ob(2)" and the "pair of worker Hu(2) and work object Ob(3)" for the captured image Im in FIG. 3, resulting in reduced estimation accuracy.

(Extraction of Estimation Target by Information Processing Device According to the Present Embodiment)
Therefore, the information processing device 10 executes the following process to improve the estimation accuracy of the estimation process for estimating the contents of the HOI (i.e., the task Op). That is, the information processing device 10 first recognizes (determines) whether or not there is actually an interaction between the "human (worker Hu) captured in the captured image Im" and the "object (work target Ob) captured in the captured image Im". Then, the information processing device 10 estimates the contents of the operation (HOI) that the human is performing with respect to the object only for the pair of the human and the object that is determined to have an interaction between them.

In other words, before estimating the content of the "operation Op as HOI" for a human-object pair, the information processing device 10 employs a filter that excludes human-object pairs that are determined to have "no interaction between the two," i.e., executes the following determination process.

That is, the information processing device 10 determines "whether or not there is an interaction between the worker Hu and the work object Ob" for each of all pairs, each of which is composed of a worker Hu detected from the captured image Im and a work object Ob detected from the captured image Im. In other words, the information processing device 10 extracts only "pairs in which there is an interaction between the worker Hu and the work object Ob" from among all pairs, each of which is composed of a worker Hu detected from the captured image Im and a work object Ob detected from the captured image Im.

For example, the information processing device 10 detects two workers Hu and four work objects Ob from the captured image Im in FIG. 3, and detects a total of eight pairs, each consisting of a worker Hu and a work object Ob. The information processing device 10 determines the presence or absence of an interaction between the worker Hu and the work object Ob for each of the above-mentioned eight pairs on a model basis. Then, the information processing device 10 extracts only two pairs in which there is an interaction between the worker Hu and the work object Ob from the above-mentioned eight pairs. For the captured image Im illustrated in FIG. 3, the information processing device 10 extracts a pair of a worker Hu (1) and a work object Ob (2) and a pair of a worker Hu (2) and a work object Ob (3) from the above-mentioned eight pairs.

Then, the information processing device 10 performs HOI detection only for the "pair of worker Hu and work object Ob" that is determined to have "interaction between them," that is, estimates the content of the work Op that the worker Hu is performing on the work object Ob. In the example of FIG. 3, the information processing device 10 performs HOI detection only for the "pair of worker Hu (1) and work object Ob (2)" and the "pair of worker Hu (2) and work object Ob (3)."

The information processing device 10 does not perform HOI detection for all "pairs of a worker Hu and a work object Ob" detected from the captured image Im, but performs HOI detection only for "pairs of a worker Hu and a work object Ob" between which there is an interaction.

Therefore, the information processing device 10 can reduce the amount of calculations related to HOI detection and shorten the time required for HOI detection. In other words, the information processing device 10 can reduce the amount of calculations required when estimating the content of the work Op that the worker Hu captured in the captured image Im is performing on the work object Ob captured in the captured image Im, shorten the time required for the estimation, and speed up the estimation process.

In addition, the information processing device 10 can improve the accuracy of HOI detection by performing HOI detection (estimation of the content of the work Op) only for pairs of workers Hu and work objects Ob that have interactions between them.

(About transfer learning)
The information processing device 10 executes a determination process and an estimation process, which are distinguished from each other, in an analysis process for estimating the content of the work Op performed by the worker Hu on the work object Ob from the captured image Im. The determination process is a process for estimating "the presence or absence of an interaction between the worker Hu captured in the captured image Im and the work object Ob captured in the captured image Im". The estimation process is a process for estimating "the content of the work Op performed by the worker Hu captured in the captured image Im on the work object Ob captured in the captured image Im". In particular, the estimation process is a process for estimating the content of the "work Op as HOI" only for "a pair of the worker Hu captured in the captured image Im and the work object Ob captured in the captured image Im" that is determined by the determination process to "have an interaction between the two".

When the analysis unit 120 that performs the analysis process is realized as a neural network (NN), the information processing device 10 realizes the following effects by distinguishing between the judgment unit 122 (judgment model 152) related to the judgment process and the estimation unit 123 (estimation model 153) related to the estimation process. That is, the information processing device 10 enables transfer learning using multiple data sets DS of different domains by distinguishing between the judgment model 152 and the estimation model 153, each of which is a trained model, in the analysis unit 120 realized as a NN. The analysis unit 120 has a network design that allows the judgment model 152 (A) constructed by machine learning for "data set DS (A) prepared in factory Fa (A)" to be used for analysis of the captured image Im (C) captured in factory Fa (C). Details will be described later.

(Overview of the information processing device according to this embodiment)
An overview of the information processing device 10 will be provided in advance to facilitate understanding of the information processing device 10, which will be described in detail below with reference to FIG.

To generate support information that can be used to assist workers Hu in efficiently performing operations Op, it is useful to visualize process information in the factory Fa.

Conventionally, there is a known method for recognizing (estimating) the content of an action being performed by a human (worker Hu) from pose estimation information that estimates the pose (posture) of the human. However, in the factory Fa, the worker Hu is basically continuing some kind of work Op. Therefore, in order to divide the process, that is, to estimate the content of the work Op being performed by the worker Hu, it is necessary to take into account not only the pose of the worker Hu, but also information on the interference between the worker Hu and an object (work target Ob).

Therefore, the information processing device 10 regards the work Op that the worker Hu is performing on the work object Ob as an interaction (HOI) between the worker Hu and the work object Ob. Then, the information processing device 10 performs, for example, HOI detection, an estimation of the content of the work Op that the worker Hu is performing on the work object Ob.

However, conventional methods that only recognize human actions (tasks Op) through HOI, which is the interference and interaction between a human (worker Hu) and an object (object, work subject Ob), have the following problems:

That is, in the conventional method, in order to recognize the work Op of a worker Hu in each of the multiple factories Fa, data must be collected in each of the multiple factories Fa to recognize the work Op performed by the worker Hu. That is, in the conventional method, for example, in order to recognize the work Op of a worker Hu in factory Fa(A), data must be collected in factory Fa(A) to recognize the work Op performed by the worker Hu. Similarly, in the conventional method, in order to recognize the work Op of a worker Hu in factory Fa(B), data must be collected in factory Fa(B) to recognize the work Op performed by the worker Hu. That is, in the conventional method, in order to recognize the work Op of a worker Hu in factory Fa(X), data must be collected in factory Fa(X) to recognize the work Op performed by the worker Hu.

The information processing device 10 accumulates knowledge that can be used for HOI detection, that is, knowledge for estimating the contents of the work Op, and, for example, collects data for learning in at least one of the factories Fa(A) and Fa(B). The information processing device 10 can then divert the learned model (specifically, the judgment model 152) constructed by machine learning on the collected data to the estimation of the contents of the work Op in the factory Fa(C).

When the analysis unit 120 that performs the analysis process is realized as a neural network, the information processing device 10 achieves the following effects by separating the network related to the judgment process (judgment model 152) from the network related to the estimation process (estimation model 153). That is, the information processing device 10 enables transfer learning of rule information (knowledge) for estimating work Op, and for example, knowledge learned in a certain factory Fa (X) can be diverted to the estimation of work Op in another factory Fa (Y).

Also, conventionally, a rule-based method is known for recognizing (estimating) the content of an action (task Op) being performed by a human (worker Hu) from the interference (interaction) between the human (worker Hu) and an object (work object Ob).

However, as the number of types of actions (tasks Op) increases, the number of rules that need to be prepared in advance to estimate the content of an action for an HOI pair (i.e., a pair of a human (worker Hu) and an object (work object Ob)) rapidly increases, and creating such rules manually becomes cumbersome.

Therefore, the information processing device 10 removes "pairs of worker Hu and work object Ob" that do not have interactions from all possible "pairs of worker Hu and work object Ob" on a model basis. After that, the information processing device 10 performs HOI detection (i.e., estimation of the content of the work Op) on a model basis, for example, only for "pairs of worker Hu and work object Ob" that have interactions.

The information processing device 10 eliminates "pairs of workers Hu and work objects Ob" that do not have interactions on a model basis, eliminating the need for the complicated rule preparation required by conventional methods for such elimination.

In addition, the information processing device 10 can improve the accuracy of estimating the content of the work Op by removing "pairs of a worker Hu and a work object Ob" where no interaction exists.

Furthermore, by removing "pairs of worker Hu and work object Ob" where no interaction exists, the information processing device 10 can reduce the amount of calculations required to estimate the content of the work Op and shorten the time required to estimate the content of the work Op. In other words, the information processing device 10 can increase the calculation speed by extracting "pairs of worker Hu and work object Ob" for which estimation of the content of the work Op is required, and estimating the content of the work Op only for the extracted "pairs of worker Hu and work object Ob."

The information processing device 10 differs from conventional methods that "use information that estimates a human pose (posture) and information about an object to analyze (estimate) an action (task Op) in a rule-based manner, or analyze an action by combining multiple methods" in the following ways. That is, the information processing device 10 realizes end-to-end task estimation processing, for example, by using a neural network. In addition, the information processing device 10 employs a network structure that enables transfer learning, and can improve the estimation accuracy related to the estimation of the content of a task Op by reusing existing knowledge.

The information processing device 10 does not make a rule-based determination of the presence or absence of an interaction between "the worker Hu captured in the captured image Im" and "the work object Ob captured in the captured image Im." The information processing device 10 discovers and generates rules relating to the presence or absence of an interaction between the worker Hu and the work object Ob by machine learning, and uses these rules to determine the presence or absence of an interaction between the captured worker Hu and the work object Ob for the captured image Im.

The information processing device 10 detects all "pairs of workers Hu and work objects Ob" captured in the captured image Im, and automatically extracts from among them only "pairs of workers Hu and work objects Ob" that have interactions between them on a model-based basis.

(Summary of the information processing device according to this embodiment)
The contents described above with reference to Figures 2 and 3 can be summarized as follows: That is, the information processing device 10 is an operation estimation device that estimates, from an image Im capturing an image of a workplace Ar, the contents of an operation Op being performed by a worker Hu captured in the captured image Im, and includes a detection unit 121, a determination unit 122, and an estimation unit 123.

The detection unit 121 detects from the captured image Im the worker Hu captured in the captured image Im and the work object Ob captured in the captured image Im.

The determination unit 122 determines, on a model basis, from the captured image Im or its feature information, whether or not there is an interaction between the worker Hu and the work object Ob detected by the detection unit 121. The determination unit 122 determines, on the basis of the captured image Im or its feature information, whether or not there is an interaction between the worker Hu and the work object Ob detected by the detection unit 121, using the determination model 152, which is a trained model. The determination model 152 is a trained model that receives as input the "captured image Im or feature information generated from the captured image Im" and outputs the "existence of an interaction between the worker Hu captured in the captured image Im and the work object Ob captured in the captured image Im."

The estimation unit 123 estimates the content of the work Op that the worker Hu is performing on the work object Ob for a combination (pair) of the worker Hu and the work object Ob for which the determination unit 122 has determined that an interaction exists.

According to the above configuration, the information processing device 10 uses the judgment model 152, which is a trained model, to judge whether or not there is an interaction between a "pair of a worker Hu and a work object Ob" detected from the captured image Im. Then, for the "pair of a worker Hu and a work object Ob" for which it is judged that there is an interaction, the information processing device 10 estimates the content of the work Op being performed by one of the pair, the worker Hu, on the other of the pair, the work object Ob.

In other words, the information processing device 10 determines whether or not there is an interaction between the worker Hu and the work object Ob detected from the captured image Im on a model-based basis, rather than a rule-based basis.

Therefore, the information processing device 10 has the effect of being able to estimate the work Op being performed by the worker Hu, even if the number of types of work Op being performed by the worker Hu increases, without the need to prepare complicated rules for estimating the content of the work Op.

In addition, for a pair of worker Hu and work object Ob that is determined to have "an interaction" based on the model, the information processing device 10 estimates the content of the work Op that the worker Hu is performing on the work object Ob.

In other words, the information processing device 10 estimates the content of the work Op being performed by the worker Hu only for pairs determined to have "an interaction" rather than for all pairs of the worker Hu and the work object Ob captured in the captured image Im.

Therefore, the information processing device 10 has the effect of reducing the amount of calculation required to estimate the content of the work Op that the worker Hu captured in the captured image Im is performing on the work object Ob captured in the captured image Im.

In the information processing device 10, the detection unit 121 detects "the worker Hu captured in the captured image Im" and "the work object Ob captured in the captured image Im" from the captured image Im using the detection model 151, which is a trained model. The detection model 151 is a trained model that receives the captured image Im as input and outputs information related to the "worker Hu captured in the captured image Im" and information related to the "work object Ob captured in the captured image Im".

According to the above configuration, the information processing device 10 uses the detection model 151 to detect "the worker Hu captured in the captured image Im" and "the work object Ob captured in the captured image Im" from the captured image Im.

In other words, the information processing device 10 detects "the worker Hu captured in the captured image Im" and "the work object Ob captured in the captured image Im" from the captured image Im on a model-based basis, not a rule-based basis.

Therefore, the information processing device 10 has the effect of being able to detect "the worker Hu captured in the captured image Im" and "the work object Ob captured in the captured image Im" from the captured image Im without the need for complicated rule development.

In the information processing device 10, the estimation unit 123 uses the estimation model 153, which is a trained model, to estimate the content of the work Op being performed by one worker Hu, who is one of a pair determined by the determination unit 122 that "an interaction exists," on the work object Ob, who is the other of the pair. The estimation model 153 is a trained model that receives the "captured image Im or its feature information" as input and outputs the "content of the work Op being performed by the worker Hu captured in the captured image Im on the work object Ob captured in the captured image Im."

According to the above configuration, for a pair determined to have an "interaction," the information processing device 10 uses the estimation model 153 to estimate the content of the work Op being performed by one worker Hu of the pair on the other work object Ob of the pair.

In other words, the information processing device 10 estimates the content of the work Op that the worker Hu is performing on the work object Ob, based on the worker Hu and the work object Ob detected from the captured image Im, on a model basis, not a rule basis.

Therefore, the information processing device 10 has the effect of being able to estimate the work Op being performed by the worker Hu, even if the number of types of work Op being performed by the worker Hu increases, without the need to prepare complicated rules for estimating the content of the work Op.

The information processing device 10 further includes a process information acquisition unit 130 (acquisition unit) and a support information generation unit 140 (generation unit). The process information acquisition unit 130 acquires process information generated from an operation result La indicating the content and result of the operation Ac performed by the equipment 40 from a PLC 20 (control device) that controls the operation Ac of the equipment 40 installed in the work area Ar. The support information generation unit 140 generates support information that is information that can be used to support the efficient execution of the work Op by the worker Hu from the content of the work Op of the worker Hu estimated by the estimation unit 123 and the process information acquired by the process information acquisition unit 130.

According to the above configuration, the information processing device 10 acquires process information generated from "operation result La indicating the content and result of operation Ac performed by equipment 40 installed in the work area Ar" from the PLC 20. Then, the information processing device 10 generates support information, which is information that can be used to support the efficient execution of the work Op by the worker Hu, from the "content of the work Op of the worker Hu" estimated from the captured image Im and the process information.

Therefore, the information processing device 10 has the effect of generating support information that can be used to support the efficient execution of the work Op by the worker Hu by combining the "contents of the work Op of the worker Hu" estimated from the captured image Im with the process information.

§2. Configuration Example The information processing device 10, the outline of which has been explained above, will now be explained in detail with reference to FIG.

Figure 1 is a block diagram showing the main components of an information processing device. The information processing device 10 shown in Figure 1 includes, as functional blocks, an image acquisition unit 110, an analysis unit 120, a process information acquisition unit 130, a support information generation unit 140, and a storage unit 150, for example.

In addition to the above-mentioned functional blocks, the information processing device 10 may include the following units. That is, the information processing device 10 may include a management unit that controls the storage (saving) of the captured image Im in a predetermined storage area (storage area), a measurement unit that measures the time (stay time) that the worker Hu stays in the workplace Ar, an analysis unit that associates process information with the stay time, etc.

To ensure conciseness of description, configurations of the information processing device 10 that are not directly related to this embodiment have been omitted from the description and block diagram. However, depending on the actual implementation situation, the information processing device 10 may be provided with these omitted configurations.

The above-mentioned functional blocks of the information processing device 10 can be realized, for example, by the arithmetic unit reading out a program stored in a storage device (storage unit 150) realized by a ROM (read only memory), NVRAM (non-volatile random access memory), etc., into a RAM (random access memory) (not shown) or the like, and executing the program. Examples of devices that can be used as the arithmetic unit include a CPU (Central Processing Unit), a GPU (Graphic Processing Unit), a DSP (Digital Signal Processor), an MPU (Micro Processing Unit), an FPU (Floating point number Processing Unit), a PPU (Physics Processing Unit), a microcontroller, or a combination of these.

First, the image acquisition unit 110, analysis unit 120, process information acquisition unit 130, and support information generation unit 140 will be described in detail below.

(Function blocks other than the memory unit)
The captured image acquisition unit 110 acquires an image Im captured by the ceiling camera 30 of the work area Ar from the ceiling camera 30 , and outputs the acquired captured image Im to the detection unit 121 .

The analysis unit 120 analyzes the captured image Im captured by the ceiling camera 30, and executes an analysis process that estimates "the content of the work Op being performed by the worker Hu captured in the captured image Im on the work object Ob captured in the captured image Im." The analysis unit 120 then outputs the estimated content of the work Op as an analysis result for the captured image Im, for example, to the support information generation unit 140 and the video storage server of FIG. 2. The analysis unit 120 may output the analysis result for the captured image Im to the video storage server of FIG. 2 together with the captured image Im.

The analysis unit 120, for example, performs a model-based analysis on the captured image Im, and outputs "the content of the work Op being performed by the worker Hu captured in the captured image Im on the work object Ob captured in the captured image Im." The analysis unit 120 is realized, for example, as a neural network (NN).

The analysis unit 120 includes a detection unit 121, a determination unit 122, and an estimation unit 123.

The detection unit 121 extracts characteristic information (feature amount) of the captured image Im from the captured image Im, and specifically outputs "information related to the worker Hu (body part of the worker Hu) captured in the captured image Im" and "information related to the work object Ob captured in the captured image Im." In other words, the detection unit 121 detects "the worker Hu (body part of the worker Hu) captured in the captured image Im" and "the work object Ob captured in the captured image Im" from the captured image Im.

"Information related to the worker Hu captured in the captured image Im" is information indicating, for example, "the presence or absence (whether or not the worker Hu is captured in the captured image Im), the spatial arrangement (position in space), and the posture of the worker Hu captured in the captured image Im."

"Information related to the work object Ob captured in the captured image Im" is, for example, information indicating "the presence or absence of the work object Ob captured in the captured image Im, its spatial arrangement, and what the work object Ob captured in the captured image Im is."

The detection unit 121 uses the detection model 151 acquired by referring to the memory unit 150 to extract feature information of the captured image Im from the captured image Im, that is, to detect the "worker Hu and work object Ob" captured in the captured image Im.

The detection unit 121 outputs the feature information of the captured image Im extracted from the captured image Im to the determination unit 122 and the estimation unit 123. In other words, the detection unit 121 notifies the determination unit 122 and the estimation unit 123 of the "worker Hu (body part of the worker Hu) captured in the captured image Im" and the "work object Ob captured in the captured image Im" detected from the captured image Im.

The determination unit 122 performs the following determination for the pair of "worker Hu captured in the captured image Im" and "work object Ob captured in the captured image Im" using the captured image Im or the feature information of the captured image Im extracted by the detection unit 121. That is, the determination unit 122 determines whether or not there is an interaction between the pair of "worker Hu captured in the captured image Im" and "work object Ob captured in the captured image Im". In other words, the determination unit 122 determines whether or not there is an interaction between the "worker Hu captured in the captured image Im" and the "work object Ob captured in the captured image Im" detected from the captured image Im by the detection unit 121.

The determination unit 122 uses the determination model 152 acquired by referring to the memory unit 150 to determine whether or not there is a relationship between the "worker Hu and the work object Ob" detected from the captured image Im, based on the captured image Im or the feature information of the captured image Im.

The determination unit 122 notifies the estimation unit 123 of the result of the determination, and outputs to the estimation unit 123, for example, a pair of "worker Hu captured in the captured image Im" and "work object Ob captured in the captured image Im" that is determined to have an interaction between the two.

The estimation unit 123 uses the captured image Im or the feature information of the captured image Im extracted by the detection unit 121 to estimate "the content of the work Op being performed by the worker Hu captured in the captured image Im on the work object Ob captured in the captured image Im."

The estimation unit 123 executes the above-mentioned estimation using the estimation model 153 acquired by referring to the storage unit 150. That is, the estimation unit 123 uses the estimation model 153 to estimate "the content of the work Op being performed by the worker Hu captured in the captured image Im on the work object Ob captured in the captured image Im" from the captured image Im or the feature information of the captured image Im.

In particular, the estimation unit 123 performs the above estimation only for the pair of "worker Hu captured in captured image Im" and "work object Ob captured in captured image Im" that is determined by the determination unit 122 to have "an interaction between the two." In other words, the estimation unit 123 estimates the content of the work Op that one worker Hu of the pair is performing on the other work object Ob of the pair only for the pair that is determined to have "an interaction between the two."

The process information acquisition unit 130 acquires process information from the control device, PLC 20, and outputs the acquired process information to the support information generation unit 140.

The support information generating unit 140 uses the content of the work Op (A) estimated by the analysis unit 120 (particularly the estimation unit 123) and the process information to generate support information that can be used to support the efficient execution of the work Op by the worker Hu. The support information generating unit 140 outputs the generated support information, for example, to the video storage server of FIG. 2.

(Regarding the memory unit)
The storage unit 150 is a storage device that stores various data used by the information processing device 10. The storage unit 150 may non-temporarily store (1) a control program, (2) an OS program, (3) application programs for executing various functions possessed by the information processing device 10, and (4) various data read when executing the application programs, which are executed by the information processing device 10. The above data (1) to (4) are stored in a non-volatile storage device such as a read only memory (ROM), a flash memory, an erasable programmable ROM (EPROM), an electrically EPROM (EEPROM (registered trademark)), or a hard disk drive (HDD). The information processing device 10 may include a temporary storage unit (not shown). The temporary storage unit is a so-called working memory that temporarily stores data used for calculations and calculation results, etc., during the course of various processes executed by the information processing device 10, and is composed of a volatile storage device such as a random access memory (RAM). Which data is to be stored in which storage device is appropriately determined based on the purpose of use, convenience, cost, physical constraints, etc. of the information processing device 10. The storage unit 150 further stores a detection model 151, a determination model 152, and an estimation model 153.

The detection model 151 is a model for the detection unit 121 to extract feature information (feature amount) of the captured image Im from the captured image Im. In other words, the detection model 151 is a model for the detection unit 121 to detect "the worker Hu (body part of the worker Hu) captured in the captured image Im" and "the work object Ob captured in the captured image Im" from the captured image Im.

The detection model 151 is a trained model that, for example, receives an image Im as input and outputs "information related to the worker Hu (body part of the worker Hu) captured in the image Im" and "information related to the work object Ob captured in the image Im."

The judgment model 152 is a model for the judgment unit 122 to "use the captured image Im or feature information of the captured image Im to judge whether or not there is an interaction between the "pair of worker Hu and work object Ob captured in the captured image Im." In other words, the judgment model 152 is a model for the judgment unit 122 to "determine whether or not there is an interaction between the "worker Hu detected from the captured image Im by the detection unit 121" and the "work object Ob detected from the captured image Im by the detection unit 121."

The judgment model 152 is a trained model that, for example, takes a captured image Im (feature information of the captured image Im) as input and outputs information indicating "whether or not there is an interaction between the worker Hu captured in the captured image Im and the work object Ob captured in the captured image Im."

The estimation model 153 is a model that allows the estimation unit 123 to estimate the following content for "the pair of worker Hu and work object Ob captured in the captured image Im" using the captured image Im or feature information of the captured image Im. In other words, the estimation model 153 is a model that allows the estimation unit 123 to estimate "the content of the work Op being performed by the worker Hu captured in the captured image Im on the work object Ob captured in the captured image Im."

The estimation model 153 is a trained model that, for example, takes a captured image Im (feature information of the captured image Im) as input and outputs information indicating "the content of the work Op being performed by the worker Hu captured in the captured image Im on the work object Ob captured in the captured image Im."

For the information processing device 10 having the above configuration, the relationship between the detection unit 121 (detection model 151), judgment unit 122 (judgment model 152), and estimation unit 123 (estimation model 153) when the analysis unit 120 that performs the analysis process is realized as a neural network will be outlined using Figure 4.

(An example of an analysis part implemented as a neural network (NN))
4 is a diagram illustrating an example of the analysis unit 120 realized as a neural network. As described above, the analysis unit 120 may be realized as a neural network, and in particular, may be realized as a convolutional neural network (CNN) or a deep neural network (DNN). The analysis unit 120 realized as a neural network includes a feature extractor R not shown in FIG. 4, an interaction determiner P in FIG. 4, and a HOI classifier C in FIG. 4.

The feature extractor R (Representation network (feature extractor)) is, for example, Faster R-CNN, performs object detection, and corresponds to the detection unit 121 (detection model 151). The feature extractor R extracts feature information (feature) of the captured image Im from the captured image Im. Specifically, the feature extractor R inputs the captured image Im to the detection model 151, which is a trained model. The detection model 151, which has received the captured image Im as input, outputs the feature information of the captured image Im, specifically, outputs "information related to the worker Hu (body part of the worker Hu) captured in the captured image Im" and "information related to the work object Ob captured in the captured image Im".

In the example shown in FIG. 4, in response to input of a captured image Im, the following information is output as feature information of the captured image Im. That is, information (Pose Map) indicating the posture of the worker Hu captured in the captured image Im is output as feature information of the captured image Im. In addition, information (Spatial Maps) indicating the spatial arrangement of each of the worker Hu and the work object Ob captured in the captured image Im is output as feature information of the captured image Im. In addition, other information (Human Feature) relating to the worker Hu captured in the captured image Im and other information (Object Feature) relating to the work object Ob captured in the captured image Im are output as feature information of the captured image Im.

The interaction determiner P (interactive predictor) in FIG. 4 corresponds to the determination unit 122 (determination model 152). The interaction determiner P determines whether or not there is an interaction between the "pair of worker Hu and work object Ob" captured in the captured image Im from the feature information of the captured image Im, and outputs the determination result. Specifically, the interaction determiner P inputs the feature information of the captured image Im to the determination model 152, which is a trained model. The determination model 152, which has received the input of the feature information of the captured image Im, determines whether or not there is an interaction between the "pair of worker Hu and work object Ob" captured in the captured image Im, and outputs the determination result.

The interaction determiner P performs the following determination for the captured image Im illustrated in FIG. 4 using the determination model 152. That is, the interaction determiner P first detects two pairs captured in the captured image Im, specifically, a pair of a worker Hu (1) and a work object Ob (1) and a pair of a worker Hu (1) and a work object Ob (2). For each of the two pairs, the interaction determiner P determines whether or not there is an interaction between the worker Hu, one of the pair, and the work object Ob, the other of the pair. The interaction determiner P determines that there is an interaction between the worker Hu and the work object Ob for the pair of the worker Hu (1) and the work object Ob (1). The interaction determiner P determines that there is no interaction between the worker Hu and the work object Ob for the pair of the worker Hu (1) and the work object Ob (2). The interaction determiner P outputs the determination result to the HOI classifier C, and notifies the HOI classifier C of, for example, only the "pair of worker Hu(1) and work object Ob(1)" for which it has determined that "there is an interaction between the worker Hu and the work object Ob."

The interaction determiner P may further use the following values for the output of the determination model 152 that receives the input of feature information of the captured image Im to determine whether or not there is an interaction between the "pair of worker Hu and work object Ob" captured in the captured image Im. That is, the interaction determiner P may use a value obtained by converting the score of object detection by the feature extractor R, etc., into a "value between 0 and 1" using logistic regression. The object detection score is, for example, a score that indicates the possibility of the presence or absence of each of the worker Hu and work object Ob detected from the captured image Im.

The HOI classifier C in FIG. 4 is, for example, iCAN, and corresponds to the estimation unit 123 (estimation model 153). The HOI classifier C outputs information indicating "the content of the work Op being performed by the worker Hu captured in the captured image Im on the work object Ob captured in the captured image Im" from the feature information of the captured image Im. Specifically, the HOI classifier C inputs the feature information of the captured image Im to the estimation model 153, which is a trained model. The estimation model 153, which has received the input of the feature information of the captured image Im, outputs information indicating "the content of the work Op being performed by the worker Hu captured in the captured image Im on the work object Ob captured in the captured image Im".

Here, the HOI classifier C uses the output of the interaction determiner P to extract "pairs of a worker Hu and a work object Ob, for which the estimation model 153 estimates the content of the work Op as the HOI between the two." In other words, the HOI classifier C estimates the content of the work Op that the worker Hu is performing on the work object Ob as the HOI between the two only for "pairs of a worker Hu and a work object Ob" that are determined to have "an interaction between the two."

Specifically, the HOI classifier C calculates a score for "the work Op performed by the worker Hu on the work object Ob" as the HOI between the two only for "pairs of a worker Hu and a work object Ob" that are determined to have "an interaction between the two." For the captured image Im shown in the example of FIG. 4, the HOI classifier C detects the HOI only for "pairs of a worker Hu (1) and a work object Ob (1)" that are determined to have "an interaction between the two" by the interaction determiner P, and outputs, for example, the HOI score.

In FIG. 4, the following values are shown as examples of scores for the operation Op (i.e., HOI) performed by worker Hu (1) on work object Ob (1). That is, examples are shown in which the score for "Carry" is "0.96," the score for "Hold" is "0.89," the score for "Paint" is "0.92," and the score for "Ride" is "0.03."

In the analysis of the captured image Im by the analysis unit 120, prior to the estimation process by the HOI classifier C, the interaction determiner P determines whether or not estimation process is required for the "pairs of worker Hu and work object Ob" detected from the captured image Im. In other words, in the analysis unit 120, the interaction determiner P eliminates pairs that do not require estimation process by the HOI classifier C from all "pairs of worker Hu and work object Ob" in the captured image Im that can be assumed from the feature information extracted from the captured image Im by the feature extractor R. Then, the HOI classifier C detects the HOI between the two only for the remaining pairs, that is, estimates the content of the work Op that the worker Hu is performing on the work object Ob (performs verb estimation).

Therefore, the analysis unit 120 can reduce the computational cost of the estimation process by the HOI classifier C (i.e., the estimation unit 123) in advance, and as a result, the computational speed of the entire analysis process of the captured image Im can be improved.

Next, the learning process for constructing the trained models, detection model 151, judgment model 152, and estimation model 153, will be described below.

(Learning process)
The analysis unit 120 realized as a neural network constructs each of the detection model 151, the judgment model 152, and the estimation model 153, which are trained models, by, for example, the following method. That is, the analysis unit 120 constructs the detection model 151, the judgment model 152, and the estimation model 153 by supervised learning on a data set DS, which is a collection of "captured images Im or data in which feature information of the captured images Im is given a predetermined label." Hereinafter, the "captured images Im or data in which feature information of the captured images Im is given a predetermined label" is referred to as "learning data." In addition, the process in which "the analysis unit 120 constructs the detection model 151, the judgment model 152, and the estimation model 153 by supervised learning" is referred to as a "learning process."

For example, the learning data is data in which the captured image Im is labeled with the following information. That is, in the learning data, the captured image Im is labeled with information related to the worker Hu (body part of the worker Hu) captured in the captured image Im (e.g., information indicating the presence or absence of the worker Hu, the spatial arrangement (position in space) of the worker Hu, the posture of the worker Hu, etc.). In addition, in the learning data, the captured image Im is labeled with information related to the work object Ob captured in the captured image Im (e.g., information indicating the presence or absence of the work object Ob, the spatial arrangement of the work object Ob, what the work object Ob is, etc.).

Furthermore, in the learning data, the captured image Im is labeled with information indicating "whether or not there is an interaction between the worker Hu captured in the captured image Im and the work object Ob captured in the captured image Im."

In addition, in the learning data, the captured image Im is labeled with information indicating "the content of the work Op being performed by the worker Hu captured in the captured image Im on the work object Ob captured in the captured image Im."

The information attached as a label to the captured image Im in the learning data is the same as the information that a trained model constructed by machine learning on a dataset DS, which is a collection of learning data, outputs in response to an input of a captured image Im (or feature information of the captured image Im).

(Building a detection model as a trained model)
In the learning process, the analysis unit 120 realized as a neural network constructs a detection model 151 that outputs the following information in response to an input of a captured image Im as a learned model through supervised learning on the dataset DS. That is, the detection model 151 is a learned model that receives the captured image Im as an input and outputs "information related to the worker Hu (body part of the worker Hu) captured in the captured image Im" and "information related to the work object Ob captured in the captured image Im". In particular, the detection model 151 is a learned model that outputs "information related to the worker Hu (body part of the worker Hu) captured in the captured image Im" and "information related to the work object Ob captured in the captured image Im" as feature information (feature amount) of the captured image Im.

As mentioned above, in the learning data, the captured image Im is labeled with "information related to the worker Hu (body part of the worker Hu) captured in the captured image Im" and "information related to the work object Ob captured in the captured image Im."

The analysis unit 120, which is realized as a neural network, stores the detection model 151 constructed through learning in the memory unit 150.

(Building a decision model as a trained model)
In the learning process, the analysis unit 120 realized as a neural network constructs, as a learned model, a determination model 152 that outputs the following information in response to an input of a captured image Im or feature information (feature amount) of the captured image Im, by supervised learning on the dataset DS: That is, the determination model 152 is a learned model that receives an input of a captured image Im (feature information of the captured image Im) and outputs information indicating "the presence or absence of an interaction between a worker Hu captured in the captured image Im and a work object Ob captured in the captured image Im."

As mentioned above, in the learning data, the captured image Im is labeled with information indicating "whether or not there is an interaction between the worker Hu captured in the captured image Im and the work object Ob captured in the captured image Im."

The learning data included in the "dataset DS in which the analysis unit 120 realized as a NN performs supervised learning to construct the judgment model 152" may be the following data. That is, it may be data in which information indicating "whether or not there is an interaction between the worker Hu captured in the captured image Im and the work object Ob captured in the captured image Im" as a label is attached to the feature information (feature amount) of the captured image Im. That is, it may be data in which information indicating "whether or not there is an interaction between the worker Hu captured in the captured image Im and the work object Ob captured in the captured image Im" is attached as a label to the "feature information extracted from the captured image Im by the detection model 151".

The analysis unit 120, implemented as a neural network, stores the judgment model 152 constructed through learning in the memory unit 150.

(Building an estimation model as a trained model)
In the learning process, the analysis unit 120 realized as a neural network constructs, as a learned model, an estimation model 153 that outputs the following information in response to an input of a captured image Im or feature information (feature amount) of the captured image Im, by supervised learning on the dataset DS: That is, the estimation model 153 is a learned model that receives an input of a captured image Im (feature information of the captured image Im) and outputs information indicating "the content of the work Op being performed by a worker Hu captured in the captured image Im on a work object Ob captured in the captured image Im."

As mentioned above, in the learning data, the captured image Im is labeled with information indicating "the content of the work Op being performed by the worker Hu captured in the captured image Im on the work object Ob captured in the captured image Im."

The learning data included in the "dataset DS in which the analysis unit 120 realized as a NN performs supervised learning to construct the estimation model 153" may be the following data. That is, it may be data in which information indicating "the content of the work Op being performed by the worker Hu captured in the captured image Im on the work object Ob captured in the captured image Im" as a label is attached to the feature information (feature amount) of the captured image Im. That is, it may be data in which information indicating "the content of the work Op being performed by the worker Hu captured in the captured image Im on the work object Ob captured in the captured image Im" is attached as a label to the "feature information extracted from the captured image Im by the detection model 151".

The analysis unit 120, which is realized as a neural network, stores the estimation model 153 constructed by learning in the memory unit 150.

§3. Operation examples (operation examples for each mode)
The information processing device 10 operates in a normal mode or a transfer learning mode.

In normal mode, the analysis unit 120, which is realized as a neural network, constructs an interaction determiner P (determination model 152) and an HOI classifier C (estimation model 153) by learning on the same dataset DS.

In the transfer learning mode, the data set DS learned by the analysis unit 120 implemented as a neural network to construct the interaction classifier P (determination model 152) is different from the data set DS learned by the analysis unit 120 to construct the HOI classifier C (estimation model 153).

In the transfer learning mode, the analysis unit 120, which is realized as a neural network, constructs an interaction determiner P (determination model 152) by learning on a dataset DS(X) that includes, for example, multiple datasets DS from different domains. For example, in the transfer learning mode, the analysis unit 120 constructs a determination model 152(X) by learning on a dataset DS(X) that includes "dataset DS(A) prepared in factory Fa(A)" and "dataset DS(B) prepared in factory Fa(B)."

In the data set DS(X), the captured image Im captured in at least one of the factories Fa(A) and Fa(B) is labeled with information indicating whether or not there is an interaction between the "worker Hu and work object Ob" captured in the captured image Im. In the transfer learning mode, the analysis unit 120 uses the data set DS(X) to learn whether or not there is an interaction between the "worker Hu and work object Ob" captured in the captured image Im.

The judgment model 152(X) constructed from the learning of the dataset DS(X) can cooperate with the estimation model 153(C) constructed from the learning of the dataset DS(C), which is a dataset DS different from the dataset DS(X). In other words, the judgment model 152(X) constructed from the learning of the dataset DS(X) can determine whether or not there is an interaction between the "worker Hu and work object Ob" captured in the captured image Im(C) for an image Im(C) captured in the factory Fa(C).

In the transfer learning mode, the estimation model 153(C) uses the output of the judgment model 152(X) to extract pairs for which the HOI is to be classified (i.e., the content of the work Op is to be estimated). That is, the estimation model 153(C) to which the captured image Im(C) (or its feature information) is input classifies the HOI only for pairs for which the judgment model 152(X) to which the captured image Im(C) (or its feature information) is input judges that "there is an interaction between the two."

Because the "contents of the work Op that the worker Hu is performing on the work object Ob" vary from factory to factory Fa, the "contents of the work Op that the worker Hu is performing on the work object Ob" are highly domain-dependent. Therefore, learning of the "contents of the work Op that the worker Hu is performing on the work object Ob" needs to be performed for each factory Fa.

In contrast, there is no relationship, or the relationship is considered to be sufficiently small, between the factory Fa and "whether or not there is an interaction between the worker Hu and the work object Ob," and "whether or not there is an interaction between the worker Hu and the work object Ob" is less dependent on the domain. Therefore, the knowledge for "determining whether or not there is an interaction between the worker Hu and the work object Ob in a certain factory Fa (C)" can be learned from data collected in another factory Fa (X).

Therefore, in the transfer learning mode, the analysis unit 120 determines whether or not there is an interaction between the worker Hu and the work object Ob in the factory Fa(C) using the judgment model 152(X) constructed by learning from the data collected in the factory Fa(X).

The learning process and analysis process in normal mode and transfer learning mode are described in detail below with reference to Figures 5 and 6.

(Normal mode)
Fig. 5 is a diagram for explaining an outline of the process executed by the information processing device 10 in the normal mode. Fig. 5(A) is a diagram for explaining an outline of the learning process executed by the information processing device 10 in the normal mode, and Fig. 5(B) is a diagram for explaining an outline of the analysis process executed by the information processing device 10 in the normal mode.

In FIG. 5, "R", "P", and "C" respectively denote the feature extractor R (detection model 151), the interaction determiner P (determination model 152), and the HOI classifier C (estimation model 153). As described above, the interaction determiner P (determination model 152) determines whether there is an interaction (interactive/non-interactive) between the "pair of worker Hu and work object Ob" captured in the captured image Im. In addition, the HOI classifier C (estimation model 153) estimates the content of the work Op being performed by the worker Hu, one of the pair, on the work object Ob, the other of the pair, as the HOI of the "pair of worker Hu and work object Ob" captured in the captured image Im.

In normal mode, the analysis unit 120 constructs a trained model to be used for analyzing "images Im captured in a certain factory Fa" as follows. That is, the analysis unit 120 constructs a model for analysis from a collection (dataset DS) of "learning data in which a specific label is attached to 'images Im captured in that certain factory Fa or feature information of images Im captured in that certain factory Fa'".

In normal mode, the captured images Im included in the learning data (teacher data) and the captured images Im input to the trained model constructed by learning from the dataset DS, which is a collection of that learning data, are captured in the same factory Fa.

For example, in normal mode, the analysis unit 120 generates a trained model to be used for analyzing "image Im(A) captured in factory Fa(A)" as follows. That is, the analysis unit 120 constructs a trained model to be used for analyzing image Im(A) by supervised learning on a dataset DS(A), which is a collection of "data in which a predetermined label is attached to image Im(A) or feature information of image Im(A)".

Therefore, in normal mode, the dataset DS trained to construct the interaction determiner P (determination model 152) and the dataset DS trained to construct the HOI classifier C (estimation model 153) are the same dataset DS.

For example, in FIG. 5(A), a judgment model 152(A) and an estimation model 153(A), which are trained models for analyzing an image Im(A) captured in a factory Fa(A), are both constructed by training on a dataset DS(A).

As shown in FIG. 5B, in normal mode, analysis processing of the captured image Im(A) is performed using a determination model 152(A) and an estimation model 153(A) that are constructed by learning on a common data set DS(A).

In other words, in the analysis process of the captured image Im(A) in normal mode, the judgment model 152(A) judges whether or not there is an interaction between the "pair of worker Hu and work object Ob" captured in the captured image Im(A).

In addition, in the analysis process of the captured image Im(A) in the normal mode, the estimation model 153(A) estimates the contents of the work Op as the HOI of a pair determined by the determination model 152(A) that "there is an interaction between the two." In FIG. 5(B), limiting the pairs for which the contents of the work Op are estimated to those determined to "have an interaction between the two" is called "interactive suppression."

(Transfer learning mode)
Fig. 6 is a diagram for explaining an overview of the process executed by the information processing device 10 in the transfer learning mode. Fig. 6A is a diagram for explaining an overview of the learning process executed by the information processing device 10 in the transfer learning mode, and Fig. 6B is a diagram for explaining an overview of the analysis process executed by the information processing device 10 in the transfer learning mode.

In FIG. 6, "R", "P", and "C" respectively denote the feature extractor R (detection model 151), the interaction determiner P (determination model 152), and the HOI classifier C (estimation model 153). As described above, the interaction determiner P (determination model 152) determines whether there is an interaction (interactive/non-interactive) between the "pair of worker Hu and work object Ob" captured in the captured image Im. In addition, the HOI classifier C (estimation model 153) estimates the content of the work Op being performed by the worker Hu, one of the pair, on the work object Ob, the other of the pair, as the HOI of the "pair of worker Hu and work object Ob" captured in the captured image Im.

The method by which the analysis unit 120 constructs the estimation model 153 in the learning process in the transfer learning mode is similar to the method by which the analysis unit 120 constructs the estimation model 153 in the learning process in the normal mode.

Specifically, the analysis unit 120 constructs an estimation model 153 that estimates the content of an operation Op being performed by a worker Hu on a work object Ob in a certain factory Fa from an image Im (or its characteristic information) captured in the certain factory Fa, as follows. That is, the analysis unit 120 constructs the estimation model 153 from a set (dataset DS) of "learning data in which a predetermined label is attached to 'an image Im (or its characteristic information) captured in the certain factory Fa'".

For example, in the transfer learning mode, the analysis unit 120 generates an estimation model 153(C) to be used for analyzing "an image Im(C) captured in a factory Fa(C)" as follows. That is, the analysis unit 120 constructs the estimation model 153(C) by supervised learning on a dataset DS(C), which is a collection of "data in which a specific label is attached to the captured image Im(C) or feature information of the captured image Im(C)".

In contrast, the method by which the analysis unit 120 constructs the judgment model 152 in the learning process in the transfer learning mode differs from the method by which the analysis unit 120 constructs the judgment model 152 in the learning process in the normal mode.

The analysis unit 120 constructs a judgment model 152 for judging the presence or absence of an interaction between a "pair of a worker Hu and a work object Ob" captured in a captured image Im (or its feature information) captured in a certain factory Fa, as follows. That is, the analysis unit 120 constructs the judgment model 152 from a set (dataset DS) of "learning data in which a predetermined label is attached to 'a captured image Im (or its feature information) captured in a factory Fa other than the certain factory Fa'".

For example, in the transfer learning mode, the analysis unit 120 generates a judgment model 152(C) to be used for analyzing "an image Im(C) captured in a factory Fa(C)" as follows. That is, the analysis unit 120 sets the judgment model 152(A) constructed from learning on the data set DS(A) as the judgment model 152(C).

In order to construct the judgment model 152(C), the analysis unit 120 may further learn a dataset DS(B), which is a collection of "data in which a specific label is attached to an image Im(B) (or its characteristic information) captured in a factory Fa(B)."

In other words, in the transfer learning mode, the analysis unit 120 may use the judgment model 152(X) constructed from learning on the dataset DS(X) including the dataset DS(A) and the dataset DS(B) as the judgment model 152(C).

As shown in FIG. 6A, in the transfer learning mode, the data set DS(X) that the analysis unit 120 learns to construct the judgment model 152 is different from the data set DS(C) that the analysis unit 120 learns to construct the estimation model 153. In the transfer learning mode, the analysis unit 120 builds the judgment model 152(C) (i.e., the judgment model 152(X)) from learning on the data set DS(X), and builds the estimation model 153(C) from learning on the data set DS(C).

As shown in FIG. 6B, in the transfer learning mode, the analysis process for the captured image Im(C) is performed using a judgment model 152(C) (= judgment model 152(X)) and an estimation model 153(C) constructed from learning on mutually different data sets DS.

In other words, in the analysis process for the captured image Im(C) in the transfer learning mode, the judgment model 152(X) judges whether or not there is an interaction between the "pair of worker Hu and work object Ob" captured in the captured image Im(C).

In addition, in the analysis process for the captured image Im (C) in the transfer learning mode, the estimation model 153 (C) estimates the content of the work Op as the HOI of a pair determined by the determination model 152 (X) to have "an interaction between the two." In FIG. 6B, limiting the pairs for which the content of the work Op is estimated to those determined to have "an interaction between the two" is called "interactive suppression."

The contents described above using Figures 5 and 6 can be summarized as follows. That is, the information processing device 10 operates in a normal mode or a transfer learning mode.

In normal mode, the information processing device 10 constructs a judgment model 152 (e.g., judgment model 152(A)) by learning from a dataset DS (e.g., dataset DS(A)) collected in a certain factory (e.g., factory Fa(A)). Then, using the constructed judgment model (e.g., judgment model 152(A)), the judgment unit 122 judges whether or not there is an interaction between a worker Hu and a work object Ob for an image Im(A) captured in a work area Ar of the certain factory (e.g., factory Fa(A)).

In the transfer learning mode, the information processing device 10 constructs a judgment model 152 (e.g., judgment model 152(X)) by learning from a dataset DS (e.g., dataset DS(X)) collected in a certain factory (e.g., factory Fa(X)). Then, the judgment unit 122 uses the constructed judgment model (e.g., judgment model 152(X)) to judge whether or not there is an interaction between a worker Hu and a work object Ob for an image Im(C) captured in a work area Ar of another factory Fa (e.g., factory Fa(C)).

According to the above configuration, the information processing device 10 operates in a normal mode or a transfer learning mode. In the normal mode, the factory Fa where the captured image Im is captured, for which the presence or absence of an interaction between the worker Hu and the work object Ob is determined, is the same as the factory Fa where the dataset for constructing the judgment model 152 is collected. In the transfer learning mode, the factory Fa where the captured image Im is captured, for which the presence or absence of an interaction between the worker Hu and the work object Ob is determined, is different from the factory Fa where the dataset for constructing the judgment model 152 is collected.

Here, the developer of the information processing device 10 confirmed that the presence or absence of an interaction between the worker Hu captured in the captured image Im and the work object Ob captured in the captured image Im can be determined sufficiently independently of the influence of the environment in which the captured image Im was captured.

In other words, for the judgment model 152 that judges whether or not there is an interaction, the judgment model 152(X) constructed by learning on a data set DS(X) collected in a certain factory Fa(X) can be applied to judging an image Im(Y) captured in another factory Fa(Y). Specifically, in response to an input of an image Im(Y) or its feature information, the judgment model 152(X) can output the presence or absence of an interaction between a worker Hu captured in the captured image Im(Y) and a work object Ob captured in the captured image Im(Y).

Therefore, in the transfer learning mode, the information processing device 10 has the effect of being able to use a judgment model 152(Y) learned in another factory Fa(Y) to judge an image (X) captured in a certain factory Fa(X).

(Details of analysis process)
Fig. 7 is a flow diagram for explaining an overall outline of the analysis processing executed by the information processing device 10. In the flow diagram illustrated in Fig. 7, steps S210 and S220 are processes executed by the detection unit 121 (detection model 151) which is the feature extractor R. Steps S310 and S320 are processes executed by the determination unit 122 (determination model 152) which is the interaction determiner P. Steps S410 to S430 are processes executed by the estimation unit 123 (estimation model 153) which is the HOI classifier C.

As shown in FIG. 7, the captured image acquisition unit 110 acquires sensor data from the ceiling camera 30, specifically, acquires the captured image Im (S110). The captured image acquisition unit 110 outputs the acquired captured image Im to the analysis unit 120, in particular to the detection unit 121.

The analysis unit 120 (particularly, the detection unit 121) generates feature information (feature amount) from the captured image Im. The detection unit 121 generates information related to the worker Hu (body part of the worker Hu) captured in the captured image Im as feature information of the captured image Im, and performs, for example, detection of the worker Hu (body part of the worker Hu) captured in the captured image Im from the captured image Im. In other words, the detection unit 121 acquires information related to the worker Hu (body part of the worker Hu) captured in the captured image Im from the captured image Im (e.g., information indicating the presence or absence of the worker Hu, the spatial arrangement of the worker Hu, the posture (pose) of the worker Hu, etc.) (S210).

The detection unit 121 also generates information related to the work object Ob captured in the captured image Im as feature information of the captured image Im, and performs, for example, detection of the work object Ob captured in the captured image Im from the captured image Im. In other words, the detection unit 121 acquires information related to the work object Ob captured in the captured image Im (e.g., information indicating the presence or absence of the work object Ob, the spatial arrangement of the work object Ob, what the work object Ob is, etc.) from the captured image Im (S220).

The detection unit 121 outputs characteristic information of the captured image Im acquired (generated) from the captured image Im (e.g., information relating to each of the worker Hu and the work object Ob captured in the captured image Im) to the determination unit 122 and the estimation unit 123.

The determination unit 122 obtains (generates) pair list information, which is a list of pairs of "workers Hu captured in the captured image Im" and "work objects Ob captured in the captured image Im" from the feature information obtained from the detection unit 121 (S310). The pair list information shows all pairs (e.g., eight pairs in the captured image Im in FIG. 3), each of which is "made up of 'any one of the workers Hu captured in the captured image Im' and 'any one of the work objects Ob captured in the captured image Im'".

The determination unit 122 determines whether there is an interaction between the worker Hu and the work object Ob for each of all pairs listed in the pair list information (S320).

When the determination unit 122 determines that there is an interaction between the worker Hu and the work object Ob for a pair listed in the pair list information (YES in S320), it notifies the estimation unit 123 of the pair determined to have an interaction between the worker Hu and the work object Ob. When the determination unit 122 determines that there is no interaction between the worker Hu and the work object Ob for all pairs listed in the pair list information (NO in S320), the information processing device 10 returns to S110 and acquires a new captured image Im from the ceiling camera 30.

The estimation unit 123 identifies pairs that are determined by the determination unit 122 to have an interaction between a worker Hu and a work object Ob, that is, associates the worker Hu and the work object Ob that are determined to have an interaction between them (S410).

The estimation unit 123 estimates the content of the work Op that the worker Hu is performing on the work object Ob for the worker Hu associated with the work object Ob in S410 (S420). The estimation unit 123 detects the work Op that the worker Hu is performing on the work object Ob as the HOI only for pairs that are determined by the determination unit 122 to have "an interaction between the worker Hu and the work object Ob," that is, estimates the content of the work Op.

The estimation unit 123 (analysis unit 120) outputs the estimated content of the work Op (work content), for example, to the support information generation unit 140 and the video storage server of FIG. 2 (S430).

The process executed by the information processing device 10, which has been described using FIG. 7 so far, can be summarized as follows. That is, the control method executed by the information processing device 10 is a control method of the information processing device 10 that estimates, from an image Im captured of the workplace Ar, the content of the work Op being performed by the worker Hu captured in the captured image Im. The control method includes detection steps (S210 and S220), a determination step (S320), and an estimation step (S420).

The detection step detects the worker Hu captured in the captured image Im and the work object Ob captured in the captured image Im from the captured image Im.

The judgment step is a model-based determination of the presence or absence of an interaction between the worker Hu and the work object Ob detected in the detection step from the captured image Im or its feature information. The judgment step uses the judgment model 152, which is a trained model, to determine the presence or absence of an interaction between the worker Hu and the work object Ob detected in the detection step from the captured image Im or its feature information. The judgment model 152 is a trained model that receives as input the "captured image Im or feature information generated from the captured image Im" and outputs the "presence or absence of an interaction between the worker Hu captured in the captured image Im and the work object Ob captured in the captured image Im."

The estimation step estimates the content of the work Op that the worker Hu is performing on the work object Ob for a combination (pair) of the worker Hu and the work object Ob that is determined to have an interaction in the determination step.

According to the above configuration, the control method uses the judgment model 152, which is a trained model, to judge whether or not an interaction exists between a "pair of a worker Hu and a work object Ob" detected from a captured image Im. Then, for a "pair of a worker Hu and a work object Ob" for which it is judged that an interaction exists, the control method estimates the content of the work Op being performed by one of the pair, the worker Hu, on the other of the pair, the work object Ob.

In other words, the control method determines whether or not there is interaction between the worker Hu and the work object Ob detected from the captured image Im on a model-based basis, rather than a rule-based basis.

Therefore, the control method has the effect of being able to estimate the work Op being performed by the worker Hu, even if the number of types of work Op being performed by the worker Hu increases, without the need to prepare complicated rules for estimating the content of the work Op.

The control method also estimates the content of the work Op that the worker Hu is performing on the work object Ob for a pair of worker Hu and work object Ob that is determined to have "interaction" based on the model.

In other words, the control method estimates the content of the work Op being performed by the worker Hu only for pairs determined to have "interactions" rather than for all pairs of the worker Hu and the work object Ob captured in the captured image Im.

The control method thus has the effect of reducing the amount of calculation required to estimate the content of the work Op being performed by the worker Hu captured in the captured image Im on the work object Ob captured in the captured image Im.

§ 4. Modifications Up to this point, an example has been described in which the detection model 151, the judgment model 152, and the estimation model 153 are constructed as learned data by machine learning on supervised data. However, the machine learning for constructing the detection model 151, the judgment model 152, and the estimation model 153 is not limited to machine learning on supervised data. The machine learning for constructing the detection model 151, the judgment model 152, and the estimation model 153 is not limited to supervised learning, and may be other machine learning such as unsupervised learning or reinforcement learning.

Also, so far, an example has been described in which the estimation unit 123 uses the estimation model 153 to "estimate the content of the work Op being performed by the worker Hu on the work object Ob" from the captured image Im. However, the estimation unit 123 may also estimate the content of the work Op being performed by the worker Hu on the work object Ob on a rule-based basis from the captured image Im.

[Software implementation example]
The functional blocks of the information processing device 10 (specifically, the image acquisition unit 110, the analysis unit 120, the process information acquisition unit 130, and the support information generation unit 140) may be realized by a logic circuit (hardware) formed in an integrated circuit (IC chip) or the like, or may be realized by software using a CPU, GPU, DSP, etc.

In the latter case, the information processing device 10 includes a CPU, GPU, DSP, etc. that executes the commands of a program, which is software that realizes each function, a ROM or storage device (these are referred to as "recording media") in which the program and various data are recorded so as to be readable by a computer (or CPU), and a RAM or the like that deploys the program. The object of the present invention is achieved by having the computer (or CPU, GPU, DSP, etc.) read and execute the program from the recording media. The recording media may be "non-transient tangible media," such as tapes, disks, cards, semiconductor memories, programmable logic circuits, etc. The program may also be supplied to the computer via any transmission medium (such as a communication network or broadcast waves) that can transmit the program. The present invention may also be realized in the form of a data signal embedded in a carrier wave in which the program is embodied by electronic transmission.

The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. The technical scope of the present invention also includes embodiments obtained by appropriately combining the technical means disclosed in different embodiments.

10 Information processing device (work estimation device)
40 Equipment 20 PLC (control device)
121 Detection unit 122 Determination unit 123 Estimation unit 151 Detection model 152 Determination model 153 Estimation model 130 Process information acquisition unit (acquisition unit)
140 Support information generation unit (generation unit)
Ac: Action Ar: Workplace DS: Data set Fa: Factory Hu: Worker Im: Captured image La: Action result Ob: Work object Op: Work S210 (Detection step)
S220 (detection step)
S320 (Determination step)
S420 (Estimation step)

Claims (8)

A task estimation device that estimates, from a captured image of a workplace, a task performed by a worker captured in the captured image, comprising:
a detection unit that detects, from the captured image, a worker captured in the captured image and a work target captured in the captured image;
a determination unit that determines, from the captured image or the feature information, the presence or absence of an interaction between the worker captured in the captured image and the work object captured in the captured image, using a determination model that is a trained model that receives as input the captured image or feature information generated from the captured image and outputs the presence or absence of an interaction between the worker captured in the captured image and the work object captured in the captured image; and
an estimation unit that estimates the content of the work being performed by the worker on the work object for a combination of the worker and the work object for which it has been determined by the determination unit that an interaction exists;
Equipped with
the feature information includes information indicating a posture of the worker captured in the captured image, and information indicating spatial positions of the worker and the work target object captured in the captured image,
The determination unit inputs the feature information, including information indicating the posture of the worker and information indicating the respective spatial positions of the worker and the work object, into the determination model, and determines whether or not there is an interaction between the worker and the work object for each pair of the worker and the work object detected by the detection unit . 2. The work estimation device according to claim 1, wherein the detection unit detects from the captured image the worker captured in the captured image and the work object captured in the captured image using a detection model that is a trained model that receives the captured image as an input and outputs the worker captured in the captured image and the work object captured in the captured image. 3. The work estimation device according to claim 1 or 2, wherein the estimation unit uses an estimation model that is a trained model that receives as input the captured image or feature information generated from the captured image and outputs the content of the work that the worker captured in the captured image is performing on the work object captured in the captured image, for a combination of the worker and the work object for which an interaction is determined to exist by the determination unit, to estimate, from the captured image or the feature information, the content of the work that the worker is performing on the work object captured in the captured image. The task estimation device operates in a normal mode or a transfer learning mode;
In a normal mode, the determination unit determines whether or not there is an interaction between the worker and the work object for the captured image taken in the workplace of a certain factory, using the determination model constructed by learning on a data set collected in the certain factory;
4. The work estimation device according to claim 1, wherein in a transfer learning mode, the determination unit determines whether or not there is an interaction between the worker and the work object for the image taken in the work area of a different factory from the certain factory, by using the determination model constructed by learning on a dataset collected in a certain factory. an acquisition unit that acquires process information generated from an operation result indicating the content and result of the operation performed by the device from a control device that controls the operation of the device installed in the work area;
a generating unit that generates support information, which is information that can be used to support efficient execution of the work by the worker, from the content of the work of the worker estimated by the estimating unit and the process information acquired by the acquiring unit;
The task estimation device according to claim 1 , further comprising: A control method for an activity estimation device that estimates, from an image of a workplace, an activity performed by a worker captured in the image, comprising the steps of:
a detection step of detecting a worker captured in the captured image and a work target captured in the captured image from the captured image;
a determination step of determining, from the captured image or the feature information, the presence or absence of an interaction between the worker captured in the captured image and the work object detected in the detection step, using a determination model that is a trained model that receives as input the captured image or feature information generated from the captured image and outputs the presence or absence of an interaction between the worker captured in the captured image and the work object captured in the captured image;
an estimation step of estimating the content of the work being performed by the worker on the work object for the combination of the worker and the work object for which it has been determined that an interaction exists in the determination step;
Including,
the feature information includes information indicating a posture of the worker captured in the captured image, and information indicating spatial positions of the worker and the work target object captured in the captured image,
A control method in which, in the judgment step, the feature information including information indicating the posture of the worker and information indicating the respective spatial positions of the worker and the work object is input into the judgment model, and the presence or absence of interaction between the worker and the work object for each pair of the worker and the work object detected in the detection step is determined . An information processing program for causing a computer to function as the task estimation device according to any one of claims 1 to 5, the information processing program causing the computer to function as each of the units. A computer-readable recording medium having the information processing program according to claim 7 recorded thereon.

Images