JP2025059651A - Learning database construction system, control device used in said system, and learning database construction method - Google Patents

Abstract

To enable building a learning database with which the accuracy of recognizing a target object can be improved.SOLUTION: The present invention comprises: a 3D camera 24 capable of capturing an image of a target object 40; an imaging robot 20 capable of changing the position and attitude of the 3D camera 24; and a control device 10 capable of acquiring information of the image of the target object 40 captured by the 3D camera 24. The control device 10 generates three-dimensional information of the target object 40 by controlling the imaging robot 20 and carrying out three-dimensional imaging to capture the image of the target object 40 from a plurality of directions by the 3D camera 24, automatically generates annotation information for the image of the target object 40 captured by the 3D camera 24 on the basis of three-dimensional information, and builds a learning database in which a plurality of pieces of data from each of the annotation information and the three-dimension physical information of the target object 40 relative to the 3D camera 24 is adopted as learning data.SELECTED DRAWING: Figure 1

Description

The present invention relates to a learning database construction system, a control device used in the system, and a learning database construction method.

Conventionally, when constructing a database of learning data (also called teacher data) used in AI machine learning in image recognition technology, it is desirable to recognize the target object (hereinafter referred to as the target object) with high accuracy. In addition, constructing this learning database requires a lot of time and effort.

Patent Document 1 and Patent Document 2 are known as examples that disclose technology for photographing a target object and building a learning database.

Patent Document 1 discloses that in the construction stage of a learning dataset (corresponding to a learning database), object information of a target object is associated with an AR marker (two-dimensional pattern marker), which is a marker for detecting position and orientation. Patent Document 1 also discloses that the AR marker is associated with object information such as the object name of the target object in a database, and that a computer of the learning dataset creation device operates as an AR marker recognition means to recognize the AR marker and obtain object information of the target object.

Patent Document 2 also discloses a dataset storage unit that acquires from the simulator a learning dataset comprising three-dimensional coordinate data of the robot hand on the simulator when the robot hand grips a workpiece and successfully grips the workpiece through a gripping operation, and two-dimensional image data of the workpiece captured from a specified angle of view by a two-dimensional imaging device ID on the simulator, and stores multiple sets of the learning dataset, and constructs a learning model that infers the three-dimensional coordinates of the robot hand in the real world from two-dimensional images of the workpiece captured from the same angle of view as the specified angle of view by the two-dimensional imaging device ID.

Patent No. 6474179 JP 2020-82322 A

However, because the technique in Patent Document 1 uses a two-dimensional camera, there is a risk that the accuracy of recognizing the target object will be significantly reduced if the color of the background and the color of the target object are similar. In addition, because an AR marker is used, if the posture of the target object is such that the AR marker cannot be recognized, it is not possible to obtain information about the target object. This limits the posture of the target object from which object information can be obtained, making it difficult to build a highly accurate learning database.

In addition, because the technology in Patent Document 2 uses simulated images, it is unable to fully express the characteristics of real target objects, and there is a limit to how much the target object recognition accuracy can be improved, making it difficult to build a highly accurate learning database.

The present invention has been made in consideration of the above points, and its purpose is to provide a learning database construction system capable of constructing a learning database that can improve the recognition accuracy of a target object (for example, the recognition accuracy of the three-dimensional posture of a target object or the recognition accuracy of a specific three-dimensional position on the target object), as well as a control device and a learning database construction method used in the system.

The solution of the present invention for achieving the above object comprises a three-dimensional camera capable of photographing a target object, a movable body capable of changing the position and attitude of the three-dimensional camera, and a control device capable of acquiring information on an image of the target object photographed by the three-dimensional camera, the control device being characterized in comprising a three-dimensional information generation unit that generates three-dimensional information of the target object by controlling the movable body to perform three-dimensional photographing of the target object from multiple directions using the three-dimensional camera, an annotation information generation unit that automatically generates annotation information based on the three-dimensional information for an image of the target object photographed by the three-dimensional camera or a camera other than the three-dimensional camera, and a learning database construction unit that constructs a learning database using multiple data of the annotation information and the relative three-dimensional physical information of the target object with respect to the camera used when automatically generating the annotation information as learning data.

Due to this specification, the learning database is constructed using multiple pieces of learning data, each of which is annotation information and three-dimensional physical information of the target object relative to the camera used when the annotation information was automatically generated, making it possible to construct a learning database that can improve the recognition accuracy of the target object (for example, the recognition accuracy of the three-dimensional posture of the target object and the recognition accuracy of a specific three-dimensional position on the target object). For example, when performing specific processing on the target object, the three-dimensional physical information can be obtained simply by acquiring a two-dimensional image of the target object, and the recognition accuracy of the target object can be improved based on this three-dimensional physical information, making it possible to perform the specific processing on the target object with high accuracy.

The three-dimensional physical information is the relative three-dimensional posture information of the target object with respect to the camera used when automatically generating the annotation information.

This makes it possible to recognize the relative three-dimensional posture of a target object simply by acquiring a two-dimensional image of the target object. For example, when a target object is grasped by a robot, it becomes possible to optimize the grasping position.

The three-dimensional physical information may also be information about a specific three-dimensional position of the target object relative to the camera used when automatically generating the annotation information.

This makes it possible to recognize a specific three-dimensional position on a target object simply by acquiring a two-dimensional image of the target object. For example, when processing a specific position on a target object using a robot, it becomes possible to identify the processing position with high accuracy.

The control device used in the learning database construction system also falls within the scope of the technical idea of the present invention. In other words, the control device is capable of acquiring information on an image of a target object captured by a three-dimensional camera supported on a movable body and capable of changing the position and attitude by operating the movable body, and includes a three-dimensional information generation unit that generates three-dimensional information on the target object by controlling the movable body to perform three-dimensional photography of the target object from multiple directions using the three-dimensional camera, an annotation information generation unit that automatically generates annotation information based on the three-dimensional information for an image of the target object captured by the three-dimensional camera or a camera other than the three-dimensional camera, and a learning database construction unit that constructs a learning database using multiple data on the annotation information and the relative three-dimensional physical information of the target object with respect to the camera used when automatically generating the annotation information as learning data.

The learning database construction method implemented in the learning database construction system also falls within the scope of the technical idea of the present invention. In other words, it includes a three-dimensional information generation step of generating three-dimensional information of a target object by controlling a movable body supporting a three-dimensional camera to perform three-dimensional photography of the target object from multiple directions using the three-dimensional camera, an annotation information generation step of automatically generating annotation information based on the three-dimensional information for an image of the target object captured by the three-dimensional camera or a camera other than the three-dimensional camera, and a learning database construction step of constructing a learning database using multiple data of the annotation information and the relative three-dimensional physical information of the target object with respect to the camera used when automatically generating the annotation information as learning data.

Even with these specific features, as mentioned above, it is possible to construct a learning database that can improve the recognition accuracy of the target object (for example, the recognition accuracy of the target object's three-dimensional orientation or the recognition accuracy of a specific three-dimensional position on the target object).

In the present invention, three-dimensional information of a target object is generated by controlling a movable body to perform three-dimensional photography in which the target object is photographed from multiple directions by a three-dimensional camera. Furthermore, annotation information is automatically generated based on the three-dimensional information for an image of the target object photographed by the three-dimensional camera or a camera other than the three-dimensional camera. A learning database is then constructed using multiple pieces of learning data, including the annotation information and the three-dimensional physical information of the target object relative to the camera used when automatically generating the annotation information. This makes it possible to construct a learning database that can improve the recognition accuracy of the target object (for example, the recognition accuracy of the three-dimensional posture of the target object or the recognition accuracy of a specific three-dimensional position on the target object).

1 is a schematic diagram showing the overall configuration of a learning database construction system according to a first embodiment; FIG. 2 is a block diagram showing a configuration of a control device according to the first embodiment. 1 is a block diagram showing a configuration of an imaging robot according to a first embodiment. FIG. FIG. 4 is a flowchart illustrating a procedure of a preparation process according to the first embodiment. 4 is a diagram showing an example of point cloud data of a target object according to the first embodiment; FIG. FIG. 2 is a flowchart showing the procedure of deep learning processing according to the first embodiment. 1 is an image diagram showing multiple examples of bounding boxes assigned to an image of a target object according to the first embodiment; 4A to 4C are diagrams for explaining an operation of acquiring a three-dimensional posture of a target object relative to a 3D camera according to the first embodiment. FIG. 23 is a flowchart showing the procedure of a preparation process according to the seventh embodiment. 23A to 23D are diagrams for explaining an inference operation of a specific position of a target object according to the ninth embodiment.

Embodiments of the present invention will be described below with reference to the drawings. Each embodiment described below will be applied to the case where the present invention is constructed as a learning database (teacher database) that is used when a robot (movable body) performs a predetermined process on a target object (e.g., grasping the target object or processing a specific position on the target object). In the following description, identical parts and components are given the same reference numerals. As their names and functions are also the same, detailed descriptions of them will not be repeated.

First Embodiment
In this embodiment, a case will be described in which the present invention is applied to construct a learning database used for estimating the three-dimensional posture of a target object relative to a 3D camera (three-dimensional camera).

-Overall configuration and operation of the learning database construction system-
First, the overall configuration of a learning database construction system 1 according to this embodiment will be described with reference to Fig. 1. The learning database construction system 1 includes, as main components, a control device 10, an image capturing robot 20, and a mounting device 30.

The control device 10 is realized by a server, computer, etc., and performs various processes such as acquiring information on images captured by the 3D camera 24 mounted on the shooting robot 20 by transmitting and receiving various data between the shooting robot 20 and the mounting device 30 via a wired LAN or wireless LAN. The processes performed by the control device 10 will be described in detail later.

The photography robot 20 is composed of an articulated robot equipped with an arm unit 27 and a working unit 28 attached to the tip of the arm unit 27, and performs various tasks such as moving the arm unit 27 and working unit 28 to various positions and tilting them into various postures based on command information sent from the control device 10 or according to its own judgment processing (processing by the CPU 21 described later).

The mounting device 30 has a mounting table 31 on which a target object 40 that is the subject of deep learning or annotation is placed. This mounting table 31 can be rotated and tilted.

The control device 10 operates the arm unit 27 of the shooting robot 20 to change the position and orientation of the 3D camera 24, thereby photographing the target object 40 placed on the mounting table 31 from various angles and automatically annotating the target object 40, automatically adding a bounding box to the captured image of the target object 40, or segmenting the target object 40 from the captured image, and even acquiring the three-dimensional orientation of the target object 40 relative to the 3D camera 24 as learning data (details will be described later).

In this way, the learning database construction system 1 according to this embodiment enables deep learning with reduced effort on the part of the operator. The configuration and operation of each part of the learning database construction system 1 will be described in detail below.

-Control device configuration-
An embodiment of the configuration of the control device 10, which is a component of the learning database construction system 1 according to the present embodiment, will be described with reference to Fig. 2. The control device 10 includes, as main components, a CPU (Central Processing Unit) 11, a memory 12, an operation unit 13, and a communication interface 14.

The CPU 11 controls each part of the control device 10 by executing a program stored in the memory 12. For example, the CPU 11 executes a program stored in the memory 12 and performs various processes described below by referring to various data.

The CPU 11 has three-dimensional information generating unit 11a, annotation information generating unit 11b, and learning database constructing unit 11c as functional units realized by the program. The following provides an overview of the functions of each of these units.

The three-dimensional information generating unit 11a has a function of generating three-dimensional information of the target object 40 by controlling the photographing robot 20 (particularly the arm unit 27 of the photographing robot 20) to perform three-dimensional photographing of the target object 40 from multiple directions using the 3D camera 24. Specifically, the three-dimensional information generating unit 11a has a function of acquiring three-dimensional information (three-dimensional image information) of only the target object 40, and has a function of acquiring three-dimensional photographs of the entire 360-degree circumference of the target object 40 by moving and rotating the arm unit 27 of the photographing robot 20, and creating three-dimensional point cloud data of the target object 40 from the acquired RGB+depth map for 360 degrees.

The annotation information generating unit 11b has a function of automatically generating annotation information based on the three-dimensional information for an image of the target object 40 captured by the 3D camera 24. Specifically, the annotation information generating unit 11b has a function of automatically generating a bounding box that approximately circumscribes the target object 40 (or a contour line of the target object 40 by segmentation) from the three-dimensional point cloud data of the target object 40 and information on the shooting direction when the target object 40 is captured by the 3D camera 24 (the shooting direction is determined by the attitude of the 3D camera 24 according to the attitude of the arm unit 27).

The learning database construction unit 11c constructs a learning database using multiple pieces of data, each of which is the annotation information and information on the three-dimensional posture of the target object 40 relative to the 3D camera 24 (three-dimensional physical information in this invention), as learning data. Specifically, the learning database construction unit 11c has the function of photographing the target object 40 from multiple directions, learning (acquiring learning data) information on the bounding box (or the contour of the target object 40 by segmentation) assigned to the target object 40 corresponding to each direction, and information on the three-dimensional posture of the target object 40, and constructing a learning database using this learning data.

The memory 12 is realized by various RAMs, various ROMs, etc., and may be included in the control device 10, may be detachable from various interfaces of the control device 10, or may be a recording medium of another device accessible from the control device 10. The memory 12 stores programs executed by the CPU 11, data generated by execution of programs by the CPU 11, input data, other databases used in this embodiment, etc.

The operation unit 13 accepts commands from a user, administrator, etc., and inputs the commands to the CPU 11.

The communication interface 14 transmits data from the CPU 11 to the camera robot 20 via a wired or wireless LAN, and conversely, receives data from the camera robot 20 (including three-dimensional image information from the 3D camera 24) and passes it to the CPU 11.

- Configuration of the filming robot -
Next, one embodiment of the configuration of the photography robot 20, which is a component of the learning database construction system 1, will be described with reference to Fig. 3. The photography robot 20 includes, as main components, a CPU 21, a memory 22, an operation unit 23, a 3D camera 24, a light 25, a communication interface 26, an arm unit 27, and a working unit 28.

The CPU 21 controls each part of the photography robot 20 by executing a program stored in the memory 22. For example, the CPU 21 controls the posture of the arm unit 27 by adjusting the rotation angle of the motor provided in each joint of the photography robot 20.

The memory 22 is realized by various RAMs, various ROMs, etc., and stores various programs, data generated by execution of programs by the CPU 21, operation commands given by the control device 10, data input via the operation unit 23, etc.

The operation unit 23 is composed of buttons, switches, etc., and accepts various commands from the user and inputs the commands to the CPU 21.

The 3D camera 24 is realized by an RGB-D camera or the like. This 3D camera 24 can obtain the distance to each part of the captured image by using, for example, two cameras. The 3D camera 24 performs three-dimensional shooting or normal two-dimensional shooting based on instructions from the CPU 21.

The light 25 emits light in front of the 3D camera 24 according to instructions from the CPU 21.

The communication interface 26 transmits and receives data to and from other devices such as the control device 10 via the Internet, a carrier network, a router, etc. For example, the communication interface 26 receives an operation command from the control device 10 and passes it to the CPU 21.

The arm unit 27 controls the position and attitude of the 3D camera 24 attached to the arm unit 27 and the position and attitude of the working unit 28 according to instructions from the CPU 21. The 3D camera 24 is fixed to the arm unit 27. The attitude of the arm unit 27 can be identified by monitoring the rotation angle position of each joint of the shooting robot 20. Therefore, by recognizing the attitude of the arm unit 27, the position and attitude of the 3D camera 24 can be grasped. For example, by recognizing the attitude of the arm unit 27, it is possible to grasp the coordinate position (hereinafter referred to as the X-axis coordinate position) in a direction along the optical axis of the lens of the 3D camera 24 (hereinafter referred to as the X-axis direction: see FIG. 8), the coordinate position (hereinafter referred to as the Y-axis coordinate position) in a direction perpendicular to the X-axis direction and extending horizontally (hereinafter referred to as the Y-axis direction), and the coordinate position (hereinafter referred to as the Z-axis coordinate position) in a direction extending perpendicular to each of the X-axis direction and the Y-axis direction.

The working unit 28 corresponds to a human hand attached to the tip of the arm unit 27 and performs grasping operations, etc., and performs various operations to grasp the target object 40 and change the position and orientation of the target object 40 according to instructions from the CPU 21.

-Control device information processing-
Next, the information processing of the control device 10 will be described in detail with reference to Fig. 4. As a preparation process for deep learning, the CPU 11 of the control device 10 executes the process shown in Fig. 4 according to the program in the memory 12. The process shown in Fig. 4 is a process performed by the three-dimensional information generator 11a.

First, the CPU 11 accepts 3D CAD data such as 3D shape and position information of the shooting environment (table, stage, robot itself, etc.) and registers it in the memory 12 (environment information registration) (step ST1). The shooting environment here refers to objects other than the target object 40 in the image captured by the 3D camera 24 (objects to be subtracted from the captured image in future image processing), such as the mounting device 30, the background wall surface, and the working unit 28 when the working unit 28 is within the field of view of the 3D camera 24.

The CPU 11 causes the 3D camera 24 attached to the arm 27 of the image capturing robot 20 to capture an image of the target object 40, and acquires an RGB+depth map (step ST2). As described above, since the 3D camera 24 is fixed to the arm 27 of the image capturing robot 20, the CPU 11 can calculate the orientation information of the 3D camera 24 (the aforementioned X-axis coordinate position, Y-axis coordinate position, Z-axis coordinate position, etc.) from the orientation information of the image capturing robot 20 and the arm 27.

The CPU 11 obtains 3D information (information on the 3D image) of only the target object 40 by subtracting the data of the surrounding objects (data on the shooting environment) registered in step ST1 from the RGB+depth map captured in step ST2 (object data extraction in step ST3).

The CPU 11 moves or rotates the arm 27 of the imaging robot 20, or rotates or tilts the mounting base 31 (changing the imaging direction in step ST4), to capture an image from a different angle (3D imaging in step ST2). That is, the CPU 11 repeats the processes in steps ST2 to ST5 until 3D imaging of the entire 360-degree circumference of the target object 40 is completed (until a YES determination is made in step ST5).

The CPU 11 creates three-dimensional point cloud data of the target object 40 from the RGB+depth map of 360 degrees of the target object 40 (3D point cloud data creation in step ST6). Specifically, as shown in FIG. 5, three-dimensional solid point cloud data is created from the three-dimensional captured image of the target object 40.

Furthermore, it is preferable that the CPU 11 moves the arm unit 27 to take additional photographs with the 3D camera 24 so that areas where the point cloud is insufficient or where there is noise can be photographed in more detail based on the point cloud created in step ST6, and recomposes the three-dimensional point cloud (data complementation process in step ST7).

Three-dimensional point cloud data is created in this manner, and the created point cloud data corresponds to the three-dimensional information referred to in the present invention, and the process of creating the point cloud data corresponds to the three-dimensional information generation process referred to in the present invention.

After the preparatory process shown in FIG. 4, the CPU 11 of the control device 10 continues to execute the process shown in FIG. 6 as deep learning processing according to the program in the memory 12. The process shown in FIG. 6 is performed by the annotation information generating unit 11b and the learning database constructing unit 11c.

The CPU 11 causes the 3D camera 24 attached to the arm portion 27 of the photographing robot 20 to take a two-dimensional photograph of the target object 40 (step ST11).

The CPU 11 calculates how the target object 40 appears and automatically creates annotation information based on the position information and posture information of the shooting robot 20 and the arm unit 27, the position information and posture information of the target object 40, and the three-dimensional point cloud data of the target object 40 (step ST12).

Specifically, in this embodiment, as shown in, for example, FIGS. 7(a) to (c), the CPU 11 automatically generates a bounding box that approximately circumscribes the target object 40 from the three-dimensional point cloud data of the target object 40 and the shooting direction as annotation information. FIG. 7(a) shows a bounding box that is added when the target object 40 is photographed from the front, and FIG. 7(b) shows a bounding box that is added when the target object 40 is photographed from diagonally in front. These FIGS. 7(a) and (b) show a case where a bounding box surrounded by horizontal and vertical lines is added. This bounding box specifies the coordinate positions of four points that are the intersections of the horizontal and vertical lines, and in this embodiment, the coordinate positions of these four points become learning data.

Figure 7(c) shows a rotated bounding box applied when the target object 40 is photographed from diagonally behind. A rotated bounding box is obtained by removing the restriction that the straight lines constituting the bounding box must be horizontal and vertical lines when setting a minimum circumscribing rectangle for the target object 40, and it is possible to set a minimum circumscribing rectangle for the target object 40 in an area smaller than a bounding box defined by horizontal and vertical lines. When this rotated bounding box is applied, the coordinate positions of the four points that are the intersections of the straight lines constituting the rotated bounding box become the learning data.

In addition, instead of a bounding box, the annotation information may be an automatically generated contour line of the target object 40 (a boundary line separating the target object 40 from the background) obtained by performing segmentation. In this case, the boundary line separating the target object 40 from the background becomes the learning data.

The above steps ST11 and ST12 correspond to the annotation information generation process performed by the annotation information generation unit 11b.

In step ST13, in addition to the information acquired in step ST12, the three-dimensional posture of the target object 40 as seen from the 3D camera 24 (relative to the 3D camera 24) in the posture of the 3D camera 24 at the time of shooting is acquired from the posture of the arm unit 27 of the shooting robot 20 at the time of shooting (corresponding to the posture of the 3D camera 24) and the three-dimensional posture information of the target object 40 acquired in the preparation process described above (acquired in step ST2). This three-dimensional posture is also the learning data referred to in the present invention. In general, as shown in FIG. 8, the three-dimensional posture of the target object 40 relative to the 3D camera 24 can be acquired from the amount of deviation in each of the X-axis direction (horizontal direction), Y-axis direction (horizontal direction perpendicular to the X-axis direction), and Z-axis direction (vertical direction) of the lens of the 3D camera 24 at the installation position of the target object 40.

Then, in step ST14, the CPU 11 moves and rotates the arm 27 of the shooting robot 20 (changing the shooting direction in step ST14) to shoot from a different angle (shooting in step ST11). That is, the CPU 11 repeats the processes from step ST11 to step ST15 until two-dimensional shooting of the entire 360 degrees around the target object 40 is completed (until a YES judgment is made in step ST15). As a result, information on the bounding box (or the contour of the target object 40 obtained by segmentation) assigned to the target object 40 corresponding to each of the multiple directions and information on the three-dimensional posture of the target object 40 (the three-dimensional posture of the target object 40 relative to the 3D camera 24) are learned (acquired as learning data), and a learning database is constructed from these learning data.

By constructing such a learning database, it becomes possible to estimate the three-dimensional posture of the target object 40 (generate an inference model) simply by acquiring one two-dimensional image with the 3D camera 24. For example, when the target object 40 is grasped by the working unit 28 of the photographing robot 20, it becomes possible to estimate the three-dimensional posture of the target object 40 by simply acquiring one two-dimensional image with the 3D camera 24 while referring to the learning database (for example, by extracting learning data that matches the acquired two-dimensional image), and to move the working unit 28 to the optimal grasping position.

The above steps ST13 to ST15 correspond to the learning database construction process carried out by the learning database construction unit 11c.

--Effects of the embodiment--
As described above, in this embodiment, the constructed learning database is constructed with a plurality of data of annotation information and relative three-dimensional posture information of the target object 40 with respect to the 3D camera 24 as learning data. Therefore, it is possible to construct a learning database that can improve the recognition accuracy of the target object 40 (recognition accuracy of the three-dimensional posture of the target object 40). For example, when the target object 40 is grasped by the working unit 28 of the photographing robot 20, the relative three-dimensional posture information of the target object 40 can be obtained simply by acquiring a two-dimensional image of the target object 40, and the recognition accuracy of the target object 40 can be improved based on this three-dimensional posture information, and when grasping the target object 40, it becomes possible to grasp the target object 40 at an optimal position (for example, near the center of gravity of the target object 40).

Other embodiments of the present invention are described below.

Second Embodiment
In addition to the above-described embodiment, it is preferable that the CPU 11 uses the recognition result using deep learning of the target object 40 based on the annotation information that was previously automatically created, and thereafter calculates the similarity between the annotation information of the target object 40 calculated in step ST11 from the captured image including the target object 40 and the information recognized by deep learning, and if the similarity is large, performs annotation processing with greater emphasis on angles that are close to a similar angle.

Third Embodiment
In addition to the above-described embodiment, lights may be provided on the photographing robot 20, the mounting device 30, the ceiling, the wall, etc. Then, in step ST14, the CPU 11 moves or rotates the arm unit 27 of the photographing robot 20, turns the light on and off, changes the light intensity, or changes the color of the light to photograph from a different angle. That is, the CPU 11 repeats the processes of steps ST11 to ST15 until two-dimensional photographing of various light conditions for 360 degrees around the target object 40 is completed (until YES is determined in step ST15).

Fourth Embodiment
In addition to the above-described embodiment, the orientation and posture of the target object 40 may be changed by the working unit 28 mounted on the imaging robot 20. In this case, since the three-dimensional shape of the target object 40 changes, information on the changed orientation and posture of the target object 40 is linked to the three-dimensional shape of the target object 40 at that time, and is stored separately in the memory 12 for each posture of the target object 40.

When performing the deep learning process (the process shown in the flowchart of FIG. 6), the CPU 11 reads out the orientation and posture of the target object 40 stored in the memory 12, and causes the working unit 28 of the imaging robot 20 to set the orientation and posture of the target object 40 to the registered state so that the target object 40 has the correct posture, and then performs the deep learning process.

Fifth embodiment
In addition to the above-described embodiment, in step ST11, the CPU 11 causes the image capturing robot 20 to perform two-dimensional image capturing, but the image capturing robot 20 may also perform three-dimensional image capturing. Then, annotation information may be added to each piece of three-dimensional image capturing data based on the 3D point cloud data.

Sixth Embodiment
In the embodiment described above, the 3D camera 24 used in the preparation process shown in Fig. 4 is also used to capture images for the deep learning process shown in Fig. 6. However, the present invention is not limited to this, and a camera other than the 3D camera 24 used in the preparation process shown in Fig. 4 may be used to capture images for the deep learning process shown in Fig. 6.

Seventh embodiment
In the above-described embodiment, three-dimensional CAD data such as three-dimensional shape and position information of the shooting environment (table, stage, robot itself, etc.) is accepted and registered in memory 12, but the three-dimensional information of the shooting environment may also be acquired in a similar manner to that of the target object 40.

Referring to FIG. 9, in this embodiment, before placing the target object 40, three-dimensional information of the shooting environment is acquired in a manner similar to the target object acquisition method shown in FIG. 4, and if the environmental information has not been registered (if a NO judgment is made in step ST8), the acquired three-dimensional information is registered in memory 12 as environmental data (step ST9).

After that, in the same manner as in the first embodiment, the CPU 11 repeats the processes from step ST2 to step ST5 until 3D imaging of the entire 360-degree circumference of the target object 40 is completed (until a YES determination is made in step ST5).

Eighth embodiment
Some or all of the roles of the devices such as the control device 10 and the photographing robot 20 of the learning database construction system 1 of the above-described embodiment may be performed by other devices. For example, some of the roles of the control device 10 may be performed by the photographing robot 20, multiple personal computers, or multiple servers on the cloud.

Ninth embodiment
The first embodiment described above is intended to construct a learning database used to estimate the three-dimensional orientation of the target object 40 relative to the 3D camera 24. Instead, in the present embodiment, the present invention is applied to construct a learning database used to grasp a specific position on the target object 40, process a specific position, or the like.

Figure 10 is a diagram for explaining the inference operation of a specific position of the target object 40 according to this embodiment. The position P1 of the target object 40 in Figure 10 is the position to be grasped by the working unit 28. For example, it is a position near the center of gravity of the target object 40. In this embodiment, the learning data acquired by the learning database construction unit 11c described above is learned as the position (position P1) to be grasped by the working unit 28 on the target object 40. That is, in step ST13 of the deep learning process described using Figure 6, a specific position (e.g. position P1) of the target object 40 as seen from the 3D camera 24 (relative to the 3D camera 24) in the posture of the 3D camera 24 at the time of shooting is acquired from the posture of the arm unit 27 of the shooting robot 20 at the time of shooting (corresponding to the posture of the 3D camera 24) and the three-dimensional posture information of the target object 40 acquired in the preparation process described above. In this way, by photographing the target object 40 from multiple directions, information on the bounding box (or the contour of the target object 40 obtained by segmentation) assigned to the target object 40 corresponding to each direction and information on specific positions of the target object 40 are learned (learning data is acquired), and a learning database is constructed using this learning data. This makes it possible to estimate the position on the target object 40 that should be grasped by the working unit 28 by simply acquiring one two-dimensional image.

In this case, the TCP (Tool Center Point) of the imaging robot 20 may also be learned. In other words, the operation of the arm unit 27 of the imaging robot 20 is controlled based on this TCP (e.g., position P2 in FIG. 10), and when grasping the target object 40, the working unit 28 is controlled based on the position to be grasped (position P1).

The configuration of this embodiment is not limited to gripping the target object 40, and can also be used to process a specific position on the target object 40. For example, when processing (e.g., welding or other processing) position P3 of the target object 40 in FIG. 10, the learning data acquired by the learning database construction unit 11c is learned as the processing position on the target object 40, and the processing position on the target object 40 can be estimated by simply acquiring one two-dimensional image.

<Other embodiments>
The present invention is not limited to the above-described embodiments, and all modifications and applications within the scope of the claims and equivalents thereto are possible.

For example, in each of the above embodiments, the present invention has been described as being applied to construct a learning database used when the photographing robot 20 grasps the target object 40 or processes a specific position on the target object 40. The present invention is not limited to this, and may be used to construct a learning database used for other purposes. For example, a learning database may be constructed to be used in a system that recognizes the posture of a target object 40 transported on a conveyor when it reaches a predetermined position, notifies an operator of a target object 40 that is not in the predetermined posture, or corrects the posture using the photographing robot 20.

In addition, in each of the above embodiments, the placement device 30 is included as a component of the learning database construction system 1, but this placement device 30 is not a required component of the present invention. For example, the learning database may be constructed by photographing a target object 40 placed on the floor.

The present invention can be applied to a learning database construction system for improving the recognition accuracy of target objects.

1 Learning database construction system 10 Control device 11a Three-dimensional information generation unit 11b Annotation information generation unit 11c Learning database construction unit 20 Photographing robot (movable body)
24 3D camera (three-dimensional camera)
40 Target object

The solution of the present invention for achieving the above object includes a three-dimensional camera capable of photographing a target object, a movable body capable of changing the position and attitude of the three-dimensional camera, and a control device capable of acquiring information on an image of the target object photographed by the three-dimensional camera, the control device comprising a three-dimensional information generation unit that generates three-dimensional information of the target object by controlling the movable body to perform three-dimensional photographing in which the target object is photographed from a plurality of directions by the three-dimensional camera, and a three-dimensional information generation unit that generates a bounding box circumscribing the target object as annotation information from the three-dimensional information and information on the photographing direction when the target object was photographed for an image of the target object photographed by the three-dimensional camera or a camera other than the three-dimensional camera . and a learning database construction unit that constructs a learning database using multiple data of the annotation information and relative three-dimensional physical information of the target object with respect to the camera used when automatically generating the annotation information as learning data , wherein the bounding box circumscribing the target object is obtained by removing the restriction that the straight lines that constitute the bounding box are horizontal and vertical lines, and includes a rotated bounding box that has a smaller area than a bounding box defined by horizontal and vertical lines for the target object and sets a minimum circumscribing rectangle for the target object .

A control device used in the learning database construction system is also within the scope of the technical idea of the present invention. That is, a control device capable of acquiring information on an image of a target object captured by a three-dimensional camera supported on a movable body and capable of changing the position and orientation by operating the movable body, the control device comprising: a three-dimensional information generating unit that generates three-dimensional information of the target object by controlling the movable body to perform three-dimensional photography in which the target object is captured from a plurality of directions by the three-dimensional camera; and an annotation information generating unit that automatically generates, as annotation information, a bounding box circumscribing the target object from the three-dimensional information and information on the shooting direction when the target object was captured for an image of the target object captured by the three-dimensional camera or a camera other than the three-dimensional camera. and a learning database construction unit that constructs a learning database using as learning data multiple data of the annotation information and relative three-dimensional physical information of the target object with respect to the camera used when automatically generating the annotation information , wherein the bounding box circumscribing the target object is obtained by removing the restriction that the straight lines that constitute the bounding box are horizontal and vertical lines, and includes a rotated bounding box that has a smaller area than a bounding box defined by horizontal and vertical lines for the target object and sets a minimum circumscribing rectangle for the target object .

Furthermore, the learning database construction method implemented in the learning database construction system is also within the scope of the technical idea of the present invention. In other words, the method includes a three-dimensional information generating step of generating three-dimensional information of the target object by controlling a movable body that supports a three-dimensional camera to perform three-dimensional photography in which the target object is photographed from a plurality of directions by the three-dimensional camera; an annotation information generating step of automatically generating a bounding box circumscribing the target object as annotation information from the three-dimensional information and information on the photographing direction when the target object was photographed for an image of the target object photographed by the three-dimensional camera or a camera other than the three-dimensional camera; and a learning database constructing step of constructing a learning database using as learning data a plurality of data of the annotation information and three-dimensional physical information of the target object relative to the camera used when automatically generating the annotation information, wherein the bounding box circumscribing the target object is obtained by removing the restriction that the straight lines constituting the bounding box are horizontal and vertical lines, and includes a rotated bounding box that has a smaller area than a bounding box defined by horizontal and vertical lines for the target object and sets a minimum circumscribing rectangle for the target object .

Claims (5)

A three-dimensional camera capable of photographing a target object;
a movable body capable of changing the position and attitude of the three-dimensional camera;
a control device capable of acquiring information on an image of the target object captured by the three-dimensional camera,
The control device includes:
a three-dimensional information generating unit that generates three-dimensional information of the target object by controlling the movable body to perform three-dimensional photography in which the target object is photographed from a plurality of directions by the three-dimensional camera;
an annotation information generating unit that automatically generates annotation information based on three-dimensional information for an image of the target object captured by the three-dimensional camera or a camera other than the three-dimensional camera;
and a learning database construction unit that constructs a learning database using multiple pieces of data, each of which is the annotation information and the relative three-dimensional physical information of the target object with respect to the camera used when automatically generating the annotation information, as learning data. 2. The learning database construction system according to claim 1,
A learning database construction system, characterized in that the three-dimensional physical information is relative three-dimensional posture information of the target object with respect to the camera used when automatically generating the annotation information. 2. The learning database construction system according to claim 1,
A learning database construction system, characterized in that the three-dimensional physical information is information on a specific three-dimensional position of the target object relative to the camera used when automatically generating the annotation information. A control device capable of acquiring information on an image of a target object captured by a three-dimensional camera supported on a movable body and capable of changing a position and attitude by operating the movable body, comprising:
a three-dimensional information generating unit that generates three-dimensional information of the target object by controlling the movable body to perform three-dimensional photography in which the target object is photographed from a plurality of directions by the three-dimensional camera;
an annotation information generating unit that automatically generates annotation information based on three-dimensional information for an image of the target object captured by the three-dimensional camera or a camera other than the three-dimensional camera;
and a learning database construction unit that constructs a learning database using multiple pieces of data, each of which is the annotation information and the relative three-dimensional physical information of the target object with respect to the camera used when automatically generating the annotation information, as learning data. a three-dimensional information generating step of generating three-dimensional information of a target object by controlling a movable body supporting a three-dimensional camera to perform three-dimensional photography in which the target object is photographed from a plurality of directions by the three-dimensional camera;
an annotation information generating step of automatically generating annotation information based on three-dimensional information for an image of the target object captured by the three-dimensional camera or a camera other than the three-dimensional camera;
and a learning database construction step of constructing a learning database using as learning data multiple data of the annotation information and relative three-dimensional physical information of the target object with respect to the camera used when automatically generating the annotation information.

Images