JP2023003765A - Image generation device and control method thereof, image generation system, and program - Google Patents

Abstract

Kind Code: A1 In generating a virtual viewpoint image for a stereoscopic device, a virtual viewpoint is determined in consideration of the imaging range of the stereoscopic device. Kind Code: A1 An image generation device for generating a virtual viewpoint image obtained by observing a virtual space from a virtual viewpoint determines parameters of the virtual viewpoint according to a user operation, and displays a virtual viewpoint image in the virtual space of a stereoscopic device for displaying the virtual viewpoint image. An imaging range corresponding to a depth range in which stereoscopic viewing by the stereoscopic viewing device is possible is acquired based on the characteristics of the stereoscopic viewing device and the determined parameters, and the acquired imaging range is stored in the virtual space. determines whether the determined parameters are appropriate based on whether or not a predetermined condition is satisfied, and if it is determined that the determined parameters are inappropriate, the imaging range is virtualized so as to satisfy the predetermined conditions. Change the viewpoint parameters. [Selection diagram] Fig. 1

Description

The present invention relates to an image generation device, its control method, an image generation system, and a program.

2. Description of the Related Art In the field of computer graphics, objects such as people and buildings placed in a three-dimensional virtual space are imaged and displayed using a virtual camera that can move in the virtual space as a viewpoint. In the following description, an image captured by such a virtual camera (an image observed from a virtual viewpoint) is referred to as a virtual viewpoint image.

Patent Literature 1 discloses a method of setting a camera path along which a virtual camera moves, taking into account the position of an object placed in a virtual space. According to Patent Document 1, the presence or absence of an object in the vicinity of a virtual camera is detected, and when the object and the virtual camera overlap, a camera path is set so that the virtual camera moves avoiding the object.

JP 2014-235596 A

Stereoscopic devices capable of stereoscopic vision are known as display devices for displaying images. As an example of a stereoscopic device, there is a stereoscopic device using a lenticular lens that enables stereoscopic viewing even with the naked eye by delivering parallax images to the left and right eyes of a viewer. A lenticular lens enables a stereoscopic view of parallax images by arranging a large number of lenses on its surface. A range is determined. That is, the stereoscopic depth range is determined as a unique physical property of the stereoscopic device. In the following description, the unique stereoscopic viewable range of the stereoscopic device will be referred to as the imaging range. For the above reason, when generating a virtual viewpoint image, it is necessary to consider the unique characteristics of a stereoscopic device and generate a camera path that allows a desired subject to fit within the imaging range of the stereoscopic device.

However, in general, the imaging range of a stereoscopic device is not considered in generating camera paths. Therefore, for example, in the technique described in Patent Document 1, there is a possibility that a camera path is set that moves the virtual camera to a position and orientation that causes the subject to move out of the imaging range. In the case of generating a virtual viewpoint image observed from an arbitrary virtual viewpoint based on multi-viewpoint images obtained by synchronized shooting of a plurality of imaging devices, and displaying this on a stereoscopic device, the movement of the virtual viewpoint is the same as described above. Similar challenges can arise.

One aspect of the present invention provides a technique for determining a virtual viewpoint in consideration of the imaging range of a stereoscopic device in generating a virtual viewpoint image for the stereoscopic device.

An image processing apparatus according to one aspect of the present invention has the following configuration. i.e.
An image generation device that generates a virtual viewpoint image by observing a virtual space from a virtual viewpoint,
determining means for determining parameters of the virtual viewpoint according to user operation;
The imaging range, which is the imaging range in the virtual space of the stereoscopic viewing device that displays the virtual viewpoint image and corresponds to the depth range in which stereoscopic viewing is possible by the stereoscopic viewing device, is determined by the characteristics of the stereoscopic viewing device and the an acquisition means for acquiring based on the determined parameters;
determination means for determining whether the determined parameters are appropriate based on whether or not the imaging range acquired by the acquisition means satisfies a predetermined condition in the virtual space;
changing means for changing the parameter of the virtual viewpoint so that the imaging range acquired by the acquiring means satisfies the predetermined condition when the determined parameter is determined to be inappropriate; have

According to the present invention, in generating a virtual viewpoint image for a stereoscopic device, a virtual viewpoint is determined in consideration of the imaging range of the stereoscopic device.

1 is a block diagram showing a configuration example of a virtual viewpoint image generation system according to a first embodiment; FIG. FIG. 3 is a diagram showing an arrangement example of a plurality of imaging devices for providing multi-viewpoint images; FIG. 2 is a block diagram showing a hardware configuration example of an image generation device; FIG. 3 is a diagram for explaining an imaging range of a stereoscopic device; FIG. 3 is a block diagram showing an example of the functional configuration of an imaging range calculator; 4 is a flowchart showing processing for generating a virtual viewpoint image according to the first embodiment; 4A and 4B are diagrams for explaining correction processing of a virtual camera according to the first embodiment; FIG. 10 is a flowchart showing virtual viewpoint image generation processing according to the second embodiment. FIG. 8 is a diagram for explaining control of a virtual camera according to the second embodiment; 10 is a flowchart showing virtual viewpoint image generation processing according to the third embodiment. FIG. 11 is a diagram for explaining control of a virtual camera according to the third embodiment;

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In addition, the following embodiments do not limit the invention according to the scope of claims. Although multiple features are described in the embodiments, not all of these multiple features are essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant description is omitted.

[First embodiment]
In the first embodiment, a virtual viewpoint image generation system that controls a virtual camera to maintain an appropriate position/orientation and angle of view based on the imaging range of a stereoscopic device will be described.

<Configuration of Virtual Viewpoint Image Generation System>
FIG. 1 is a block diagram showing a functional configuration example of a virtual viewpoint image generation system 100 according to the first embodiment. A virtual viewpoint image generation system 100 has an instruction input device 101 , a virtual viewpoint image generation device 102 , and a display device 110 . The virtual viewpoint image generation system 100 visualizes and displays a subject existing in a virtual space as a virtual viewpoint image representing the appearance from a virtual viewpoint designated by the user. The instruction input device 101 receives a user's operation input for the virtual viewpoint. The virtual viewpoint image generation device 102 generates a virtual viewpoint image based on the virtual viewpoint indicated by the instruction input device 101 . Note that the virtual viewpoint in the present embodiment is not limited to a viewpoint freely (arbitrarily) designated by the user, and may be a viewpoint selected by the user from a plurality of viewpoint candidates provided in advance, for example. Also, in the present embodiment, the case where the virtual viewpoint image is a moving image will be mainly described, but the virtual viewpoint image may be a still image. The display device 110 displays the virtual viewpoint image generated by the virtual viewpoint image generation device 102 so as to enable stereoscopic viewing via the stereoscopic device 111 . Therefore, the virtual viewpoint image generation device 102 generates two virtual viewpoint images (stereo images) corresponding to parallax, but for the sake of simplicity, generation of one virtual viewpoint image will be described below. Hereinafter, the virtual viewpoint will be referred to as a virtual camera, and the parameters that determine the position and line-of-sight direction of the virtual viewpoint in the virtual space will be referred to as the position and orientation of the virtual camera.

The instruction input device 101 processes information input by receiving a user's instruction, and transmits the information to the virtual viewpoint image generation device 102 . The information transmitted to the virtual viewpoint image generation device 102 includes the position, orientation and angle of view of the virtual camera according to the user's operation of the virtual camera, stereoscopic device information for identifying the stereoscopic device, subject information to be imaged, It contains field information indicating areas where the subject may exist. The instruction input device 101 receives, as a user's instruction, an input using a device connected to the instruction input device 101, such as a mouse or a keyboard.

The position and orientation of the virtual camera are parameter values representing, for example, the amount of movement of the position of the virtual camera in the virtual space and the amount of change in orientation. The parameter value representing the amount of positional movement is, for example, a vector value representing the amount of movement in the XYZ directions. Also, the parameter value representing the amount of change in posture is represented by the amount of rotation about the XYZ axes, for example. These parameter values can be controlled by the user manipulating the position and orientation of the virtual camera using a controller such as a joystick connected to the instruction input device 101 for manipulating three axes of pan, tilt and roll directions. . Needless to say, other than the joystick, a user-operable device such as a mouse or keyboard may be used to manipulate the position and orientation of the virtual camera. Also, the angle of view of the virtual camera indicates the size of the field of view of the virtual viewpoint. The user can use the instruction input device 101 to operate the angle of view of the virtual camera. The virtual camera to be operated by the instruction input device 101 is not limited to one virtual camera. That is, the instruction input device 101 may be capable of operating multiple virtual cameras.

The stereoscopic device information is identification information that specifies the stereoscopic device included in the display device that displays the virtual viewpoint image, and for example, the model number of the display device (stereoscopic device) can be used. From the identification information of the stereoscopic device, it is possible to specify the physical imaging range of the stereoscopic device used to display the virtual viewpoint image. Note that the identification information of the stereoscopic device can be obtained, for example, by the user manually selecting from a list (for example, a list of model numbers) held in advance by the instruction input device 101 . However, the invention is not limited to this, and the instruction input device 101 automatically recognizes the stereoscopic device connected to the virtual viewpoint image generation system 100, and the identification information (model number) of the corresponding stereoscopic device is selected from the list. You may do so.

The subject information of the imaging target is an identifier (hereinafter referred to as ID) assigned to the subject designated as the imaging target. An ID is assigned to each subject in the virtual space. The instruction input device 101 refers to, for example, a table in which the position of each subject in the virtual viewpoint image and the ID of each subject are associated with each other, and associates the position of the subject specified by the user on the screen displaying the virtual viewpoint image. The obtained ID is acquired as subject information of an imaging target. It should be noted that as a method of designating the subject to be imaged on the screen, for example, there is a method of designating the subject by a mouse click operation. However, the designation of the subject to be imaged is not limited to designation by user operation, and the subject to be imaged may be designated by automatically specifying a subject that satisfies a predetermined condition by image processing or the like. good. For example, when generating a virtual viewpoint image of a rugby game, a player holding a ball is identified from the virtual viewpoint image by image processing, and the identified player is selected as a subject to be imaged. Note that the number of subjects set as image forming targets may be one or may be plural.

The field information indicating the area in which the subject may exist is a plurality of coordinates defining the area of a field (hereinafter referred to as set field) set in the virtual space as the area in which the subject may exist. A plurality of coordinates representing the area of the setting field can be set, for example, by the user clicking or dragging a desired location with a mouse on a screen on which the virtual viewpoint image can be confirmed. In addition to setting using a mouse, a table in which coordinates of a plurality of setting fields are recorded in advance may be prepared, and a desired setting field may be selected from this table. For example, on the table, the area of the stadium 211 shown in FIG. 2, each of the two areas obtained by dividing the stadium 211 along the center line, the area including the stadium 211 and the spectator seats, etc. can be selected. Can be registered as a configuration field.

Based on the information transmitted from the instruction input device 101, the virtual viewpoint image generation device 102 satisfies the virtual camera parameters (position and orientation, angle of view) so that the imaging range of the stereoscopic device 111 in the virtual space satisfies predetermined conditions. ). Specifically, the virtual viewpoint image generating apparatus 102 sets the virtual camera so that the relationship between the position of the subject to be imaged in the virtual space and the set field and the imaging range of the stereoscopic device 111 satisfies a predetermined condition. position, orientation, and angle of view. Then, a virtual viewpoint image observed from the virtual camera whose parameters have been determined is generated. The virtual viewpoint image generation device 102 includes, as functional blocks, a holding unit 103, a position/orientation acquisition unit 104, a subject position acquisition unit 105, an imaging range calculation unit 106, a determination unit 107, a correction unit 108, and a virtual viewpoint image generation unit 109. have.

The holding unit 103 holds stereoscopic device information, field information, and subject information received from the instruction input device 101 . The holding unit 103 provides necessary information out of the held information in response to requests from the subject position acquisition unit 105 , the imaging range calculation unit 106 and the determination unit 107 .

The position and orientation acquisition unit 104 acquires the position and orientation of the virtual camera in the virtual space based on the position and orientation of the virtual camera received from the instruction input device 101 . As described above, the position and orientation information from the instruction input device 101 indicates the amount of movement of the position and the amount of change in orientation of the virtual camera. The position/orientation acquisition unit 104 acquires the post-operation position/orientation of the virtual camera by changing the current position and orientation of the virtual camera based on the position/orientation information received from the instruction input device 101 . The acquired position and orientation are provided to the imaging range calculation unit 106 .

The subject position acquisition unit 105 acquires the position (position coordinates) of the subject in the virtual space specified by the subject information (the ID of the subject to be imaged) held in the holding unit 103 . For example, the subject position acquisition unit 105 refers to a subject position table in which the ID of the subject and the position coordinates of the subject in the virtual space are linked, and acquires the position coordinates linked to the ID of the subject specified by the subject information. do. The subject position acquisition unit 105 sends the acquired position coordinates to the determination unit 107 . Note that the position coordinates of each subject in the subject position table are sequentially updated according to the movement of the subject by the function of tracking each subject. For example, a plurality of registers linked to each of a plurality of subjects are prepared in a memory (for example, the RAM 321 in FIG. 3), and the position coordinates of each subject obtained by tracking are written to the corresponding registers. A well-known technique can be applied to the process of identifying and tracking a subject image generated from a subject image or multi-viewpoint images in computer graphics.

The imaging range calculation unit 106 calculates the imaging range of the stereoscopic device 111 determined from the current position/orientation and angle of view of the virtual camera. The imaging range calculation unit 106 acquires the angle of view of the virtual camera from the instruction input device 101, the position and orientation of the virtual camera in the virtual space from the position/orientation acquisition unit 104, and the stereoscopic device information from the holding unit 103. Based on this, the imaging range is calculated. A method of calculating the imaging range by the imaging range calculator 106 will be described later. The imaging range calculation unit 106 sends the calculated imaging range to the determination unit 107 .

The determination unit 107 determines whether the parameters (position and orientation, angle of view) of the virtual camera are appropriate based on whether the imaging range calculated by the imaging range calculation unit 106 satisfies a predetermined condition in the virtual space. do. For example, the determination unit 107 determines based on the position of the subject to be imaged obtained from the subject position obtaining unit 105, the field information obtained from the holding unit 103, and the imaging range obtained from the imaging range calculation unit 106. , whether the position, orientation, and angle of view of the virtual camera are appropriate. For example, the determining unit 107 determines that the position, orientation, and angle of view of the virtual camera are appropriate when the imaging range includes the subject to be imaged and the range of the setting field. Details of the determination method will be described later. Determination section 107 sends this determination result to correction section 108 . The flow of information between the determination unit 107 and the instruction input device 101 corresponds to second and third embodiments described later. Note that the determination unit 107 determines the suitability of one of the two virtual cameras for generating the parallax images, but determines the suitability of both the two virtual cameras for generating the parallax images. You may do so. In this case, it may be determined that the parameters of the virtual camera are appropriate if the parameters of both virtual cameras are appropriate, or that the parameters of one of the virtual cameras are appropriate. It may be determined that the parameters are adequate.

When the determining unit 107 determines that the parameters of the virtual camera are inappropriate, the modifying unit 108 modifies the parameters of the virtual camera so that the imaging range satisfies a predetermined condition. In the present embodiment, the correction unit 108 corrects the position/orientation and the angle of view of the virtual camera so that the positional relationship between the position of the subject to be imaged and the range of the setting field and the imaging range becomes appropriate in the virtual space. do. The correction unit 108 sends the corrected position/orientation and angle of view of the virtual camera to the imaging range calculation unit 106 . The image formation range calculation unit 106 calculates the image formation range again based on the corrected position, orientation, and angle of view of the virtual camera. The determination unit 107 uses the newly calculated image formation range to calculate the corrected virtual camera. It is determined whether the position, orientation, and angle of view of are appropriate. When the correction unit 108 determines that the position/orientation and angle of view of the virtual camera are appropriate, the correction unit 108 sends the position/orientation and angle of view of the virtual camera at that time to the virtual viewpoint image generation unit 109 .

The virtual viewpoint image generation unit 109 generates a virtual viewpoint image as a parallax image to be displayed on the display device 110 based on the position/orientation and angle of view of the virtual camera supplied from the correction unit 108 . The display device 110 displays the parallax images generated by the virtual viewpoint image generation unit 109 . A user can observe a stereoscopic image through the stereoscopic device 111 .

A plurality of instruction input devices 101 may be connected to one virtual viewpoint image generation device 102 , or a plurality of virtual viewpoint image generation devices 102 may be connected to one instruction input device 101 . may be Further, when connecting the virtual viewpoint image generation device 102 and the instruction input device 101 in a one-to-many or many-to-one connection, a wired communication cable may be used for the connection, or wireless communication may be used. may A configuration in which the instruction input device 101, the virtual viewpoint image generation device 102, and the display device 110 are connected via a relay device such as a router or a network hub may also be used.

<Arrangement example of a plurality of imaging devices>
The subject in the virtual space may be a subject image generated by computer graphics, or may be a subject image generated based on images actually captured by a plurality of imaging devices. FIG. 2 shows the imaging in the case of generating a subject image based on a plurality of photographed images (multi-viewpoint images) obtained by synchronous photographing of a plurality of imaging devices (when the subject image is generated based on an actually photographed image). 1 shows an example of arrangement of equipment.

For example, a plurality of imaging devices 210a to 210n are arranged at different positions so as to surround a stadium 211 where a game such as soccer or rugby is played, and multi-viewpoint images are acquired by performing synchronized photography with the imaging devices 210a to 210n. be. The virtual viewpoint image generation unit 109 acquires a foreground image by extracting a foreground area corresponding to a predetermined subject such as a person or a ball from the acquired multi-viewpoint image, and extracts a background area other than the foreground area. Get the background image. After that, the virtual viewpoint image generation unit 109 generates a foreground model representing the three-dimensional shape of a predetermined subject and texture data for coloring the foreground model based on the foreground image. The virtual viewpoint image generation unit 109 also generates a background model representing the three-dimensional shape of a background such as a stadium and texture data for coloring the background model based on the background image. Then, the virtual viewpoint image generation unit 109 maps texture data to the foreground model and the background model, and performs rendering according to the position of the virtual viewpoint, the line-of-sight direction (the position and orientation of the virtual camera), and the angle of view. By doing so, an image of the subject is generated in the virtual space. However, the method of generating the virtual viewpoint image is not limited to this. For example, various methods can be used, such as generating a subject image by projective transformation of a captured image without using a three-dimensional model.

Note that the plurality of imaging devices 210 may not be installed over the entire circumference of the stadium 211, which is the imaging area. A plurality of imaging devices may be installed only in a partial direction of the imaging area depending on restrictions on the installation location or the like. Further, it goes without saying that the number of imaging devices 210 is not limited to the example shown in FIG. For example, if the imaging area is a soccer stadium, about 30 imaging devices 210 may be installed around the stadium 211 . Imaging devices having different functions, such as a telephoto camera and a wide-angle camera, may also be installed.

<Hardware Configuration of Virtual Viewpoint Image Generation Device 102>
FIG. 3 is a block diagram showing a hardware configuration example of the virtual viewpoint image generation device 102. As shown in FIG.

A CPU (Central Processing Unit) 320 executes various processes using programs and data stored in a RAM (Random Access Memory) 321 or a ROM (Read Only Memory) 322 .

The CPU 320 performs overall operation control of the virtual viewpoint image generation device 102 and implements each functional unit of the virtual viewpoint image generation device 102 described above. Note that the virtual viewpoint image generation device 102 may have one or more dedicated hardware different from the CPU 320, and at least part of the processing by the CPU 320 may be executed by the one or more dedicated hardware. . Examples of dedicated hardware include ASICs (Application Specific Integrated Circuits), FPGAs (Field Programmable Gate Arrays), and DSPs (Digital Signal Processors).

The ROM 322 holds programs and data. The RAM 321 has a work area for temporarily storing programs and data read from the ROM 322 . Also, the RAM 321 provides a work area used when the CPU 320 executes each process. The operation input unit 323 is, for example, a three-axis controller such as a mouse, a keyboard, and a joystick, and acquires information operated by the user. However, the operation input unit 323 is not limited to these. The external interface 324 is an interface for connecting with an external device. For example, the external interface 324 transmits images to an external device via an image output port such as HDMI (registered trademark) (high-definition multimedia interface) or SDI (serial digital interface), or via Ethernet or the like. The instruction input device 101 that accepts user operations and the display device 110 having the stereoscopic device 111 can be connected to the external interface 324 .

In the virtual viewpoint image generation system 100, the virtual viewpoint image generation device 102, the instruction input device 101, and the display device 110 have been described as separate devices, but the present invention is not limited to this. Two or more of these devices may be implemented in one device. For example, the virtual viewpoint image generation device 102 and the instruction input device 101 may be realized by one device. In that case, for example, the instruction input device 101 can be implemented by the operation input unit 323 .

<Regarding the imaging range of the stereoscopic device 111>
The imaging range of the stereoscopic device 111 included in the display device 110 will be described with reference to FIG. FIG. 4A is a schematic diagram showing how the display device 110 displays a person 404 and a person 405 as subjects on the screen 403. FIG. A person 404 drawn with a solid line indicates that stereoscopic vision formed by the stereoscopic device 111 is possible. On the other hand, a person 405 drawn with a dotted line cannot be stereoscopically imaged by the stereoscopic device 111 and is observed blurred. Shooting conditions (position and orientation, angle of view) of the virtual camera at this time will be described with reference to FIGS. 4(b) and 4(c).

FIG. 4B is a schematic diagram showing a state in which the virtual camera 400 captures a virtual viewpoint image to be displayed on the display device 110. As shown in FIG. FIG. 4(c) is a schematic diagram of the imaging state of FIG. 4(b) viewed from the side. A person 404 exists at a position closer to the virtual camera 400 than a person 405 does. A display range 401 indicates a range that can be displayed on the screen of the display device 110 within the shooting range of the virtual camera 400 . Only the subject within the display range 401 is displayed on the display device 110 . Strictly speaking, the imaging range and the display range are different, but for the sake of simplicity of explanation, the imaging range and the display range 401 are assumed to be equivalent. It should be noted that the display range can be obtained from the imaging range based on the parameters of the display device, and the method for obtaining this is well known. The imaging range 402 is a range determined based on the depth range in which the stereoscopic device 111 can view the subject stereoscopically. For a subject existing inside the imaging range 402 , it is possible to provide stereoscopic vision formed through the stereoscopic device 111 . The imaging range 402 is, for example, a range having a truncated quadrangular pyramid shape obtained by cutting out a stereoscopically visible depth range from the display range 401 having a quadrangular pyramid shape. In the display range 401, the object inside the imaging range 402 can be stereoscopically observed. In FIGS. 4B and 4C, since the person 404 is positioned within the imaging range 402, the stereoscopic view formed by the stereoscopic viewing device 111 is observed. On the other hand, since the person 405 is positioned outside the imaging range 402, the stereoscopic view formed by the stereoscopic device 111 cannot be observed, resulting in a blurred image, for example.

The relationship shown in FIG. 4B between the imaging range, the display range 401, and the imaging range 402 is due to the characteristics of the display device 110 and the stereoscopic device 111. FIG. For example, one stereoscopic device uses a lenticular lens that enables stereoscopic viewing even with the naked eye by delivering parallax images to the left and right eyes of a viewer. A lenticular lens generates parallax images by arranging a large number of lenses on its surface. Physical parameters such as the number of lenticular lenses, optical path length from pixels to lenses, and pixel pitch determine the depth range in which stereoscopic viewing is possible in the stereoscopic device 111. This depth range, the position and orientation of the virtual camera, and the image are determined. The imaging range is determined from the corners.

An imaging range 402 within the imaging range of the virtual camera 400 is an estimated area where the physical imaging range of the stereoscopic viewing device exists when the generated virtual viewpoint image is displayed on the stereoscopic viewing device. is set by In this embodiment, the imaging range 402 can be calculated from the imaging conditions (position, orientation and angle of view) of the virtual camera 400 and imaging range parameters of the stereoscopic device 111 . For example, as shown in FIG. 4C, the imaging range parameter of the stereoscopic device 111 represents the imaging range by distances a and b from the virtual camera 400 (virtual viewpoint). In this case, the imaging range 402 is, for example, the distance a and b It is obtained as the area between Due to the characteristics of the stereoscopic device 111 described above, it is difficult to set the entire display range of the device as the imaging range, and the imaging range 402 is narrower than the display range 401 .

A setting field 406 indicates a range in which the subject can move, and is indicated by tape or the like in real space, for example. The setting field 406 is also set to the corresponding location in virtual space. The setting field 406 in the virtual space can be set by, for example, the user dragging the display screen of the virtual viewpoint image along the tape in the real space. It should be noted that the setting field does not have to be the range explicitly shown on the tape or the like. For example, a range surrounded by lines outside the stadium 211 in FIG. 2, a stage range in a play, or the like can be used as the setting field. The instruction input device 101 transmits the vertex coordinates forming the outer frame of the setting field 406 set in the virtual space to the virtual viewpoint image generation device 102 . The coordinates representing the setting field are held in the holding unit 103 and used to determine whether or not the shooting state of the virtual camera 400 is appropriate (whether or not the position/orientation and angle of view of the virtual camera 400 need to be corrected).

<Explanation of Image Formation Range Calculation Unit 106>
FIG. 5 is a block diagram showing a detailed functional configuration example of the imaging range calculation unit 106. As shown in FIG. The imaging range calculation unit 106 includes an imaging range calculation unit 500 and an imaging range acquisition unit 501 . Based on the angle of view α of the virtual camera 400 received from the instruction input device 101 and the position and orientation of the virtual camera 400 supplied from the position/orientation acquisition unit 104, the imaging range calculation unit 500 calculates the Calculate the shooting range.

The imaging range acquisition unit 501 calculates the position coordinates of the imaging range 402 based on the imaging range of the virtual camera 400 calculated by the imaging range calculation unit 500 and the identification information of the stereoscopic device 111 read from the holding unit 103 . do. The imaging range acquisition unit 501 has, for example, a table in which parameters indicating the imaging range (for example, distance a and distance b in FIG. 4C) are registered for each identification information (model number) of a stereoscopic device. there is The imaging range acquisition unit 501 refers to this table, acquires a parameter corresponding to the identification information (model number) indicated by the stereoscopic device information acquired from the holding unit 103, and obtains the imaging range calculated by the imaging range calculation unit 500. The position coordinates of the imaging range 402 included in are calculated. The imaging range acquisition unit 501 sends the calculated position coordinates of the imaging range 402 to the determination unit 107 .

Note that the imaging range acquisition unit 501 refers to a table in which the virtual camera parameters (position and orientation, angle of view) and the position coordinates of the imaging range of the stereoscopic device are associated and registered to obtain the current imaging range. You may make it acquire a range. The imaging range acquisition unit 501 can acquire from the table the imaging range corresponding to the current parameters of the virtual camera acquired from the instruction input device 101 and the position/orientation acquisition unit 104 . Therefore, the imaging range calculation unit 500 can be omitted, and the processing load of the imaging range acquisition unit 501 can be reduced. A table is prepared for each stereoscopic device, and the imaging range acquisition unit 501 selects a table to refer to based on the stereoscopic device information. Also, if the parameters that match the current parameters of the virtual camera are not registered in the table, the imaging range registered corresponding to the parameters that are closest to the current parameters may be used. Alternatively, an imaging range corresponding to the current parameter may be obtained by interpolation processing from imaging ranges of a plurality of parameters close to the current parameter.

<Processing for Generating Virtual Viewpoint Image According to First Embodiment>
A process of generating a virtual viewpoint image by the virtual viewpoint image generating device 102 will be described with reference to the flowchart of FIG. 6 . This process corrects the position and orientation (angle of view) of the virtual camera 400 so that the image formation range of the stereoscopic device 111 includes the object to be imaged and the setting field, and uses the corrected virtual camera. to generate a virtual viewpoint image.

In step S601, initialization is performed. Specifically, the holding unit 103 receives the identification information of the stereoscopic device 111, the subject information (subject ID) of the selected imaging target, and the field information (coordinates of the setting field) from the instruction input device 101, and retains information about In step S<b>602 , the position/orientation acquisition unit 104 acquires operation information on the position and orientation of the virtual camera from the instruction input device 101 , and the imaging range calculation unit 106 acquires information on the angle of view of the virtual camera from the instruction input device 101 . As described above, the virtual camera position/orientation manipulation information represents changes in the virtual camera position/orientation due to user manipulation by vectors.

In step S<b>603 , the position/orientation acquisition unit 104 acquires the position/orientation of the virtual camera in the virtual space based on the position/orientation information acquired from the instruction input device 101 . In step S<b>604 , the subject position acquisition unit 105 acquires the subject information of the imaging target from the holding unit 103 and acquires the position coordinates of the imaging target subject in the virtual space. The subject position acquisition unit 105 provides the acquired position coordinates to the determination unit 107 .

In step S605, the imaging range calculation unit 106 calculates the position of the imaging range in the virtual space of the stereoscopic device 111 at the current position/orientation and angle of view of the virtual camera. Specifically, the imaging range calculation unit 106 acquires the stereoscopically visible depth range based on the stereoscopic device information (identification information of the stereoscopic device 111) acquired from the holding unit 103. FIG. Further, the imaging range calculation unit 106 acquires the position/orientation and the angle of view of the virtual camera from the position/orientation acquisition unit 104 and the instruction input device 101, respectively. Then, the imaging range calculation unit 106 calculates the position coordinates of the imaging range in the virtual space based on the position and orientation of the virtual camera, the angle of view, and the depth range of the stereoscopic device 111 . The imaging range calculation unit 106 provides the position coordinates of the calculated imaging range to the determination unit 107 .

In step S606, the determination unit 107 determines whether the parameters including the position/orientation and the angle of view of the virtual camera are appropriate based on whether the imaging range in the virtual space satisfies a predetermined condition. The determining unit 107 of this embodiment determines whether the position, orientation, and angle of view of the virtual camera are appropriate based on the position of the imaging range in the virtual space, the position of the subject to be imaged, and the field information (coordinates of the setting field). determine whether For example, it satisfies both that the target subject is within the imaging range (condition 1 below) and that a predetermined percentage or more of the imaging range belongs to the set field (condition 2 below). In this case, it is determined that the imaging range in the virtual space satisfies a predetermined condition. Then, when the imaging range in the virtual space satisfies a predetermined condition, it is determined that the position/orientation and angle of view of the virtual camera are appropriate. As condition 2, for example, a predetermined ratio or more (for example, 80% or more) of a plane (cut plane) where the plane including the setting field and the imaging range intersect belongs to the setting field. be done. Alternatively, condition 2 may further include that the area of the region belonging to the set field on the cutting plane has a size equal to or larger than a predetermined value. If it is determined that at least one of Condition 1 and Condition 2 is not satisfied, it is determined that the position/orientation and angle of view of the virtual camera are inappropriate (NO in step S606). In this case, since it is necessary to correct at least one of the position and orientation and the angle of view of the virtual camera, the process proceeds to step S607. On the other hand, if both conditions 1 and 2 are satisfied, the determination unit 107 determines that the position, orientation and angle of view of the virtual camera are appropriate (YES in step S606), and the process proceeds to step S608. Although an example using conditions 1 and 2 has been shown above, the present invention is not limited to this. For example, only condition 1 may be used in the determination of step S606, or the fruit of condition 2 may be used.

In step S607, the modifying unit 108 modifies the position and orientation (angle of view) of the virtual camera so that Condition 1 and Condition 2 are satisfied. FIG. 7 shows an example of correction of the position and orientation of the virtual camera. FIG. 7 illustrates an example of moving the virtual camera (changing the position and orientation) so that the target subject fits within the imaging range and a predetermined ratio of the imaging range fits within the set field. FIG. 7A is a schematic diagram showing how the virtual camera 700 is photographing a person 701 as a subject to be imaged. FIG. 7A shows a state in which condition 1 is satisfied but condition 2 is not satisfied. That is, in the position, orientation, and angle of view of the virtual camera 700 , the person 701 , which is the subject to be imaged, is within the image formation range 702 of the stereoscopic device 111 , but a predetermined ratio of the image formation range 702 is within the setting field 703 . It shows an example that does not fit. In this case, in step S607, the modifying unit 108 modifies the position and orientation of the virtual camera 700 so that both conditions 1 and 2 are satisfied. In FIG. 7B, by correcting the position of the virtual camera 700, the person 701, which is the subject to be imaged, fits within the imaging range 702, and more than a predetermined ratio of the imaging range 702 is in the setting field 703. Show how it fits.

Correction of the position of the virtual camera as described above can be realized, for example, as follows. For example, the position and orientation of the virtual camera can be returned to the previous state (the state immediately before the position and orientation were changed by the operation). According to this method, when the user performs an operation that does not satisfy Condition 1 or Condition 2, the position, orientation, and angle of view of the virtual camera do not change, and the virtual camera remains stationary. As another method, the position, orientation and angle of view changed by the user operation are gradually returned to the position, orientation and angle of view of the virtual camera before the change until conditions 1 and 2 are satisfied. can be Alternatively, at least one of the position/orientation and the angle of view after being changed by the user's operation may be gradually changed until the conditions 1 and 2 are satisfied. For example, the virtual camera may be moved so that the moving distance of the virtual camera for correction is minimized, or the virtual camera may be moved along the line-of-sight direction (optical axis) of the virtual camera. good. Alternatively, the angle of view may be changed so as to satisfy the conditions 1 and 2 without moving the virtual camera 700 . Through the above-described processing, even if the user arbitrarily manipulates the position, orientation and angle of view of the virtual camera, the subject to be imaged is contained within the imaging range and the ratio of the setting field to the imaging range is maintained. becomes possible.

After the position and orientation (angle of view) of the virtual camera are corrected in step S607, the process returns to step S605. After that, if it is determined in step S606 that the position, orientation and angle of view of the virtual camera are appropriate (if it is determined that conditions 1 and 2 are satisfied), the process proceeds to step S608. In step S608, the virtual viewpoint image generation unit 109 generates a virtual viewpoint image using the three-dimensional shape information and textures obtained from the multi-viewpoint images, and the corrected position and orientation (angle of view) of the virtual camera.

As described above, according to the first embodiment, the virtual camera is configured so that the specified subject is included in the imaging range of the stereoscopic device, and a predetermined ratio or more of the imaging range belongs to the specified field. Position, attitude and angle of view are controlled. This makes it possible for the user to generate a virtual viewpoint image that enables a stereoscopic view of the designated subject without checking whether the subject fits within the imaging range. Also, when displaying a virtual viewpoint image on a stereoscopic device, it is possible to maintain a state in which a designated field (for example, an area in which an object may exist) can be stereoscopically viewed.

As described above, under condition 2, it is sufficient that a predetermined ratio or more of the imaging range belongs to the set field. Therefore, in FIG. 7B, the person 701, which is the subject to be imaged, exists within the setting field 703, but even if the person 701 goes out of the setting field 703, the conditions 1 and 2 are satisfied. There may be an imaging range such that . In this way, since conditions 1 and 2 are used to determine whether the position, orientation and angle of view of the virtual camera are appropriate, the setting field can be changed while flexibly maintaining the state in which stereoscopic viewing of the object to be imaged is possible. Control of the virtual camera is realized to provide a rich imaging range. Also, the user may be able to set the threshold value (predetermined ratio) used for condition 2 described above.

[Second embodiment]
In the first embodiment, the method of controlling the position coordinates of the virtual camera 700 so that the imaging range 702 contains the person 701 and is within the setting field 703 has been described. However, when the control of the first embodiment is performed, the user may feel uncomfortable that the virtual camera does not move even when the user tries to move it, or that the virtual camera does not move as expected. The second embodiment makes it possible to alleviate the user's discomfort by notifying the user that the position coordinates of the virtual camera will be corrected. In addition, in the following description, the same reference numerals are given to the same configurations as in the first embodiment.

<Functional Configuration of Virtual Viewpoint Image Generation System>
A functional configuration of a virtual viewpoint image generation system according to the second embodiment will be described with reference to FIG. As described in the first embodiment, the virtual viewpoint image generation device 102 corrects the position/orientation and angle of view of the virtual camera based on each piece of information transmitted from the instruction input device 101, Generate a virtual viewpoint image. More specifically, the virtual viewpoint image generation device 102 operates the virtual camera so that any subject falls within the imaging range (condition 1) and a predetermined ratio or more of the imaging range belongs to the set field (condition 2). control the position, orientation and angle of view to generate a virtual viewpoint image. When the determination unit 107 of the second embodiment determines that the position/orientation and the angle of view of the virtual camera are inappropriate, the determination unit 107 sends information to that effect to the instruction input device 101 so that the position/orientation and the image angle of the virtual camera are corrected. Notify the user that the corner will be modified.

Determination unit 107 determines whether the imaging range is set to a position including the object to be imaged and the setting field, and sends the determination result to correction unit 108 and instruction input device 101 . When the instruction input device 101 receives a notification from the determination unit 107 indicating that the position/orientation and angle of view of the virtual camera are determined to be inappropriate, the instruction input device 101 receives a display device (improper display device) connected to the instruction input device 101 . ) indicates that the position and orientation (angle of view) of the virtual camera will be corrected. This informs the user that the virtual camera will be modified. Note that the notification is not limited to the display on the display device, and other means perceivable by the user, such as voice notification by a speaker, may be used.

<Flowchart of control processing according to the second embodiment>
FIG. 8 is a flowchart for explaining virtual viewpoint image generation processing by the virtual viewpoint image generation device 102 according to the second embodiment. The same step numbers are given to the same processes as in the first embodiment (FIG. 6).

If the determination unit 107 determines in step S606 that the post-operation position/orientation and angle of view of the virtual camera are not appropriate (NO in step S606), the process proceeds to step S606 to correct the position/orientation and angle of view of the virtual camera. Proceed to S901. In step S901, the determination unit 107 notifies the instruction input device 101 that the position and orientation of the virtual camera after the operation are inappropriate. After that, in step S607, the position and orientation of the virtual camera are corrected as described in the first embodiment. The instruction input device 101 that has received the notification from the determination unit 107 in step S901 notifies that the position/orientation and angle of view of the virtual camera will be inappropriate and that the position/orientation and angle of view of the virtual camera will be corrected. , for example, via display device 110 to the user.

On the other hand, if the determination unit 107 determines in step S606 that the position, orientation, and angle of view of the virtual camera after the operation are appropriate (YES in step S606), the process advances to step S902. In step S902, the determination unit 107 determines whether or not the instruction input device 101 has been notified in step S901. If it is determined that notification has been made (YES in step S902), the process transitions to step S903. In step S<b>903 , the determination unit 107 stops notifying the instruction input device 101 that the position and orientation (angle of view) of the virtual camera are in an inappropriate state, which was performed in step S<b>901 . After that, the process moves to step S608. On the other hand, if it is determined that notification has not been made (NO in step S902), step S903 is skipped and the process proceeds to step S608.

By performing the above control, when the position coordinates of the virtual camera are to be corrected, it is possible to notify the user to that effect.

Note that in the above description, the determination unit 107 determines whether or not the position, orientation, and angle of view of the virtual camera are appropriate using the imaging range of the stereoscopic device 111, and notifies the instruction input device 101 according to the determination result. decided whether to do it or not. Here, the imaging range of the stereoscopic device 111 is used to determine whether or not to notify the instruction input device 101 and to determine whether or not to correct the position/orientation and angle of view of the virtual camera. However, the present disclosure is not limited to this. For example, a determination range different from the imaging range of the stereoscopic device 111 may be set for determining whether or not to notify so that the user can receive the notification at an earlier timing.

FIG. 9 shows a setting example of a determination range for notification. For example, as shown in FIG. 9A , a slightly smaller determination range 901 is provided inside the imaging range 900 of the stereoscopic device 111 . In step S<b>606 , the determination unit 107 uses the determination range 901 instead of the imaging range 900 to determine whether or not to notify the instruction input device 101 . That is, the determination unit 107 uses the imaging range 900 to determine whether the position/orientation and the angle of view of the virtual camera are appropriate, and uses the determination range 901 to determine whether or not to notify the instruction input device 101 . Also, the determination range 901 may be changed according to the motion state of the virtual camera. For example, when the moving speed of the virtual camera is fast, the determination range 901 is set narrowly as shown in FIG. 10B, and when the moving speed of the virtual camera is slow, the determination Set the range 901 wider. That is, the faster the moving speed, the smaller the determination range 901 . As a result, the user can be notified at an appropriate timing according to the moving speed of the virtual camera.

As described above, by notifying the user that the position coordinates of the virtual camera will be corrected, there is an effect of making it possible to alleviate the sense of incongruity.

[Third embodiment]
In the third embodiment, in the control processing of the first and second embodiments, the subject designated as the imaging target is permitted to exit the imaging range or exit the imaging range from the designated field. make it possible.

The same numbers as those in the first and second embodiments indicate the same components, and the description thereof is omitted here.

<Functional Configuration of Virtual Viewpoint Image Generation System>
A functional configuration of a virtual viewpoint image generation system according to the third embodiment will be described with reference to FIG. As described in the first embodiment, the virtual viewpoint image generation device 102, based on each piece of information transmitted from the instruction input device 101, determines whether an arbitrary subject falls within the imaging range and is imaged within a specified field. A virtual viewpoint image is generated by controlling the position coordinates of the virtual camera so as to fit within the range. In the third embodiment, when a permission instruction indicating “permit” is input from the instruction input device 101, regardless of whether the position/orientation and angle of view of the virtual camera are inappropriate, the correction unit 108 is controlled so that the position, orientation, and angle of view of the virtual camera are not corrected.

The instruction input device 101 sends a permission instruction to the virtual viewpoint image generation device 102 in response to a predetermined user operation. The permission instruction is, for example, a binary signal indicating whether or not to permit the position and orientation (angle of view) of the virtual camera to no longer satisfy the moving condition of the imaging range described above. The permission instruction can be set by the user via a mouse or keyboard connected to the instruction input device 101, for example.

As in the first embodiment, the determination unit 107 determines whether the imaging range is set to include the subject and the field to be imaged. However, in the third embodiment, the determination unit 107 performs determination when the permission instruction transmitted from the instruction input device 101 indicates non-permission, but does not perform determination when the permission instruction indicates permission. Therefore, when the permission instruction indicates permission, the correction unit 108 changes the position and orientation acquired by the position/orientation acquisition unit 104 regardless of whether the imaging range is set to a position including the subject and the field to be imaged. It will be output as is.

<Flow chart of control processing according to the third embodiment>
FIG. 10 is a flowchart for explaining virtual viewpoint image generation processing by the virtual viewpoint image generation device 102 of the third embodiment. The same step numbers are given to the same processes as in the first embodiment (FIG. 6).

In step S<b>1001 , the determination unit 107 receives a permission instruction set by the user via the instruction input device 101 . At step S1002, the determination unit 107 determines whether or not the permission instruction received from the instruction input device 101 at step S1001 indicates permission. If the permission instruction indicates permission (YES in step S1002), the process skips step S606 and proceeds to step S608. On the other hand, if the permission instruction indicates non-permission (NO in step S1002), the process proceeds to step S606.

By performing the control as described above, it is possible to move the designated subject out of the imaging range or out of the imaging range from the designated field. For example, by giving a permission instruction indicating permission from the instruction input device 101, the virtual camera 700 can be moved to the state shown in FIG.

In the above description, when the permission instruction indicates permission, the suitability of the position/orientation and angle of view of the virtual camera is not determined, but the present invention is not limited to this. In the configuration in which condition 1 and condition 2 are used for determination as in the first embodiment, permission instructions may be set separately for condition 1 and condition 2. FIG. For example, it may be possible to set such that the object to be imaged is permitted to leave the imaging range, but it is not permitted that a predetermined percentage or more of the imaging range does not belong to the set field. .

Alternatively, the imaging range may be divided into a plurality of parts with arbitrary size and position parameters, and different conditions for each part may be used to determine whether the position, orientation and angle of view of the virtual camera are appropriate. For example, the imaging range is divided into left and right, and the left half of the imaging range is determined using only condition 1, and the right half of the imaging range is determined using both conditions 1 and 2. make it As a result, it is possible to control so that a specific portion of the imaging range is allowed to exit the set field. By performing such control, it is possible to set the virtual camera to a position, attitude and angle of view such that a part of the imaging range is outside the set field, as shown in FIG. 11, for example. can be increased, and the image can be formed in the range where the image is to be formed without fail. It should be noted that the user may be able to set a parameter for dividing the imaging range.

As described above, according to the third embodiment, the user can appropriately relax the conditions for judging whether the parameters of the virtual camera are appropriate, and the virtual camera can be operated flexibly.
[Other embodiments]
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the claims are appended to make public the scope of the invention.

100: Virtual Viewpoint Image Generation System 101: Instruction Input Device 102: Image Generation Device 103: Holding Unit 104: Position and Orientation Acquisition Unit 105: Subject Position Acquisition Unit 106: Imaging Range Calculation Unit 107: Judgment Section 108: Correction Section 109: Virtual Viewpoint Image Generation Section 400: Virtual Camera 402: Imaging Range 403: Display Device

Claims (17)

An image generation device that generates a virtual viewpoint image by observing a virtual space from a virtual viewpoint,
determining means for determining parameters of the virtual viewpoint according to user operation;
The imaging range, which is the imaging range in the virtual space of the stereoscopic viewing device that displays the virtual viewpoint image and corresponds to the depth range in which stereoscopic viewing is possible by the stereoscopic viewing device, is determined by the characteristics of the stereoscopic viewing device and the an acquisition means for acquiring based on the determined parameters;
determination means for determining whether the determined parameters are appropriate based on whether or not the imaging range acquired by the acquisition means satisfies a predetermined condition in the virtual space;
changing means for changing the parameter of the virtual viewpoint so that the imaging range acquired by the acquiring means satisfies the predetermined condition when the determined parameter is determined to be inappropriate; An image generation device characterized by comprising: The characteristics of the stereoscopic device represent the imaging range by a distance from a virtual viewpoint,
2. The imaging range according to claim 1, wherein said obtaining means obtains, as said imaging range, a range cut out by said distance out of a display range obtained based on said determined parameter of said virtual viewpoint. Image production device. 3. The image generating apparatus according to claim 1, wherein said predetermined condition includes a condition that a position of a specific subject is within said imaging range. 4. The image generating apparatus according to claim 3, further comprising designating means for designating said specific subject by user's operation. 5. The predetermined condition according to any one of claims 1 to 4, wherein the predetermined condition includes a condition that a predetermined ratio or more of the imaging range belongs to a specific area set in the virtual space. image production device. 6. The image generation apparatus according to claim 5, wherein the specific area is set by a user's operation on the virtual viewpoint image. 7. The changing means gradually returns the determined parameter to the parameter before being determined until the determining means determines that the parameter is appropriate. 3. An image generation device according to claim 1. The changing means gradually changes at least one of the position of the virtual viewpoint, the line-of-sight direction, and the angle of view included in the parameters until the determining means determines that the parameters are appropriate. 7. An image generation device according to any one of claims 1 to 6. 9. The image generation apparatus according to any one of claims 1 to 8, further comprising notification means for notifying that said changed parameter is determined to be inappropriate by said determination means. A determination range smaller than the imaging range is set within the imaging range, and the determined parameter becomes inappropriate based on whether the determination range satisfies the predetermined condition. 9. The image generating apparatus according to any one of claims 1 to 8, further comprising notifying means for notifying. 11. The image generation apparatus according to claim 10, wherein said notification means changes said determination range based on a moving speed of a virtual viewpoint. 12. The image generating apparatus according to claim 11, wherein the notifying means reduces the size of the determination range as the moving speed increases. further comprising permission means for permitting the imaging range to no longer satisfy the predetermined condition in the virtual space in accordance with a user operation;
12. The image generation apparatus according to any one of claims 1 to 11, wherein said determination means does not determine whether said determined parameters are appropriate when said permission is granted by said permission means. Further comprising means for setting a plurality of parts in the imaging range according to a user operation,
14. The image generating apparatus according to any one of claims 1 to 13, wherein the determining means determines whether the determined parameters are appropriate using different conditions for each of the plurality of portions. an image generation device according to any one of claims 1 to 14;
a plurality of imaging devices for acquiring multi-viewpoint images for generating virtual viewpoint images in the image generation device;
a display device for displaying a virtual viewpoint image generated by the image generation device;
and an instruction input device that provides a user operation to the image generation device. A control method for an image generation device that generates a virtual viewpoint image obtained by observing a virtual space from a virtual viewpoint,
a determination step of determining a parameter of the virtual viewpoint according to a user operation;
The imaging range, which is the imaging range in the virtual space of the stereoscopic viewing device that displays the virtual viewpoint image and corresponds to the depth range in which stereoscopic viewing is possible by the stereoscopic viewing device, is determined by the characteristics of the stereoscopic viewing device and the an acquisition step of acquiring based on the determined parameters;
a determination step of determining whether the determined parameter is appropriate based on whether the imaging range acquired by the acquisition step satisfies a predetermined condition in the virtual space;
a changing step of changing the parameters of the virtual viewpoint so that the imaging range acquired by the acquiring step satisfies the predetermined condition when the determined parameters are determined to be inappropriate; A control method for an image generation device, comprising: A program for causing a computer to function as each means of the image generating apparatus according to any one of claims 1 to 14.

Images