JP2009205556A - User interface device - Google Patents

Abstract

Conventionally, it has been difficult to synchronize cursor shooting with a camera and display update on a screen.
A user interface device includes: an image ID management unit that manages an identifier of an image that changes at a constant period; and a cursor display unit that displays a cursor image associated with the image ID on a first display screen. 107, a cursor coordinate storage unit 105 that stores the position coordinates of the cursor corresponding to the image ID, a cursor recognition unit 101 that recognizes an image of the cursor displayed on the first display screen 11, and recognition by the cursor recognition unit 101 The position / direction estimation unit 102 that estimates the positional relationship between the operation terminal 2 and the first display screen 11 from the result and the stored cursor position coordinates, and the first display screen 11 based on the estimated position / direction information. The image generation unit 103 generates an image to be displayed on the screen.
[Selection] Figure 5

Description

  The present invention relates to a user interface device for operating / browsing information / content displayed on a display using an operation terminal equipped with a camera.

  Conventionally, for the purpose of realizing a user interface that intuitively operates and browses information displayed on a display, web (Web) content, menu screen, etc., by an operation terminal equipped with an optical sensor such as a camera, Techniques for estimating the position and orientation of the display have been proposed.

  First, there has been a technique for calculating indicated coordinates on a display using a camera mounted on an operation terminal such as a remote controller and a point blinking on the front of the display (see, for example, Patent Document 1). By using this technique, the pointing cursor can be moved in accordance with the direction indicated by the operation terminal held by the user.

In addition, as one of the technologies for realizing Augmented Reality type user interface, the image recognition of the explicit image marker displayed on the display is performed, and the coordinate system indicated by the image marker is estimated, so that a specific display There has been a technique for superimposing information and 3DCG on a place for display (for example, Non-Patent Document 1). These augmented reality user interfaces obtain a coordinate transformation matrix indicating the position and orientation of an image marker with high accuracy from the coordinate values of the four vertices indicated by the rectangular image marker in a coordinate system with the center of the camera image as the origin. This is an application of technology (for example, Non-Patent Document 2).
JP 2007-86995 A Yasue Kishino, Masahiko Tsukamoto, Hiroshi Sakane, Shojiro Nishio: “Autonomous Translucent Visual Marker on Computer Display”, Information Processing Society of Japan Symposium Series (Interaction 2002), Vol. 2002, no. 7, pp. 14-21, 2002. Kato, Mark, Asano, Tachibana: “Augmented reality system based on marker tracking and its calibration”, Transactions of the Virtual Reality Society of Japan Vol. 4 No. 4, pp. 607-616, 1999

  However, in the method of recognizing the blinking point on the front surface of the display, (1) it is necessary to accurately synchronize the timing of shooting with the camera and the timing of updating the screen of the display. 2) Since the position of a point is acquired using a difference image between a plurality of frames, it takes time to recognize, and an accurate coordinate value cannot be acquired when the operation terminal is moved. was there.

  In the method of displaying an explicit image marker on the display, (1) when the image marker is displayed at a specific position on the display, a part of the screen is always hidden by the image marker, which is an obstacle to the user. (2) When dynamically moving the image marker to an unobtrusive position, it becomes impossible to recognize when the image marker deviates from the camera image, and the timing of display display and camera shooting is not taken into consideration. There was a problem that the position and orientation of the display itself could not be obtained accurately.

  Furthermore, since the conventional technology does not focus on improving the accuracy near the center of the camera image, it can be used to display a pointing cursor, but it can be expanded with respect to the content displayed on the display. For the purpose of displaying a real user interface, the accuracy tends to be insufficient, and there is a problem that a CG image displayed in a superimposed manner becomes unstable.

  The present invention solves the above-mentioned conventional problems, and it is not necessary to synchronize the timing of capturing with the camera and the timing of updating the display image, and the explicit marker does not occupy the screen. The purpose is to obtain the position and orientation of the display with high accuracy.

  In order to solve the above-described conventional problems, the user interface device according to the present invention is based on an image recognition result of a captured image obtained by capturing an image of a cursor image displayed on a display screen by the cursor image. An image generation means for generating a display image to be displayed on a display screen, and a plurality of types of one cursor image that changes repeatedly at a predetermined time period. The image ID storage that stores in advance the cursor display means to be synthesized on the display image, a plurality of types of the cursor images, and the image ID for specifying the display image on which each of the cursor images is displayed. Image recognition of the cursor image included in the captured image and the stored contents of the image ID storage unit. A cursor recognizing unit for identifying an image ID of the captured cursor image from the recognized cursor image, calculating a position coordinate on the display screen of the captured cursor image, and calculating the calculated position From the cursor coordinate storage means for storing coordinates in association with the specified image ID, the recognition result by the cursor recognition means, and the position coordinates stored in the cursor coordinate storage means, the image imaging A position estimation unit configured to estimate the display image captured by the apparatus and an instruction coordinate indicating a position currently pointed to by the image capturing apparatus on the display image, and the cursor display unit sets a new image ID A corresponding new cursor image is read from the image ID storage means, and the new cursor image is estimated by the position estimation means. To appear in the serial instruction coordinates, characterized in that to combine the cursor image to the image generating unit.

  With this configuration, the timing of photographing with the camera that is the image capturing device is synchronized with the timing of updating the display image generated by the image generation unit and the display image that displays the cursor image generated by the cursor display unit. Without the explicit marker occupying the screen, the position and orientation of the display can be acquired with high accuracy.

  The image capturing apparatus includes a second display unit that displays the captured image, and the cursor display unit generates a new cursor image based on the position coordinates of the frame estimated by the position estimating unit. However, a new display image may be generated so as to be positioned at the center of the new display image displayed on the second display unit, and displayed on the second display unit.

  With the above configuration of the present invention, a pointing cursor that is a cursor image is displayed at an accurate position indicated by an operation terminal corresponding to an image pickup device, and is a display image generated by an image generation unit, and information displayed on a display And content can be operated and viewed intuitively. At the same time, the cursor image is moved to the center of the camera image, which is a new display image displayed on the second display means, so that the accuracy of the position and orientation at the center of the camera image is increased.

  Furthermore, the cursor display means may change the cursor image and the image ID only when the position coordinates of the cursor image to be displayed are moved on the display screen generated by the image generation means.

  According to the above configuration of the present invention, when the cursor does not move, the periodic change of the cursor image can be made inconspicuous.

  The cursor display means changes the size of the cursor image displayed on the display screen generated by the image generation means based on the estimated positional relationship, and the cursor coordinate storage means displays The position coordinates and size of the cursor image may be stored.

  With the above configuration of the present invention, the cursor can be displayed small when the distance between the user and the display screen is short, and the cursor can be displayed large when the distance between the user and the display screen is long. As a result, not only the visibility of the cursor can be improved, but also the accuracy of image recognition can be improved by making the size of the cursor included in the camera image close to a constant value.

  The cursor display means may change the hue component of the specific area included in the cursor image according to the change in the display image generated by the image generation means, according to the image ID. The cursor display means may change the shape of the cursor image or the locus portion of the cursor image in association with the image ID.

  With the above-described configuration of the present invention, it is possible to easily recognize the periodic change of the cursor display without conspicuous.

  Furthermore, the image capturing apparatus includes a second display unit that displays the captured image, and the image generating unit is based on position and orientation information that represents a relative position of the image capturing apparatus from the estimated position coordinates. Then, an augmented reality image to be displayed on the second display screen may be generated by synthesizing an image captured by the camera and an image generated by CG.

  According to the above configuration of the present invention, the user interface device of the present invention can be easily applied to an augmented reality user interface.

  At this time, the cursor display means determines the size of the cursor image to be displayed on the display image generated by the image generation means based on whether or not the augmented reality image is displayed on the second display screen. It may be changed.

  According to the above configuration of the present invention, in the case of augmented reality image display that requires high recognition accuracy, the cursor display can be enlarged to increase the recognition accuracy. When viewing an augmented reality image through the second display screen, the user often does not get in the way even if the cursor display on the display image generated by the image generation means is displayed large.

  Further, the image generation means cuts out a part of the display image displayed on the second display screen from the estimated position coordinates, based on position orientation information representing a relative position of the image capturing device, The part of the display image that has been cut out may be enlarged and generated.

  According to the above configuration of the present invention, it is possible to realize an application that enlarges and displays the contents displayed on the display image generated by the image generation means in an easy-to-understand manner.

  The user interface device of the present invention captures a cursor display that periodically changes in units of frames when the cursor is moved by the image capturing device of the operation terminal, and recognizes the captured cursor image as an image, thereby displaying the position of the display. The direction can be estimated accurately. Furthermore, by obtaining the indicated coordinates indicated by the operation terminal based on the estimated position and orientation, the pointing cursor can be displayed at an accurate position on the display. In the present invention, an easy-to-use user interface can be realized without installing a special LED light-emitting device on the display side or a means for synchronizing the display and timing of photographing with the image capturing means. In particular, in order to estimate the position and orientation based on the cursor display displayed in the center of the camera image, the recognition accuracy near the center of the camera image is high, and not only a simple user interface for pointing, It can also be used to realize an augmented reality user interface that requires an accurate position and orientation on a camera image. Furthermore, the accuracy of recognition can be further improved by displaying a cursor having an optimal size according to the positional relationship between the operation terminal held by the user and the display.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(Embodiment 1)
The user interface device according to the present invention is “a user who displays a new image including the cursor image based on an image recognition result of a captured image obtained by capturing an image of the cursor image displayed on the display screen. It is an example of an “interface device”, and the cursor image displayed on the first display screen of the information display device is captured by the image capturing unit of the operation terminal, and the captured cursor image is always the second display screen of the operation terminal. The cursor image displayed on the first display screen is moved and displayed so as to be displayed at the center of the screen. As a result, the cursor image displayed on the first display screen can be displayed so as to move within the first display screen following the movement of the operation terminal.

  FIG. 1 is an external view of an information display device 1 and an operation terminal 2 according to Embodiment 1 of the present invention. The operation terminal 2 is an example of “the image capturing apparatus including a second display unit that displays the captured image”. FIG. 1A shows an example of a cursor operation in which a pointing cursor image is displayed at designated coordinates on the first display screen 11 indicated by the operation terminal 2. At this time, for example, a web browser application is displayed on the first display screen 11, and the cursor image moves by moving the operation terminal 2. According to the position where the cursor image has moved, the user operates the user input unit 22 such as an operation button, thereby pressing a button displayed near the cursor or performing a data drag and drop operation. Can do. Also, in the figure, nothing is displayed on the second display screen 23.

  In FIG. 2, the cursor image displayed on the first display screen 11 is photographed by the operation terminal 2, and the photographed cursor image is image-analyzed to be displayed on the first display screen 11 in the next frame. It is a figure explaining the mechanism which determines the display position of a cursor image. In FIG. 2A, the image data of the cursor image taken by the operation terminal 2 is transmitted to the information display device 1, and the information display device 1 analyzes the cursor image data to determine the next display position of the cursor image. It is a figure which shows an example of the feedback loop to determine. First, when the information display device 1 is activated, the cursor image is displayed at a predetermined position on the first display screen 11. The cursor terminal is photographed by the operation terminal 2 (1), and image data obtained by the photographing is transmitted to the main body of the information display device 1 (2). The user interface device 10 provided in the information display device 1 analyzes the image data received from the operation terminal 2 and determines the next display position of the cursor image (3). At this time, the next display position of the determined cursor image is the display of the cursor image on the first display screen 11 so that the cursor image is displayed at the center of the second display screen 23 of the operation terminal 2. The position is determined. In accordance with this determination, the display position of the cursor image is moved on the first display screen 11. The operation terminal 2 captures a cursor image that is moved and displayed on the first display screen 11 (4), and transmits image data obtained by the capture to the information display device 1. Note that the present invention is actually applicable even when the operation terminal 2 does not have the second display screen 23. However, when the cursor image is displayed on the second display screen 23 of the operation terminal 2, the operation terminal 2 changes the second image according to the movement of the cursor image displayed on the first display screen 11. The display position of the cursor image displayed on the display screen 23 is moved. FIG. 2B is a diagram illustrating a relationship between a cursor image displayed on the first display screen 11 and a cursor image when the information display device 1 is photographed from an oblique direction. As shown on the right in FIG. 2B, the cursor image is displayed so as to look circular when viewed from the front. On the other hand, when the operation terminal 2 captures this cursor image in an oblique direction from a position away from the information display device 1, as shown on the left in FIG. The shape to be represented is smaller than the original image and becomes an ellipse. The information display device 1 analyzes the image data obtained as shown in the left of FIG. 2B, and the cursor image displayed as shown in the right of FIG. 2B is as shown in the left of FIG. 2B. Analyzing the position to be photographed after deformation.

  FIG. 3 is a flowchart showing operations of the operation terminal 2 and the information display device 1 when the cursor image shown in FIG. As shown in the figure, first, image data to be displayed on the first display screen 11 is generated and displayed on the main body of the information display device 1 (S201). The operation terminal 2 captures the image displayed on the first display screen 11 (S202), and transmits the captured image data to the information display device 1 (S203). The information display device 1 that has received the image data from the operation terminal 2 analyzes the received image data, calculates the image ID and position coordinates of the cursor imaged on the operation terminal 2, and stores them in the cursor coordinate storage unit 105. Let Then, the position coordinates on the first display screen 11 currently pointed to by the operation terminal 2 are estimated from the position orientation of the operation terminal 2 (S204), and are displayed on the first display screen 11 in the next frame. The position of the power cursor is determined, and the cursor corresponding to the next frame is displayed at the determined position (S205). The information display device 1 updates the image ID of the cursor to the image ID corresponding to the newly displayed cursor (S206). The operation terminal 2 displays the image of the first display screen 11 taken in step S202 on the second display screen 23 (S207).

  FIG. 1C is a diagram illustrating an example of a bitmap image displayed on the second display screen 23 when the application to be executed is an augmented reality user interface. FIG. 1C superimposes a three-dimensional CG image related to the content of the first display screen 11 included in the camera image on the background, with the camera image taken by the image pickup unit 21 as the background. FIG. This CG image can be displayed as if a three-dimensional object actually exists in the space in front of the display by using an augmented reality display technology (for example, Non-Patent Document 2). Compared with the CG image displayed on the first display screen 11, it is possible to intuitively understand a stereoscopic effect, a real scale feeling, and the like.

  FIG. 4 is a flowchart showing operations of the operation terminal 2 and the information display device 1 when executing the application of the augmented reality user interface shown in FIG. As shown in the figure, first, image data to be displayed on the first display screen 11 is generated in the main body of the information display device 1, and an image 320 is displayed (S301). The operation terminal 2 captures a part of the image displayed on the first display screen 11 (region 321 surrounded by a broken line) (S302). The captured image of the area 321 is displayed as a display image 322 on the second display screen 23. The operation terminal 2 transmits the captured image data to the information display device 1 (S303). Here, when the user operates the user input unit 22 or the like of the operation terminal 2, the operation terminal 2 accepts and accepts input data that is an instruction meaning to click a button or the like around the cursor image displayed. The input data is transmitted to the information display device 1 (S304). The information display device 1 that has received the image data and the input data from the operation terminal 2 analyzes the position of the operation terminal 2 from the received image data (S305) and displays it on the second display screen 23. A background image 323 for 2 and a three-dimensional image (MR) 324 to be displayed on the second display screen 23 using an augmented reality user interface specified by input data are generated and synthesized (S306). . The information display device 1 transmits the combined MR composite image for the operation terminal 2 to the operation terminal 2 (S307). After transmitting the MR composite image to the operation terminal 2, the information display device 1 generates an image to be displayed on the first display screen 11 as a background image in the next frame, and further displays it on the generated background image. Determine the position of the cursor to be done. Then, the cursor image corresponding to the next frame is synthesized and displayed at the determined position on the background image (S308). The information display device 1 updates the image ID of the cursor to the image ID corresponding to the newly displayed cursor (S309). When the MR composite image is received from the information display device 1 in step S307, the operation terminal 2 displays the image 325 in which the MR received in step S307 is superimposed on the background image and displayed on the second display screen 23. (S310).

  Hereinafter, the configuration of the information display device 1 and the operation terminal 2 will be described in detail.

(Appearance of information display device 1)
In FIG. 1A, the information display device 1 has a first display screen 11. Specifically, the first display screen 11 may be a display screen (display) of a digital television. Specifically, the digital television may be a plasma display, a liquid crystal display, or an organic EL display. The information display device 1 may be a personal computer. In this case, the display and the computer may be integrated, or may be separated and connected by a communication means such as a cable or wireless. The information display device 1 targeted by the present invention requires at least one display screen, but other hardware configurations can be applied to general-purpose configurations.

(External appearance of operation terminal 2)
In FIG. 1A, an operation terminal 2 is a remote controller including an image capturing unit 21, a user input unit 22, and a second display screen 23. The information display device 1 communicates with the information display device 1 by wireless communication such as IEEE801.11n. Information can be transmitted and received using a wired communication means such as a means or a LAN cable. Details will be described below.

  The image capturing unit 21 is a camera capable of capturing color image data. Specifically, the image capturing unit 21 may be an image sensor such as a CCD or a CMOS provided with a filter and a lens for acquiring a color image. In this embodiment, image information of a color image is used, but a black and white image may be used. The image pickup unit 21 that can be used in the present invention can correspond to any format as long as it is an image sensor that can take an image displayed on the display that is the first display screen 11.

  The user input unit 22 is input means for inputting user instructions. Specifically, a touch pad, a joystick, or the like that can input two-dimensional coordinates, or a button that can be turned ON / OFF can be combined. Further, a touch panel device integrated with the second display screen 23 may be included. The input from the user input unit 22 is not only used for event processing at the operation terminal 2 but also to the application processing unit 14 of the information display device 1 via the data transmission unit 24 and the data reception unit 13. It is transmitted and used as an event input such as an application selection operation.

  The second display screen 23 is a display screen attached to the operation terminal 2 and can display a GUI (Graphical User Interface) image generated by the second image generation unit 108 of the user interface device 10. In particular, when the application is an augmented reality user interface as shown in FIG. 1C, related information and CG are superimposed and displayed on the camera image captured by the image capturing unit 21 of the operation terminal 2. To do. As a result, the user can superimpose information and CG related to the display content on the display as if looking at the display (first display screen 11) through the second display screen 23 of the operation terminal 2. Can be seen. These technologies are called video see-through augmented reality technologies.

(Configuration of information display device 1 and operation terminal 2)
FIG. 5 is a block diagram illustrating an example of a configuration of main parts of the information display device 1 and the operation terminal 2 according to Embodiment 1 of the present invention. The information display device 1 includes a user interface device 10, a first display screen 11, a data transmission unit 12, a data reception unit 13, and an application processing unit 14. The configurations included in the external view of FIG. 1 are numbered the same as the numbers in FIG. 1, and since these configurations are the same, description thereof is omitted.

  The operation terminal 2 includes an image capturing unit 21, a user input unit 22, a second display screen 23, a data transmission unit 24, a data reception unit 25, and a screen display unit 26. Since the image capturing unit 21, the user input unit 22, and the second display screen 23 have already been described with reference to the external view of FIG. 1, the same reference numerals are assigned and description thereof is omitted here.

  The data transmission unit 24 and the data reception unit 25 are the same as the data transmission unit 12 and the data reception unit 13, and specifically, wireless communication means such as IEEE801.11n or wired communication means such as a LAN cable. A processing unit may be used.

  The screen display unit 26 is a processing unit for displaying the bitmap data of the screen transmitted via the data receiving unit 25 on the second display screen 23. However, as with the cursor display unit 107, when the operation terminal 2 uses a window system such as Microsoft Windows Vista (registered trademark), the bitmap data is not drawn on the screen but actually drawn. Processes until the system is requested to draw.

  Hereinafter, the user interface device 10 will be described in detail.

  The user interface device 10 includes a cursor recognition unit 101, a position / orientation estimation unit 102, a first image generation unit 103, an image ID management unit 104, a cursor coordinate storage unit 105, a cursor data storage unit 106, a cursor display unit 107, and a second image. A generation unit 108 is provided.

  The cursor recognition unit 101 recognizes the image of the cursor image included in the captured image and recognizes the image ID of the captured cursor image based on the storage content of the image ID storage unit. The color image data captured by the image capturing unit 21 is received via the data receiving unit 13 and the cursor area information included in the image is acquired. Specifically, the input color image data may be in the RGB format. The RGB format represents a color with three components of R (red), G (green), and B (blue). First, the input color image data is converted into the HSV format. The HSV format is a color expressed by three components of H (hue), S (saturation), and V (lightness), and an example of a conversion formula from RGB format to HSV is shown in FIG. In FIG. 6, R, G, and B are values from 0 to 1.0, H is from 0 to 360.0, and S and V are from 0 to 1.0.

  Next, the cursor recognizing unit 101 cuts out (extracts) the cursor area by a labeling process. Here, the labeling process is a process of grouping specific color areas and assigning numbers as identifiers. Specifically, a binarized image may be scanned in units of pixels, a connection relationship between adjacent pixels may be obtained, and labels may be assigned. For example, assume that the cursor image shown in FIG. 7A is composed of three colors: green in FIG. 7B, yellow-green in FIG. 7C, and red in FIG. 7D. On the other hand, the areas cut out by the labeling process are two areas, a green area in FIG. 7B and a yellow-green area in FIG. At this time, the binarization process in the labeling process may generate an image in which the area including two of the green area and the yellow-green area is set to 1 and the other areas are set to 0. As a method for discriminating between the green area and the yellow-green area, the HSV format color data may be discriminated by a conditional expression such as “H (hue) is 70 or more and 135 or less”. In the HSV format, when H is 120, the color is green, and when H is 85, the color is yellow-green. However, the image captured by the camera is exactly the same as the color displayed on the display due to the influence of illumination and the atmosphere. The color cannot always be acquired. For this reason, when the value of H (hue) is within a certain range, it is determined that the color is a predetermined color. In this way, by using H (hue) for discrimination, it is possible to obtain a recognition result that is less likely to be adversely affected by changes in room brightness. The two colors, green and yellow-green, are difficult to distinguish by human eyes, but have the feature that they can be easily identified quantitatively by comparing the H (hue) components of images taken with a camera. .

  Specifically, when a color image including a cursor as shown in FIG. 8A is taken, four label areas as shown in FIG. 8B can be acquired. Further, as shown in FIG. 9A, the position (representative coordinates) and size (area) data of each label can be acquired. Specifically, the center coordinates of the area may be used as the representative coordinates, or feature points on the outline of the area may be used. Here, the cursor image in FIG. 8A includes three triangles (label 0, label 1, label 2) having the same area as shown in FIG. The base triangle label 3) is composed of a total of four triangles. Therefore, by selecting four regions that are candidates for the cursor image from the labeling results and selecting a region having the maximum area from the four regions, the base triangle included in the cursor can be determined. . After that, by identifying the labels in the order of label 3 → label 1 → label 0 → label 2 from the positional relationship of the four areas from label 0 to label 3, and obtaining the representative coordinates of each, It is possible to acquire the four-vertex data that constitutes it.

  Further, the HSV values of the representative colors are compared from the labeling result of FIG. 9A, and the image ID is determined based on the correspondence information between the color value and the image ID as shown in FIG. The correspondence data in FIG. 10 may be stored in the cursor data storage unit 106. Through the above processing, the four vertex data constituting the cursor and the image ID can be acquired, and the cursor data shown in FIG. 9B can be generated. However, the cursor data generated at this stage is cursor candidate data, and a plurality of cursor data may be generated including erroneous cursor data. In this case, the correct cursor data may be selected from a plurality of candidates when the position / direction estimation unit 102 estimates the position / direction of the cursor.

  Note that the method for discriminating periodically changing image IDs is not limited to the method described above. For example, in the case of the arrow-type cursor display shown in FIGS. 11A, 11B, and 11C, an image that differs from three cursor images due to a difference in hue value of a specific area constituting the arrow. ID can be acquired. In this case, it can be realized by linearly approximating the outline of the cursor, extracting a feature point as shown in FIG. 12, and acquiring a color value of a determined area based on the coordinates of the feature point. Even in the case of the same arrow-shaped cursor, as shown in FIG. 11D, a barcode-like ID may be added separately from the cursor shape. Furthermore, as shown in FIG. 11 (e), the ID may be expressed by coding the shape of the cursor locus display portion. As described above, the cursor display according to the present invention may have any design as long as at least several types of IDs can be identified using color and shape information as clues. The present invention is not limited to cursor design.

  FIG. 13A is a diagram illustrating a change in the position of the cursor for each frame. FIG. 13B is a diagram illustrating an example in which image IDs are sequentially incremented and assigned to the frames illustrated in FIG. FIG. 13C is a diagram illustrating an example in which the same image ID is assigned to a frame in which the cursor does not move with respect to each frame illustrated in FIG. For example, as shown in FIG. 13A, the cursor is not moved between frame number 23 and frame number 24. In the above example, as shown in FIG. 13B, the image ID of the cursor is simply incremented and assigned for each frame. However, if the cursor does not move as shown in FIG. The same image ID may be used without incrementing the ID. When assigning an image ID as shown in FIG. 13C, it is necessary to determine whether or not the cursor has moved between frames. However, even when the frame rate is high, the cursor moves little. There is an effect that it is possible to cope without increasing the pattern of the image ID.

  Next, the position / orientation estimation unit 102 reads: “From the recognition result by the cursor recognition unit and the position coordinates stored in the cursor coordinate storage unit, the display image captured by the image capturing apparatus; The position of the display in the coordinate system centered on the camera based on the result of recognizing the cursor from the result of image recognition of the cursor. The direction is estimated, and further, the designated coordinates on the display indicated by the operation terminal are calculated. This will be described in detail below.

  First, the coordinate values of the four vertices constituting the cursor shown in FIG. 9B and the image ID of the cursor are acquired from the cursor recognition unit 101. Note that the number of the four vertices may be four or more. For example, if the cursor image has an arrow shape shown in FIG. 11, it is possible to obtain the seven vertices shown in FIG. 12 by approximating the outline of the cursor to a straight line. A method for generating a 4 × 4 coordinate transformation matrix indicating a three-dimensional position and orientation from a quadrilateral with known coordinate values (or vertex data of four or more vertices) is specifically described in Non-Patent Document 2. You can use the method that is used. This method is a general calculation method in the field of augmented reality research, and the coordinate transformation matrix indicated by the image marker can be determined using the approximate calculation of the least square method. However, since the approximation error of the least squares method is not always 0, any approximation error that is equal to or greater than a threshold value may be treated as a position direction estimation failure. In addition, when there are a plurality of cursor candidates, data having a minimum approximation error may be selected as the cursor data. With the above calculation, a 4 × 4 coordinate transformation matrix M_c indicating the position and orientation of the cursor image in the coordinate system with the camera as the origin can be generated.

  Next, the coordinate value P1 (P1x, P1y) of the cursor on the display is acquired from the cursor coordinate storage unit 105 based on the cursor image ID. By converting this coordinate value into a relative position from the center of the display, a 4 × 4 coordinate conversion matrix M_d for converting a coordinate system having the cursor as the origin into a coordinate system having the center of the display as the origin. Can be generated. As a result, by multiplying the two matrices M_c and M_d, it is possible to generate a 4 × 4 coordinate transformation matrix M_cd that indicates the position and orientation of the display with the center of the camera image as the origin. A plane that passes through the origin of the display (that is, the parallel movement component T (Tx, Ty, Tz) of M_cd) and is normal to the vector Rz (Rz_x, Rz_y, Rz_z) of the Z-axis direction component of M_cd; From the intersection of the straight line x = y = 0 indicating the center point of the camera image, the cursor pointing coordinates P2 (P2x, P2y) can be calculated.

  Finally, a distance D between the display and the operation terminal is generated from the parallel movement component T (Tx, Ty, Tz) of the 4 × 4 coordinate transformation matrix M_cd indicating the position and orientation of the display. Specifically, D = √ (Tx * Tx + Ty * Ty + Tz * Tz) may be used. However, √ (A) represents the square root of A. In addition, since the absolute value has no meaning for this distance, any equation may be used as long as the value increases in proportion to the distance.

  As a result of the above calculation, the position / orientation estimation unit 102 uses the 4 × 4 coordinate transformation matrix Mcd indicating the position / direction of the display and the designated coordinates (pointing coordinates indicated by the operation terminal 2) P2 (P2x, P2y). ), The distance D between the display and the operation terminal can be determined.

  The first image generation unit 103 is based on “image generation means for generating a display image to be displayed on the display screen” and “position / orientation information indicating the relative position of the image capturing device from the estimated position coordinates”. The image generation means for generating an augmented reality image to be displayed on the second display screen by synthesizing an image captured by the camera and an image generated by CG, and the image capturing from the estimated position coordinates The image generation for generating a part of the display image displayed on the second display screen based on the position / orientation information indicating the relative position of the device and enlarging and generating the part of the display image that has been cut out Means ”or“ the image generation means for generating an image to be displayed on the web browser ”, and renders a screen to be displayed on the first display screen 11 in accordance with the application processing unit 14. It is intended. For example, when the target application is a web browser, a screen as displayed on the first display screen 11 in FIG. At this time, not only the user input from the user input unit 22 but also the designated coordinates P2 on the display generated by the position / orientation estimation unit 102 are used as an input interface, thereby controlling the application with an intuitive input means. The screen generated as a result can be displayed on the first display screen 11.

  The image ID management unit 104 is an example of “an image ID storage unit that stores in advance a plurality of types of cursor images and image IDs for specifying display images on which the cursor images are displayed” in association with each other. Yes, it manages the ID of the cursor image to be displayed in the next frame. In the present invention, since the display and the camera are not synchronized, it is impossible to determine how many frames before the cursor image recognized by the camera is the display request. For this reason, the image ID is for storing a predetermined number of frames of the position coordinates of the cursor displayed in the past. Here, the fixed number of frames is displayed on the first display screen 11 after the cursor display unit 107 issues a cursor display request, and further the camera image captured by the image capturing unit 21 is displayed on the user interface device. What is necessary is just to determine with the maximum elapsed time taken until 10 recognizes. For example, the maximum time from the cursor display request until it is displayed on the display is 0.1 second, and the image displayed on the display is captured by the image capturing unit 21 and the user interface device 10 recognizes the image. If the maximum time is 0.1 seconds, 0.2 seconds is the maximum time. At this time, if the screen is updated up to 30 frames per second, the cursor coordinates up to 6 frames before may be stored. In this case, the image ID can be expressed by six integers of 0 to 5, and the image ID management unit 104 has 0, 1, 2, 3, 4, 5, 0, 1, 2, 3, 4, The ID of the cursor to be displayed in the next frame is determined by periodically incrementing the ID, such as 5, 0, 1,.

  The cursor coordinate storage unit 105 calculates “position coordinates on the display screen of the captured cursor image, and stores the calculated position coordinates in association with the specified image ID. The coordinate value and the size of the cursor displayed on the cursor display unit 107 are stored in association with the image ID of the cursor managed by the image ID management unit 104. The cursor display unit 107 is an example of “the cursor display unit that changes the size of the cursor image displayed on the display screen generated by the image generation unit based on the estimated positional relationship”. For example, when the first display device 11 is a display having a resolution of 1920 × 1080, the coordinate value can be expressed as a two-dimensional coordinate in which the upper left pixel of the display is (0, 0) and the lower right is (1919, 1079). . The size of the cursor may be a representation of the displayed scaling associated with the number of pixels. Specifically, the saved cursor coordinates may be saved as data corresponding to the image ID as shown in FIG.

  The cursor data storage unit 106 is a storage unit for storing the image data of the cursor in association with the image ID. Specifically, as shown in FIG. 10, the bitmap data itself of the cursor image corresponding to the image ID is stored. Alternatively, it may be a parameter table for generating bitmap data from an image ID.

  The cursor display unit 107 is a “cursor display unit that synthesizes a plurality of types of one cursor image that repeatedly changes at a predetermined time period on the display image by the image generation unit”, and a “new image ID corresponding to a new image ID”. The cursor that causes the image generation unit to synthesize the cursor image so that a new cursor image is read from the image ID storage unit and the new cursor image is displayed at the designated coordinates estimated by the position estimation unit. Based on the position coordinates of the frame estimated by the display means or the position estimation means, the new cursor image is positioned at the center of the new display image displayed on the second display means. It is an example of the “cursor display unit” that generates a new display image and displays it on the second display unit. Further, the cursor display unit 107 is “the cursor display unit that changes the hue component of the specific area included in the cursor image according to the change of the display image generated by the image generation unit according to the image ID”. , "The cursor display means for changing the shape of the cursor image in association with the image ID", "the cursor display means for changing the locus portion of the cursor image in association with the image ID" or "the second display It is an example of the “cursor display unit” that changes the size of the cursor image displayed on the display image generated by the image generation unit based on whether or not an augmented reality image is displayed on the screen. The cursor display unit 107 displays a cursor at designated coordinates indicated by the operation terminal 2. Hereinafter, the processing of the cursor display unit 107 will be described in detail.

  First, from the position / orientation estimation unit 102, the designated coordinates P2 (P2x, P2y) on the display and the distance D between the operation terminal and the display are received. Next, the cursor position C (Cx, Cy) on the screen is determined from the designated coordinates P2 (P2x, P2y), and the cursor size Sc is determined from the distance D. Specifically, the cursor position may be C (Cx, Cy) = P2 (P2x, P2y). Further, the cursor size Sc may be obtained by the equation Sc = k1 * D + k2 using predetermined constants k1 and k2. As a result, when the user leaves the display, the cursor is displayed in a large size, so that it is possible to improve the visibility of the cursor display seen by the user, and the cursor can be easily recognized. Further, the cursor size Sc may be changed according to the type of application executed by the application processing unit 14. For example, when Sc1 is used for displaying the pointing cursor, the scale factor may be adjusted to be 2 * Sc1 when using the augmented reality user interface. In the present invention, the recognition error on the camera image can be adjusted in units of pixels by adjusting the size of the cursor, and the recognition accuracy required by the application can be provided. When only low accuracy is required, reducing the size of the cursor has the effect of making the appearance of the cursor on the display inconspicuous.

  Next, regarding the cursor coordinates and the size of the cursor, the cursor is compared with the value displayed in the previous frame, and if it is the same as the previous frame, the cursor display is not changed. At this time, when the difference from the previous frame is within a certain range, it may be determined that the value is the same and the value of the previous frame may be used. As a result, the shaking of the pointing due to camera shake or the like can be suppressed, and the cursor display can be stabilized. If it is determined that it is different from the previous frame, the image ID of the cursor image of the next frame is acquired from the image ID management unit 104, and the cursor image corresponding to the image ID is received from the cursor data storage unit 106 (FIG. 13 ( In the case of c)). Next, the cursor coordinate storage unit 105 is requested to update the cursor coordinates C (Cx, Cy) corresponding to the image ID. For example, in the state of FIG. 14A stored in the cursor coordinate storage unit 105, the image ID is 2, the cursor size estimated by the position / orientation estimation unit 102 is 110, and the cursor coordinates (231, 210). ), The cursor coordinate data stored in the cursor coordinate storage unit 105 is updated as shown in FIG. Finally, the cursor display unit 107 draws a bitmap of a cursor image having an image ID of 2 on the screen so that the cursor size is 110 and the image center of the cursor is (231, 210). However, when the user interface device 10 uses a window system such as Windows Vista (registered trademark), the system does not actually draw the cursor but draws a bitmap including the cursor on the screen. Process until requested. For this reason, the user interface device 10 does not manage when the cursor is displayed on the screen. Processing may be performed until the cursor is directly drawn on a bitmap (frame buffer) displayed on the screen without going through the window system.

  The second image generation unit 108 generates an image (bitmap data) to be displayed on the second display screen 23. Specifically, if the application is a screen enlargement display application, an image obtained by acquiring the designated coordinates P2 on the display and cutting out a bitmap in the vicinity of the designated coordinates on the application screen displayed by the application processing unit 14 ( For example, FIG. 1B is generated. This image data is finally displayed on the second display screen 23 via the data transmission unit 12, the data reception unit 25, and the screen display unit 26. Further, when the application is an augmented reality user interface, a 4 × 4 coordinate transformation matrix M_cd indicating the position and orientation of the display is received from the position and orientation estimation unit 102, and the camera is determined based on the position and orientation indicated by the coordinate transformation matrix. By superimposing and displaying information related to the contents of the display displayed on the image with CG, a composite image as shown in FIG. 1C can be generated. The composite image data is finally displayed on the second display screen 23 via the data transmission unit 12, the data reception unit 25, and the screen display unit 26.

  The above is the description of the user interface device 10.

  Next, the data transmission unit 12 and the data reception unit 13 may be specifically processing units of wireless communication means such as IEEE801.11n or wired communication means such as a LAN cable. Any communication means can be applied to the present invention.

  The application processing unit 14 processes an application that operates on the information display device 1, and a specific application may be a web browser, a 3D CAD application, or the like. The drawing process requested by the application is sent to the second image generation unit 108 for processing. The present invention can be applied to any application that displays information on a screen.

(Explanation of processing flow)
Regarding the user interface device 10 configured as described above, the flow of processing will be described using a flowchart.

  The flowchart in FIG. 15 is a process from estimating the position and orientation of the display from the image information acquired by the image capturing unit 21 of the operation terminal 2 until displaying the cursor at the position indicated by the operation terminal 2. The process in FIG. 15 is a process called (executed) at regular time intervals while the user operates the operation terminal 2. This fixed time may be 1/30 second or 1/60 second. Further, the fixed time does not need to be an accurately determined time, and the processing of FIG. 15 may be repeated continuously. Whether or not the user is operating the operation terminal 2 may be determined by the user pressing the button to instruct the start of operation, or using an acceleration sensor or a gyro sensor, and the user holding the terminal. You may judge by whether it exists.

  First, a color image (for example, FIG. 8A) is captured by the image capturing unit 21 of the operation terminal 2, and is sent to the cursor recognition unit 101 via the data transmission unit 24 and the data reception unit 13 (step S101). ). The cursor recognition unit 101 performs a labeling process (step S102) using a predetermined color value (hue value) as a threshold, and assigns a label to the cursor area. The labeling result data is as shown in FIG. 9A. As a result, the cursor recognizing unit 101 generates candidate cursor data as shown in FIG. 9B. At this time, if there is no cursor candidate or if the four-vertex data and the image ID are not correctly recognized, the process ends as if the cursor could not be recognized (step S103). Next, if there are one or more cursor candidates, the position / orientation estimation unit 102 performs a coordinate transformation matrix Mcd indicating the position / orientation of the display and designated coordinates P2 (P2x, P2y) on the display for each candidate. ), A distance D between the operation terminal and the display is generated (step S104, step S105, and step 106, respectively). At this time, if the least square method recognition error used at the time of estimation is greater than a threshold value with a recognition error, the process is terminated, assuming that it has not been correctly estimated. Next, when a plurality of cursors are recognized, a cursor candidate having the smallest recognition error may be selected.

  Next, the cursor display unit 107 receives the instruction coordinates P2 (P2x, P2y) on the display and the distance D between the operation terminal and the display from the position / orientation estimation unit 102, and the cursor coordinates C (Cx of the next frame) , Cy) and the cursor size Sc are calculated (step S107).

  As a result, it is determined whether or not the position coordinate or the size of the cursor changes from the previous frame (step S108), and only when it has changed, the image ID management unit 104 is requested to update the image ID ( In step S109, the image ID of the next frame is received, and in accordance with the image ID, data update in the cursor coordinate storage unit 105 (step S110) and cursor display processing in the cursor display unit 107 are performed (step S111). . Here, the cursor display unit 107 indicates that “the cursor display unit changes the cursor image and the image ID only when the position coordinates of the cursor image to be displayed are moved on the display screen generated by the image generation unit. Is an example.

  Furthermore, according to the position / orientation of the cursor determined by the above processing, the first image generation unit 103 generates a screen according to the contents of the application. For example, when the cursor is moved over the button, the color of the button is changed, or when the cursor is moved over a specific button, a selection operation event is transmitted from the user input unit 22 to the application processing unit 14. Then, a separate window is activated to perform processing such as updating the display of the application. In the case of augmented reality type user interface display as shown in FIG. 1C, a CG image is synthesized on the camera image based on the coordinate transformation matrix generated in step S104. It is displayed on the display screen 23. However, these processes may operate independently of the process flow of FIG.

  This completes the description of the processing flow of the user interface device shown in FIG.

  As described above, by specifying the image ID of the cursor image controlled to be displayed in the center of the second display screen 23, the cursor image is displayed on the first display screen 11 how many frames ago. Since it can be known whether the position has been moved from the displayed position, a new cursor image can be displayed at a position that accurately follows the movement of the operation terminal 2.

  FIG. 1B is a diagram illustrating an example of a bitmap image displayed on the second display screen 23 when the application to be executed is an enlarged display application. The application screen shown in FIG. 1B is not a camera image taken by the operation terminal 2 but a part of the bitmap displayed on the first display screen 11 that is cut out and enlarged. By using this enlarged display application, the user can enlarge and display the content of the screen pointed to by the operation terminal 2. The processing procedure for executing such an enlarged display application is as follows, corresponding to the flowchart of FIG. That is, in step S306, instead of synthesizing the background image and the MR image, the second image generation unit of the information display device 1 generates an enlarged image of the background image, and in step S307, the generated enlarged image is sent to the operation terminal 2. Send. In step S <b> 310, the data reception unit 25 of the operation terminal 2 receives the enlarged image for display from the data transmission unit 12 of the information display device 1 and displays the received enlarged image on the second display screen 23.

(Embodiment 2)
FIG. 16 is a block diagram illustrating an example of a configuration of main parts of the information display device 1601 and the operation terminal 1602 according to the second embodiment of the present invention. Although the external view is the same as FIG. 1, in the second embodiment, the operation terminal 1602 is a mobile phone terminal and is an information device independent of the information display device 1601. In FIG. 16, the same components as those in FIGS. 1 and 5 are denoted by the same reference numerals, and the description thereof is omitted.

  Here, the difference between the configuration of the second embodiment shown in FIG. 16 and the configuration of the first embodiment shown in FIG. 5 is that the user interface device 10 is in the operation terminal 1602 or an information display device. Whether it is within 1 or not. Specifically, in the second embodiment, an external display of a mobile phone terminal may be used as the first display screen 11. In this case, the content displayed on the first display screen 11 is generated by the first image generation unit 103 of the user interface device 10 in the mobile phone terminal.

  The information display device 1601 includes a first display screen 11, a data receiving unit 13, a display mode management unit 15, and a screen display unit 26. Unlike the first embodiment, the screen display unit 26 receives image data sent via the data receiving unit 13 and displays an image on the first display screen 11. At this time, the display mode management unit 15 manages the display mode of the screen, and only when the external video input is a mode in which image information from the operation terminal 1602 is received, the data received by the data receiving unit 13 is input to the screen display unit 26. Control to send. Specifically, this mode may be changed by an instruction from the user, or may be changed using a mode switching request from the operation terminal 1602 via the data receiving unit 13.

  Next, the operation terminal 1602 includes a user interface device 10, an application processing unit 14, an image capturing unit 21, a user input unit 22, a second display screen 23, and a data transmission unit 24. Therefore, unlike the first embodiment, user input data and color image data input or captured by the operation terminal 1602 are sent directly to the user interface device 10 in the operation terminal 1602.

  Since other configurations are the same as those of the first embodiment, description thereof is omitted.

  In the second embodiment, the information display device 1601 and the operation terminal 1602 are completely different devices. In such a case, the first display screen 11 of the information display device 1601, the image capturing unit 21 of the operation terminal 1602, and the like. However, it is very difficult to operate with timing synchronization. In the present invention, it is possible to estimate an accurate position and orientation even when it is difficult to synchronize display display and camera shooting.

(Embodiment 3)
FIG. 17 is a block diagram showing an example of the configuration of the information display device 1701 and the operation terminal 1702 according to Embodiment 3 of the present invention. The information display device 1701 and the operation terminal 1702 according to the third embodiment do not include a user interface device in either device, but generate an image to be displayed on each display screen by each image generation unit. This is a difference from the first and second embodiments.

  Specifically, in the operation terminal 1702, as compared with the operation terminal 2 shown in FIG. 5, the screen display unit 26 is changed to the second image generation unit 108, and the cursor recognition unit 101 is newly added. . That is, in the operation terminal 1702 of the third embodiment, unlike the screen display unit 26, the image data received from the second image generation unit in the user interface device 10 is not displayed on the second display screen 23 as it is. The second image generation unit 108 displays on the second display screen 23 from the captured image captured by the image capturing unit 21, the cursor image received from the information display device 1701, the image ID, the position and orientation of the operation terminal 1702, and the like. A display image to be generated is generated. The operation terminal 1702 further includes a cursor recognition unit 101, and the cursor recognition unit 101 has a function of identifying an image ID of a cursor image by image recognition from a cursor image captured by the image imaging unit 21. The recognition result recognized by the cursor recognition unit 101 is given to the application processing unit 14 and the position / orientation estimation unit 102 via the data transmission unit 24 of the operation terminal 1702 and the data reception unit 13 of the information display device 1701.

  Correspondingly, in the information display device 1701, the cursor recognition unit 101 and the second image generation unit 108 are eliminated from the user interface device 10 shown in FIG. With such a configuration, there is an effect that the processing load of the user interface device 10 in the information display device 1 is reduced. Accordingly, the operation terminal 1702 needs to be not a simple remote controller but a mobile phone or the like having functions of cursor recognition and generating a display image to be displayed on the second display screen.

  Each functional block in the block diagrams (FIGS. 5, 16, 17 and the like) is typically realized as an LSI which is an integrated circuit. These may be individually made into one chip, or may be made into one chip so as to include a part or all of them. For example, the functional blocks other than the memory may be integrated into one chip.

  The name used here is LSI, but it may also be called IC, system LSI, super LSI, or ultra LSI depending on the degree of integration.

  Further, the method of circuit integration is not limited to LSI's, and implementation using dedicated circuitry or general purpose processors is also possible. An FPGA (Field Programmable Gate Array) that can be programmed after manufacturing the LSI or a reconfigurable processor that can reconfigure the connection and setting of circuit cells inside the LSI may be used.

  Further, if integrated circuit technology comes out to replace LSI's as a result of the advancement of semiconductor technology or a derivative other technology, it is naturally also possible to carry out function block integration using this technology. Biotechnology can be applied.

  In addition, among the functional blocks, only the means for storing the data to be encoded or decoded may be configured separately instead of being integrated into one chip.

  The user interface device according to the present invention is useful as a user interface for operating / browsing content and information displayed on a display such as a digital television device or a personal computer. Further, the present invention can be applied to a user interface when an external display is output from a mobile phone terminal, a user interface of a game that is played by linking a stationary game machine and a portable game, and the like.

It is an external view of the operation terminal and information display apparatus in Embodiment 1 of this invention. The cursor image displayed on the first display screen 11 is photographed by the operation terminal 2 and the photographed cursor image is analyzed to display the cursor image to be displayed on the first display screen 11 in the next frame. It is a figure explaining the mechanism which determines a position. It is a flowchart which shows operation | movement of the operating terminal 2 and the information display apparatus 1 when performing the display of the cursor image shown to Fig.2 (a). It is a flowchart which shows operation | movement of the operating terminal 2 and the information display apparatus 1 in the case of performing the application of the augmented reality type user interface shown in FIG.1 (c). It is a block diagram which shows an example of a structure of the principal part of the operating terminal and information display apparatus in Embodiment 1 of this invention. It is the figure which showed the formula for converting the image of RGB format in Embodiment 1 of this invention into the image of HSV format. It is a conceptual diagram for demonstrating an example of the cursor image in Embodiment 1 of this invention, and its color structure. It is a conceptual diagram for demonstrating the color image data and labeling process in Embodiment 1 of this invention. It is a conceptual diagram for demonstrating the labeling result data in Embodiment 1 of this invention, and the candidate data of a cursor. It is a conceptual diagram for demonstrating the correspondence of image ID and cursor image in Embodiment 1 of this invention. It is a conceptual diagram for demonstrating an example of the arrow type cursor image in Embodiment 1 of this invention. It is a conceptual diagram for demonstrating the feature point acquirable from the arrow type cursor image in Embodiment 1 of this invention. It is a conceptual diagram for demonstrating the relationship between the position change of the cursor image and image ID in Embodiment 1 of this invention. It is a conceptual diagram for demonstrating the flow of the data processing memorize | stored in the cursor coordinate memory | storage part in Embodiment 1 of this invention. It is a flowchart for demonstrating the flow of the process in Embodiment 1 of this invention. It is a block diagram which shows an example of a structure of the principal part of the operating terminal and information display apparatus in Embodiment 2 of this invention. It is a block diagram which shows an example of a structure of the principal part of the operating terminal and information display apparatus in Embodiment 3 of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1,1601,1701 Information display apparatus 2,1602,1702 Operation terminal 10 User interface apparatus 11 1st display screen 12 Data transmission part 13 Data reception part 14 Application processing part 15 Display mode management part 21 Image imaging part 22 User input part 23 Second display screen 24 Data transmission unit 25 Data reception unit 26 Screen display unit 101 Cursor recognition unit 102 Position orientation estimation unit 103 Image generation unit 104 Image ID management unit 105 Cursor coordinate storage unit 106 Cursor data storage unit 107 Cursor display unit 108 Second image generation unit

Claims (17)

A user interface device that displays a new image including the cursor image based on an image recognition result of a captured image obtained by the image capturing device capturing a cursor image displayed on a display screen,
Image generating means for generating a display image to be displayed on the display screen;
Cursor display means for synthesizing a plurality of types of one cursor image that repeatedly changes at a predetermined time period on the display image by the image generation means;
Image ID storage means for storing a plurality of types of cursor images and image IDs for specifying display images on which the cursor images are displayed in association with each other;
Cursor recognition means for recognizing the cursor image included in the captured image and identifying the image ID of the captured cursor image from the recognized cursor image based on the stored contents of the image ID storage means; ,
Cursor coordinate storage means for calculating a position coordinate on the display screen of the captured cursor image, and storing the calculated position coordinate in association with the specified image ID;
The display image picked up by the image pickup device and the image pickup device on the display image currently indicate from the recognition result by the cursor recognition means and the position coordinates stored in the cursor coordinate storage means. Position estimation means for estimating the indicated coordinates indicating the current position,
The cursor display means reads out a new cursor image corresponding to a new image ID from the image ID storage means, and displays the new cursor image at the indicated coordinates estimated by the position estimation means. A user interface device characterized by causing the image generation means to synthesize the cursor image. The image capturing apparatus includes a second display unit that displays the captured image.
The cursor display means is arranged so that the new cursor image is positioned at the center of the new display image displayed on the second display means based on the position coordinates of the frame estimated by the position estimation means. The user interface device according to claim 1, wherein a new display image is generated and displayed on the second display unit. The cursor display means changes the cursor image and the image ID only when the position coordinates of the cursor image to be displayed are moved on the display screen generated by the image generation means. A user interface device as described. The cursor display means changes the size of the cursor image displayed on the display screen generated by the image generation means based on the estimated positional relationship,
The user interface device according to claim 2, wherein the cursor coordinate storage unit stores a position coordinate and a size of the displayed cursor image. The said cursor display means changes the hue component of the specific area | region contained in the said cursor image according to the change of the display image produced | generated by the said image production | generation means according to the said image ID. 2. The user interface device according to 2. The user interface device according to claim 2, wherein the cursor display unit changes the shape of the cursor image in association with the image ID. The user interface device according to claim 2, wherein the cursor display unit changes a locus portion of the cursor image in association with the image ID. The image capturing apparatus includes a second display unit that displays the captured image.
The image generation unit combines the image captured by the image capturing device and the image generated by CG based on the estimated position coordinates based on the position and orientation information representing the relative position of the image capturing device. The user interface device according to claim 2, wherein an augmented reality image to be displayed on the second display screen is generated. The cursor display means changes the size of the cursor image displayed on the display image generated by the image generation means based on whether or not the augmented reality image is displayed on the second display screen. 9. The user interface device according to claim 8, wherein: The image generation means cuts out a part of the display image displayed on the second display screen based on the position and orientation information indicating the relative position of the image capturing device from the estimated position coordinates. The user interface device according to claim 2, wherein the partial display image is enlarged and generated. The user interface device according to claim 2, wherein the image generation unit generates an image to be displayed on a web browser. An information display device that displays a new image including the cursor image on the basis of an image recognition result of a captured image obtained by the image capturing device capturing a cursor image displayed on a display screen,
Image generating means for generating a display image to be displayed on the display screen;
Cursor display means for synthesizing a plurality of types of one cursor image that repeatedly changes at a predetermined time period on the display image by the image generation means;
Image ID storage means for storing a plurality of types of cursor images and image IDs for specifying display images on which the cursor images are displayed in association with each other;
Cursor data receiving means for receiving cursor data representing the cursor image that is an image recognition result of the cursor image included in the captured image from the image capturing device;
Cursor recognition means for specifying an image ID of the captured cursor image based on the cursor data received by the cursor data receiving means and the stored contents of the image ID storage means;
The position coordinates on the display screen of the captured cursor image are calculated from the cursor data received by the cursor data receiving means, and the calculated position coordinates are stored in association with the specified image ID. Cursor coordinate storage means for
The display image picked up by the image pickup device from the cursor data received by the cursor data receiving means and the position coordinates stored in the cursor coordinate storage means, and the image pickup device on the display image Including position estimation means for estimating the indicated coordinates indicating the position currently pointed to,
The cursor display means reads out a new cursor image corresponding to a new image ID from the image ID storage means, and displays the new cursor image at the indicated coordinates estimated by the position estimation means. An information display device characterized by causing the image generation means to synthesize the cursor image. An operation terminal for displaying a new image including the cursor image on the information display device based on an image recognition result of a captured image obtained by photographing a cursor image displayed on a display screen of the information display device. ,
Imaging means for imaging a cursor image displayed on a display screen of the information display device;
Cursor data transmission means for recognizing the cursor image included in the captured image captured by the imaging means and transmitting cursor data representing the cursor image as the image recognition result to the information display device;
Cursor data receiving means for receiving cursor data of a cursor image to be newly displayed from the information display device;
Second image generating means for generating a display image by synthesizing a new cursor image with the captured image based on the captured image captured by the imaging means and the cursor data received by the cursor data receiving means. When,
An operation terminal comprising: second display means for displaying the display image generated by the second image generation means. A digital television device comprising: a display screen for displaying a display image generated by the image generation means; and the user interface device according to claim 1. A mobile phone terminal comprising: the image pickup device; and the user interface device according to claim 2. A program for a user interface device that displays a new image including the cursor image on the basis of an image recognition result of a captured image obtained by the image capturing device capturing a cursor image displayed on a display screen, Image generating means for generating a display image for displaying a computer on a display screen, cursor display means for combining a plurality of types of one cursor image that changes repeatedly at a predetermined time period on the display image by the image generating means, Image ID storage means for storing in advance a plurality of types of cursor images and image IDs for specifying display images on which the cursor images are displayed, and the cursor image included in the captured image. An image of the cursor image that is image-recognized and imaged based on the stored contents of the image ID storage means Cursor recognition means for identifying an ID from the recognized cursor image; calculating position coordinates on the display screen of the captured cursor image; and calculating the position coordinates as the identified image ID The display image picked up by the image pickup device from the cursor coordinate storage means stored in association, the recognition result by the cursor recognition means, and the position coordinates stored in the cursor coordinate storage means, It functions as a position estimation unit that estimates a pointed coordinate indicating a position that the image capturing device currently points on a display image, and the cursor display unit displays a new cursor image corresponding to a new image ID. Read from the storage means, so that the new cursor image is displayed at the indicated coordinates estimated by the position estimation means, Program for synthesizing the cursor image to the serial image generating means. An integrated circuit that displays a new image including the cursor image based on an image recognition result of a captured image obtained by the image capturing device capturing a cursor image displayed on a display screen;
Image generating means for generating a display image to be displayed on the display screen;
Cursor display means for synthesizing a plurality of types of one cursor image that repeatedly changes at a predetermined time period on the display image by the image generation means;
Image ID storage means for storing a plurality of types of cursor images and image IDs for specifying display images on which the cursor images are displayed in association with each other;
Cursor recognition means for recognizing the cursor image included in the captured image and identifying the image ID of the captured cursor image from the recognized cursor image based on the stored contents of the image ID storage means; ,
Cursor coordinate storage means for calculating a position coordinate on the display screen of the captured cursor image, and storing the calculated position coordinate in association with the specified image ID;
The display image picked up by the image pickup device and the image pickup device on the display image currently indicate from the recognition result by the cursor recognition means and the position coordinates stored in the cursor coordinate storage means. Position estimation means for estimating the indicated coordinates indicating the current position,
The cursor display means reads out a new cursor image corresponding to a new image ID from the image ID storage means, and displays the new cursor image at the indicated coordinates estimated by the position estimation means. An integrated circuit characterized by causing the image generation means to synthesize the cursor image.

Images