JP2003323239A - Information processing apparatus and information processing method, storage medium, and computer program - Google Patents

Abstract

(57) [Problem] To provide a suitable operation environment for selecting desired identification information from a captured image of a real world scene in which identification information of visibility is arranged. SOLUTION: An "image & ID storage unit" and a "capture button" are provided on the information terminal. Then, when the user points the mobile terminal toward the desired optical beacon and the display screen of the mobile terminal indicates that the optical beacon has been recognized, the user presses the “capture button” to display the image and I
D (that is, information received from the optical beacon) is recorded in the “image & ID storage unit”. Thereafter, when the captured image is displayed on the screen, the user can select an optical beacon in a comfortable posture.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an information processing apparatus and an information processing method for starting a predetermined process in response to a user operation, a storage medium, and a computer program, and more particularly to a user's operation via a display screen. The present invention relates to an information processing device and an information processing method for receiving a selection operation and activating a predetermined process, a storage medium, and a computer program.

More specifically, the present invention relates to an information processing apparatus and an information processing method for providing a user operation environment on a photographed image of a real world landscape, a storage medium, and a computer program, and in particular, the identification of visibility. The present invention relates to an information processing apparatus and an information processing method, a storage medium, and a computer program that provide an operation environment for selecting desired identification information from a photographed image of a real world landscape in which information is arranged.

[0003]

2. Description of the Related Art In the modern age of advanced information processing technology and information communication technology, information devices such as personal computers and personal digital assistants are ubiquitous in the real world such as offices and homes. To do. In such an environment, actively connect to devices such as "Ubiquitous computing" that connects devices to obtain desired information anytime and anywhere, and situations in the real world (such as real-world things and user positions). It is expected that the augmented reality system (Augmented Reality: AR) used for

The concept of ubiquitous computing is that the environment of computers that can be used regardless of where a person moves is the same. In other words, because it is "anytime, anywhere", the ultimate ubiquitous computing is not necessarily a computer or PDA (Personal Dig
Ital Assistant) and information terminals such as mobile phones are not necessarily required.

However, it is attempted to specify a computer or a peripheral device (that is, a target such as a user terminal) as a data transfer destination on the network, or to obtain a real world situation such as information related to the position of the user or a real world object. Then, it is necessary to know the name (or the device-specific ID, the network address, the host name, the resource identifier such as the URL / URI) of the person in front of him. That is, in terms of user operations, the computers are linked only in an indirect form, and the intuition is somewhat lacking.

As a technique for transferring the user's identification information and acquiring the real world situation such as the user's position by omitting such a complicated procedure, "Cyber Code (C
A method using real-world computing such as a visual code such as "ybercode)" or an RF tag has been proposed. According to these methods, the user does not need to consciously access the network, and the information related to the object can be acquired from the ID of the object automatically picked up.

For example, a function such as an application, a device ID, a network address, a host name, a URL, and other object-related information are registered in advance for the visual code and the identification information of the RF tag. Then, in response to recognizing the cyber code from the image captured by the camera, the computer executes the registered application (for example, “activates mail”), or based on the recognized ID, the network of the other party.・ Search for an address and automatically connect,
Resource access can be performed based on the recognized URL.

However, when the visibility identification information such as the visual code is used, the size of the code changes according to the distance. That is, since the code becomes smaller as the object is far away, it is necessary to form a code having a large pattern in order to recognize a far object.
In other words, the information transmission method based on this method lacks robustness against distance. For example, in order to recognize a building in the distance, it is necessary to paste a huge code into the building, which is not realistic.

Further, in the case of the RF tag, the user has to point or contact the RF tag to the tag reading device. That is, it is possible to recognize only an object at a close range and not a far object.

Therefore, in Japanese Patent Application No. 2002-57836 already assigned to the present applicant, an optical signal including a blinking pattern of LEDs is used as a data transmission medium, and data robust against distance is used. Communication systems have been proposed. For example, in real-world objects such as indoor furniture, appliances, and outdoor buildings, a transmitter including two or more light emitting units such as LEDs arranged in a predetermined physical arrangement is installed, and each light emitting unit has a predetermined bit. Data transmission is performed by blinking with a blinking pattern representing long transmission data. One of the receivers is equipped with a light receiving unit consisting of a two-dimensional light receiving surface, decodes transmission data based on the received blinking pattern, and recognizes spatial information of the object based on the blinking position on the two-dimensional light receiving surface. To do.

That is, according to the data communication system using such an optical signal, it is possible to simultaneously obtain information such as an ID from an object in the real world and at the same time recognize the spatial position of the object. -It can be an effective way to build computing.

[0012]

If the data communication system using the optical signal described above is applied to an augmented reality system, a service using real world information such as the position of the user can be provided. In this case, the system supports all aspects of daily life by utilizing the vast amount of information on the network by presenting information according to real-world things in the vicinity of the user or in the field of view by optical signals. be able to. The user can use the augmented reality system simply by holding the mobile terminal.
For example, when a user visits a CD shop by holding a camera-equipped mobile terminal in a shopping mall, a recommended new song is displayed on the terminal. Also, looking at the signboard of the restaurant, the impression of the food is displayed.

In such an application, the optical signal is a resource identifier, such as a URL / URI or IP address, for extracting information content from the information space. Also, the optical signal present in the image taken by the camera is
It has a role as link information or an anchor for searching the storage location of the desired information content.

However, by capturing the optical signal with a camera,
There is a usability problem for the user when invoking a link search or other process assigned to an optical signal.

In such a case, the user holds the camera-equipped information terminal and performs a certain selection operation on the terminal when a desired optical signal is detected. For example, it suffices if the camera can be selected in a posture holding a desired real-world object (for example, a billboard of a building).
It is impossible to confirm which optical signal is actually detected when the hand is extended and the object is directed toward the object. This is because when the terminal is moved to a place where the display screen of the terminal can be easily seen (for example, in front of the body), the optical signal disappears outside the field of view of the camera.

Further, when the user holds the camera-equipped information terminal in a desired line-of-sight direction to take a picture, a plurality of optical signals may be detected in the same image frame. In such a case, it is necessary to select a desired optical signal from the photographed image or perform a selection confirmation operation, but a complicated terminal operation cannot be performed in the posture of holding the camera. Also, if the terminal screen is moved to a place where it is easy to see, the optical signal will be shaken off the screen frame, and it will not be possible to select it accurately.

The present invention has been made in view of the above technical problems, and its main purpose is to accept a user's selection operation via a display screen and suitably start a predetermined process. An excellent information processing apparatus and information processing method capable of
It is to provide a storage medium and a computer program.

A further object of the present invention is to provide an excellent information processing apparatus and information processing method, a storage medium, and a computer program that can provide a suitable user operation environment on a photographed image of a real world landscape. To do.

Another object of the present invention is to provide excellent information processing that can provide a suitable operating environment for selecting desired identification information from a photographed image of a real world landscape in which visibility identification information is arranged. An object is to provide an apparatus, an information processing method, a storage medium, and a computer program.

[0020]

The present invention has been made in consideration of the above problems, and a first aspect thereof is an information processing apparatus for activating a predetermined process in response to a user operation, or An information processing method, an image input means or step for inputting a landscape image in the real world including visibility identification information, and a decoding means for decoding the visibility identification information included in the input image, Storage means or step for storing the input image and the decoding result in association with each other, image display means or step for displaying the input image stored in the storage means or step, and appearance on the display screen of the image display means or step User input means or step for accepting a user selection operation for the visibility identification information to be performed, and a service for activating a processing service corresponding to the visibility identification information selected by the user An information processing apparatus or the information processing method characterized by comprising a row unit or step.

Here, the visibility identification information is, for example, 1 which is capable of optically discriminating byte strings forming data.
Although displayed as the above pattern, of course, other forms of visibility data such as a visual code may be used.

The information processing apparatus or the information processing method according to the first aspect of the present invention can be applied to an augmented reality system using data communication using an optical signal as a data transmission medium. The augmented reality system supports all aspects of daily life by utilizing the vast amount of information on the network by presenting information according to real world objects in the vicinity of the user or in the field of view by using optical signals. To do. Then, the user can use the augmented reality system only by holding an information terminal equipped with a function of detecting an optical signal, such as a camera.

In such a case, the user holds up the camera-equipped information terminal and carries out a certain selection operation on the terminal when a desired optical signal is detected. However,
In a posture in which the camera is held over a desired real-world object (for example, a billboard of a building), it is not possible to confirm a captured image of the camera. Also, when trying to view the display screen of the terminal, the optical signal is no longer visible outside the field of view of the camera. When a plurality of optical signals are detected in the same captured image, a desired optical signal cannot be accurately selected without looking at the captured image.

On the other hand, according to the information processing apparatus or the information processing method according to the first aspect of the present invention, the user captures a desired real world landscape image with a camera or the like and includes it in the input image. The visibility identification information is detected, and the decoded identification information is recorded in association with the input image. When the photographing of the real world landscape is completed in this way, the user uses the photographed image displayed on the screen again to select the visibility identification information in a comfortable posture and activate the desired processing service. It is possible.

It is also possible to record many real-world scenes here and there, slowly view the photographed image when time is available, and activate the processing service by using the desired identification information. Is.

In the case where the identification information code of the visibility identification information installed on the monitor changes according to the content of the monitor, the content of the information is displayed while the user confirms and selects the identification information. May change,
It is very inconvenient for the user. In contrast,
According to the present invention, even in such a case, since the identification information can be easily recorded by capturing an image, it is possible to surely obtain the identification information of the visibility, and its effectiveness is great. Is.

The second aspect of the present invention is a computer-readable physical form of computer software written to execute a process for activating a predetermined operation in response to a user operation on a computer system. Computer-readable storage medium, the computer software includes an image input step of inputting a real-world landscape image including visibility identification information, and decoding the visibility identification information included in the input image. A decoding step of converting the input image and a decoding result in association with each other, an image display step of displaying the input image stored in the storage step, and a screen displayed in the image display step. A user input step that accepts a user selection operation for the visible identification information that appears,
A service execution step of activating a processing service corresponding to the visibility identification information selected by the user, and the storage medium.

The storage medium according to the second aspect of the present invention is a medium for providing computer software in a computer-readable format to a general-purpose computer system capable of executing various program codes, for example. Such a medium is, for example, a DVD (Digital Vers
atile Disc), CD (Compact Disc), FD (Flexible Disc)
Disk), MO (Magneto-Optical disc), and other removable storage media. Alternatively, it is technically possible to provide computer software to a specific computer system via a transmission medium such as a network (whether the network is wireless or wired).

Further, the storage medium according to the second aspect of the present invention is a computer system in which a predetermined computer
A computer for realizing software functions
It defines a structural or functional collaborative relationship between software and a storage medium. In other words, the second aspect of the present invention
By installing the predetermined computer software in the computer system via the storage medium according to the first aspect, a cooperative action is exerted on the computer system and the information processing apparatus according to the first aspect of the present invention. The same effect as that of the information processing method can be obtained.

Further, a third aspect of the present invention is a computer program written in a computer-readable format so as to execute a process for activating a predetermined operation in response to a user operation on a computer system. The image input step of inputting a landscape image in the real world including the visibility identification information, the decoding step of decoding the visibility identification information included in the input image, and the input image and the decoding result. A storage step of storing the data in association with each other, an image display step of displaying the input image stored in the storage step, and a user selection operation for the visibility identification information appearing on the screen displayed in the image display step are accepted. The user input step and the service execution step that starts the processing service corresponding to the visibility identification information selected by the user. Is a computer program characterized by comprising the flop, the.

A computer according to the third aspect of the present invention.
The program defines a computer program written in a computer-readable format so as to realize a predetermined process on a computer system. In other words, by installing the computer program according to the third aspect of the present invention in the computer system, a cooperative action is exerted on the computer system, and the information processing according to the first aspect of the present invention is performed. It is possible to obtain the same effect as that of the device or the information processing method.

Further objects, features and advantages of the present invention are as follows.
It will be apparent from the embodiments of the present invention described later and the more detailed description based on the accompanying drawings.

[0033]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described below in detail with reference to the drawings.

A. Configuration of Data Communication System The present invention is realized by utilizing a ubiquitous network in which data transmission is performed by recognizing visibility identification information with a camera. That is, a plurality of optical identification information is attached to a physical object existing in the real world, and a two-dimensional image sensor captures the image to perform image processing and image recognition, thereby transmitting data from the object. Interpret.

As the optical identification information attached to the real-world object, an optical signal composed of a blinking light source such as an LED can be used in addition to the visual code encoded in the color space pattern. In the latter case,
By encoding data into a signal composed of a time-series blinking pattern and transmitting the data, it is possible to perform data communication that is robust with respect to distance.

The image sensor is, for example, a CMO.
Like an S (Complementary Metal Oxide Semiconductor) sensor or CCD (Charge Coupled Device) sensor, it has a configuration in which a myriad of light-receiving elements, that is, pixels, are arranged in a two-dimensional array, and optical signals and their spatial information are stored in all pixels. Decode with. The image sensor can capture the scene as a normal camera, as well as receive optical signals located in the field of view of the image sensor from long distances.

First, the principle of a data communication system using an optical signal as a data transmission medium will be described.

FIG. 1 schematically shows the configuration of a data communication system 10 which uses optical signals. As shown in the figure, the data communication system 10 includes transmitters 20A, 20A installed in respective objects scattered in a real space.
B, 20C ... And a receiver 50 carried and used by the user.

The transmitter 20 transmits an optical signal in which a byte string as transmission data is displayed as one or more patterns capable of being optically separated. One of the receivers 50 receives the displayed optical pattern, recognizes the received light image, calculates the transmitted byte string based on the blinking pattern represented by the optical signal, and also detects the optical signal. The detection position on the light receiving surface is calculated. Then, the processing service according to the combination of the byte string and the light receiving position is executed.

Each of the transmitters 20A ... Is installed in an easily visible place on the surface of the real world object such as an outer wall of a building or a signboard, and has one or more light emitting portions. The light emitting unit is composed of, for example, a microcomputer-controlled LED. Then, the transmission data in a digital format having a predetermined bit length is modulated by a predetermined modulation method, and is encoded into a display format in which a bit 1 is represented by turning on the LED and a bit 0 is represented by turning off the LED, and a predetermined bit (probably N bits Optical signal corresponding to the long transmission data is generated. Hereinafter, such a transmitter is also referred to as “optical beacon”.

The transmission data transmitted by each transmitter 20A, etc. includes an object ID, its network address, host name, resource identifier such as URL and URI, and other object related information.

FIG. 2 schematically shows the internal structure of the transmitter 20. As shown in the figure, the transmitter 20 includes a transmission data generation unit 21, a data processing unit 22, and a pattern display unit 23.

In the transmission data generation unit 21, this system 1
An arbitrary byte string is generated as 0 transmission data. This could be, for example, a button, a joystick, a track
It is composed of a combination of an input circuit that receives an input from a user such as a ball and an A / D conversion circuit. Alternatively,
The transmission data may be received from an external device (not shown) such as a personal computer that is interface-connected via a wired or wireless communication medium, or the character string or byte string read from the nonvolatile memory may be used as the transmission data. Good.

The information processing section 22 reads the data from the transmission data generating section and generates a series of patterns by the procedure described later. The generated pattern is displayed in the pattern display unit 2
Sent to 3.

The pattern display section 23 displays a plurality of temporally or spatially patterns that can be optically separated. For example, a blinking LED can be used to display a bit sequence of 0 and 1 (display of a temporal pattern). Further, by arranging a plurality of (N) LEDs on a two-dimensional lattice and turning each LED on or off, N
Bits of information can be displayed as a spatial pattern.

The data transfer rate of the transmitter 20 is LED
Is determined by the driving speed at which the blinks. For example, in the case of general light emitting diode products currently on the market, 4
Flashing can be performed at a frequency of about kHz. However, this is limited by the response speed of the light receiving block 51 on the receiver 50 side.

For example, as a light receiving portion in which the light receiving blocks 51 are arranged in a two-dimensional matrix, a CCD image
When a sensor is used, its response speed is about 60 Hz, which is too slow to transmit and receive the LED blinking data. On the other hand, if the CMOS image sensor described above is 12
Since it has a response speed of kHz or higher, the transmitter 20
Even if the LED is driven at the maximum speed on the side, the blinking pattern will be sufficiently detected on the side of the receiver 50.

If the transmission data has a length of 8 bits, 25
It is possible to transmit 5 types of data. 8-bit transmission data is Manchester-encoded with a 4 kHz carrier and transmitted as a 22-bit packet including a start code. As a result, even if the optical signal is hidden during packet transmission due to an obstacle or the like, data can be transmitted in packet units.

FIG. 3 shows the data processing unit 2 in the transmitter 20.
2 shows the internal configuration of No. 2. As shown in the figure, a data input unit 31 for reading transmission data and a central processing unit (CPU: Central Processing Uni) for executing a pattern generation process.
t) and the like, and a RAM (Random Access Me) for loading an execution program for pattern generation and temporarily storing work data during pattern generation processing.
mory) 33 and a ROM (Read Only Memory) for storing the program code executed by the arithmetic processing circuit 32.
y) and the like, and a pattern output section 35 for outputting the generated pattern. The arithmetic processing circuit 32, the input / output units 31 and 35, and the memories 33 and 34 are connected by a bus 36.

As shown in FIG. 1, one receiver 50 includes a plurality (M) of light receiving blocks 51-1, 51-2,
, 51-M, a data processing unit 60 that integrally processes the outputs of the respective light receiving blocks 51-1 ..., and an application unit 70 that executes a predetermined service according to the data processing result.

The light-receiving blocks 51-1, 51-2, ... Are actually arranged in a two-dimensional matrix of n × m (= M) as shown in FIG. It constitutes a light-receiving part consisting of a light-receiving surface. Such a light receiving unit can be configured by using, for example, a CMOS image sensor, and one pixel corresponds to one light receiving block. Since the CMOS image sensor generally has a higher driving speed than the CCD sensor, the blinking pattern can be read even when the LED is driven at a relatively high frequency on the transmitter side.

FIG. 5 schematically shows the internal structure of one light receiving block 51 in the receiver 50. As shown in the figure, one light receiving block 51 includes a light receiving element 81 for generating an electric signal according to the amount of received light, a band pass
A filter (BPF) 82, a phase lock loop (PLL) 83, an A / D converter 84, an imaging data storage memory 85, a binarized data storage memory 86,
It is composed of a transfer data storage memory 87 and a control unit 88 for comprehensively controlling and synchronously driving the inside of the light receiving block 51.

The light receiving element 81 is composed of a CMOS sensor element. In addition, other processing modules for driving the light-receiving element and decoding are installed in FPGA (Field Programmable G
ateArray) and the like.
The imaging data storage memory 85 is used to temporarily store the grayscale pixel value in the light receiving element 81 when the receiver 50 operates as a camera.

The blinking data of the LED from the transmitter 20 side is imaged on the two-dimensional light receiving surface of the light receiving portion via a condenser lens system (not shown), and corresponds to the optical axis of the LED on the two-dimensional light receiving surface. The light receiving block at the position can detect the blinking pattern. In addition, the light receiving section including the CMOS image sensor not only detects the blinking signal of the LED for each pixel, that is, each light receiving block, but also functions as a camera for capturing an image formed on the light receiving surface.

The data collecting unit 60 includes the light receiving blocks 51.
-1 ... Outputs are processed in an integrated manner. The contents of the processing are the processing of the blinking signal of the LED detected for each pixel, that is, each light receiving block, and the image frame imaged by each light receiving block 51-1, ..., 151-M arranged in a two-dimensional matrix. There are two types of processing.

FIG. 6 shows the data processing unit 6 in the receiver 50.
0 schematically shows the internal configuration of 0. As shown in the figure, the data processing unit 60 includes an image input unit 61 that reads an image from the light receiving unit and a central processing unit (CPU) that detects a pattern, calculates a position, and calculates a byte string from an extracted pattern. An arithmetic processing circuit 62 such as a RAM and a RAM used for loading an execution program and temporarily storing work data
, A program memory 64 such as a ROM for storing the program code executed by the arithmetic processing circuit 62, the type and position of the recognized pattern, and the result for outputting the received byte string. The output unit 65 is provided. Further, the arithmetic processing circuit 62 and the input / output unit 6
1, 65 and memories 63, 64 are connected by a bus 66.

The application section 70 executes a predetermined service according to the data collection result by the data processing section 60. For example, the application unit 70 corresponds to the transmission data from the transmitter 20 that is decoded based on the data collection result, in addition to performing image processing and image recognition on the captured image obtained from the light receiving unit that functions as a camera. Execute the processing service.

The transmission data from the transmitter 20 is the device I
D, network address, host name, URL
Resource identifiers such as URIs and URIs, other object related information, and the like, and the application unit 70 can perform processing related to objects. For example, based on the URL decoded from the received optical signal, the information related to the object is searched in the WWW (World Wide Web) information providing space, the search result is displayed, and further, it is displayed on the image taken by the camera. The search results can be displayed as an overlay (NaviCam). Also,
A network connection between the object and the user terminal can be established based on the decoded device ID and network address of the object (gaz).
e-link).

The receiver 50 is, for example, a digital camera using a CMOS image sensor, which is a USB (Univer
sal Serial Bus) is configured by connecting to a computer via an interface such as. In such a case, the application unit 70 corresponds to an application executed on a computer and superimposes a computer image based on digital data decoded from an optical signal on a captured (scene) image input from a digital camera. The synthesized image can be displayed.

An image sensor for photographing a scene such as a digital camera can be additionally equipped with a function as the receiver 50. For example, when transmitting an optical signal as a high-speed blinking pattern, a CMOS image sensor which is a device capable of high-speed sampling is applied. A general digital camera shoots at 30 fps, but even if the speed is further increased, it exceeds the ability of a person to recognize a moving image. From this, it is possible to add value to the CMOS image sensor by providing a "decode mode" for receiving and processing an optical signal by utilizing the extra time other than the "camera mode" for capturing a scene at 30 fps. You can

In decode mode, for example, 12 kHz
The sampling of is repeated 200 times, the 8-bit optical signal of carrier frequency 4 kHz sent from the transmitter 20 side is decoded by all the light receiving blocks of the image sensor, and the reception / recognition image of the optical signal is created at 15 fps. You can This recognition image is the result of decoding the value of each pixel of the image, and both the transmission data represented by the optical signal and the spatial information of the transmitter emitting the optical signal (that is, the real world situation of the real world object) Is included.

In the data communication system 10 shown in FIG. 1, the transmitter 20 transmits data by displaying a byte string as transmission data as one or more patterns that can be optically separated. One of the receivers 50 receives the displayed optical pattern on the two-dimensional light receiving surface, recognizes the received light image, calculates the transmitted byte string based on the series of optical patterns, and The position of the static pattern on the light receiving surface is calculated. Even if the position of the display pattern output by the transmitter 20 changes between image frames, the position is tracked to enable reception of an arbitrary byte string.

FIG. 7 shows a data transmission processing procedure on the transmitter 20 side in the form of a flowchart.

First, 1 of the data (byte string) to be transmitted
The data for the frame is read from the data input unit 31 (step S1).

Next, the data acquired in the previous step is
It is converted into a data frame format (described later) as shown in FIG. 8 and temporarily stored in the memory 33 (step S
2).

Next, the generated data frame is read from the memory 33, and in the pattern display section 35,
Data is transmitted by displaying an optically distinguishable pattern such as blinking of LED light or change of intensity of light (step S3).

Then, it is determined whether the data transmission should be continued or stopped (step S4). If the data transmission is to be continued, the process returns to step S1 and the same optical pattern display process as described above is performed.

FIG. 8 shows a format of a data frame used for data transmission in the data communication system 10 using an optical signal.

At the beginning of this data frame, a byte string (start code) indicating the start of the frame is arranged. This byte string is a unique byte string that does not occur anywhere else.

Next, the transmission data (payload) is stored in a predetermined length. When the payload has a variable length, the size of the payload (or frame) may be stored at the beginning.

At the end of the data frame, a checksum for checking whether the transmitted data string has an error is added. However, this checksum is not essential and may be omitted. If the same byte string as the start code occurs in the payload or checksum,
Need to convert to another byte sequence.

Here, an example of a conversion method for preventing the above start code from occurring in the data part and the checksum will be described. In this example, the start codes are A ₁ , A ₂ ,
A ₃ and A _{4 are} the minimum transmission data series, and A ₁
To A ₄ are different from each other.

First, the payload and the data of the checksum portion are sequentially examined. When the pattern A ₁ , A ₂ , A ₃ is found, it is replaced with the pattern A ₁ , A ₂ , A ₃ , B. However, B is a value different from A ₁ to A ₄ . By this conversion, the start code becomes a series that does not occur in the payload or checksum.

Next, the method of inverse conversion will be described. Reverse
In the conversion, A from the beginning of the data ₁, A₂, A₃, B and
Look for the series called A₁, A₂, A₃Change to the series
Replace. By this reverse conversion, the data string is the data before conversion
Return to.

FIG. 9 shows, in the form of a flow chart, a detailed processing procedure of data frame generation in step S3 of the data transmission processing shown in FIG.

First, a pattern counter k for counting the number of patterns transmitted is set to an initial value 0 (step S11).

Next, the memory 33 for the data frame
Then, the k-th minimum transmission data is read and converted into a display pattern (step S12). The minimum transmission data is converted into a display pattern using the conversion table.

Then, the pattern is transferred to the pattern display section 23, and data is transmitted by displaying a pattern that can be optically distinguished such as blinking of light and change of intensity (step S13).

Then, the pattern counter k is incremented by 1 (step S14). Then, it is checked whether all the data in the data frame have been transmitted (step S15), and if there is data that has not been transmitted, step S15.
Returning to step 12, the next minimum unit of data is transmitted.

By switching the display pattern from the transmitter in this manner, the byte string can be transmitted. Next, a method of receiving data at the receiver will be described.

FIG. 10 shows, in the form of a flow chart, the procedure for receiving processing on the receiver 50 side of the data transmitted from the pattern display section 23 in the form of a pattern that can be optically discriminated. This reception process is actually realized by the data processing unit 60 recognizing the received light image by the light receiving unit 50.

First, image reading and pattern recognition are performed (step S21). The pattern recognition method differs depending on what is used as the display pattern. In this embodiment, all the optical patterns existing on the received light image and the positions of the respective optical patterns are detected.

Next, the transmission data is restored by converting the recognized pattern into an N-bit byte string (minimum transmission data) (step S22). This conversion table is an inverse conversion of the conversion table (described above) used to convert the minimum transmission data into a pattern on the transmitter 20 side.

Then, all the patterns received by the light receiving surface are registered in the data buffer provided in the working memory 54 (step S23). Details of the registration method will be given later.

Then, the end judgment is made (step S2).
4). Then, if the data reception is to be continued, the process returns to step S21, and the decoding process of the transmission data in the minimum transmission data unit similar to the above is continued.

FIG. 11 schematically shows the structure of the reception data buffer provided in the working memory 54.
The reception pattern registration method in step S23 will be described with reference to FIG.

In the data buffer, an entry is prepared for each series of recognized patterns, that is, for each transmitter.
As shown in FIG. 11, each entry is composed of a position (X, Y) where a pattern is detected on the light receiving surface and a byte sequence obtained by converting the light receiving pattern.

On the light receiving surface, the optical pattern is received moment by moment. Then, when the optical pattern is converted using the conversion table to obtain the byte series, this is registered in the corresponding entry. The registration referred to here means that the bytes are sequentially connected. The byte sequence finally generated by concatenating the byte sequences becomes the data sent from the corresponding transmitter. In the example shown in FIG. 11, points (X ₁ , Y ₁ ), (X ₂ , Y ₂ ), (X ₃ ,
The transmission data 1 to 4 from the optical beacon detected by Y ₃ ), (X ₄ , Y ₄ ), ... Are stored in each entry.

B. Augmented Reality System By applying the above-mentioned data communication system using optical signals to an augmented reality system, it is possible to provide a service using real world information such as a user's position. In this case, the user simply holds the mobile terminal, and the system presents information according to real-world things in the vicinity of the user or in the field of view, and uses the vast amount of information on the network to carry out daily life. Can support all aspects.

For example, a transmitter having an LED as a light source is installed in a predetermined place in the real world such as a signboard, a lighting, a building lamp, and a traffic signal. When the user looks at the signboard of the restaurant on an information terminal such as a PDA equipped with (or connected to) a camera as a receiver, the user acquires the object ID of the restaurant transmitted in the form of an optical signal from the transmitter. be able to. Furthermore, this object ID
Can be retrieved from the database to retrieve information such as a restaurant menu and display it on the PDA screen.

When a movie poster in front of the viewer is viewed with a camera-equipped information terminal, the movie ID information is acquired from a transmitter installed near the poster, and a database search is performed based on this ID information. By this, the trailer movie of the movie is taken out and displayed on the screen of the information terminal.

When a building such as a building or bridge is distant from the information terminal with a camera, the ID information of the building is extracted by blinking a lamp attached to the building, and based on this ID information. By searching the database, the event information held in the building or its neighborhood is extracted and displayed on the screen of the information terminal.

FIG. 12 schematically shows the configuration of the augmented reality system 100 according to this embodiment.

The augmented reality system 100 includes an optical beacon 110 installed on each object in the real world,
An information terminal 130 with an ID recognition camera that captures an image of the real world and recognizes an optical signal emitted by the optical beacon 110, and a service provider (server) 1 that provides various information services to the information terminal 130.
It is composed of 50 and.

The optical beacon 110 is the transmitter 2 shown in FIG.
0, and the information terminal 130 with an ID recognition camera
Corresponds to the receiver 50, and these are unidirectionally connected by a data communication system based on optical signals.

The information terminal 130 and the service provider 150 are interconnected via a wide area computer network such as the Internet.

It is assumed that the information terminal 130 has a small size, such as a PDA or a mobile phone, and is owned by each user. Further, in FIG. 12, the optical beacon 11
0 and one service provider 150 each, but of course various real-world objects (billboards and posters on street corners, exterior walls of buildings, etc.)
A myriad of optical beacons are scattered near each other to send out unique data, and a myriad of service providers 150
May reside on a computer network and provide various information services.

FIG. 13 further shows the internal structure of the information terminal 130 with the ID recognition camera.

The ID recognition camera section 131 corresponds to the light receiving section 50 and the data processing section 60, and is operated from the optical beacon 110 in accordance with the "camera mode" for photographing the scenery of the real world and the mechanism as shown in the previous section A. The two operation modes of "decoding mode" for decoding the transmission signal of 1 are executed in a time-division format.

The computer section 132 corresponds to the application section 70, and is composed of a CPU, a RAM, a ROM, and the like. The CPU activates a computer program under an execution environment provided by an operating system (OS) to perform various operations. Providing processing services. For example, the computer unit 132 provides, in addition to the processing service of the main body of the information terminal 130, an extended processing service that integrates the real world captured image under the camera mode of the ID recognition camera unit 131 and the decoding result of the optical signal under the decoding mode. To do. The execution program of the CPU is RO
It is permanently written to M or downloaded via a computer network and loaded into RAM for processing.

The computer section 132 has the information terminal 130.
A display unit (for example, a liquid crystal display) 133 as a user interface of the main body and a user input unit 134
Is equipped with.

On the display unit 133, in addition to the normal terminal operation screen, a real world landscape image captured by the ID recognition camera unit 131 is displayed and an extended processing service screen such as a decoding result of the received optical signal is displayed. indicate.

In addition to the normal terminal input function, the user input section 134 has a capture button (described later) and an optical beacon selection function on the extended processing service screen. The user input unit 134 is composed of a device for performing an input operation such as a button, a cursor button, a jog dial, and a trackball. In the present embodiment, some of these operation units include a capture button, a selection button, an EXI
It is assigned to the T button (described later). The cursor button is used to tilt in the left-right direction in response to the pressing operation and transmit the tilted direction to the computer unit 132.

The image & ID storage unit 135 stores the real world scenery image picked up by the ID recognition camera unit 131 and the decoding results of one or more optical signals detected on the scenery image in association with each other. Image & ID storage 1
The numeral 35 is composed of, for example, a nonvolatile semiconductor memory such as NVRAM or flash memory, or a hard disk device.

The computer section 132 is a network
It is connected to an external computer network via the interface unit 136. The network interface unit 136 implements a network protocol such as TCP / IP, a mobile phone network, a wired or wireless LAN, or a USB (Universal Serial).
Bus), IEEE 1394, Bluetooth, etc., to connect to a computer network.

After starting the extended processing service of the information terminal 130, the user points the ID recognition camera 131 toward the desired optical beacon 110. As a result, the image of the camera is displayed on the screen of the display unit 133 together with the information obtained from the optical beacon 110 or the information provided from the service provider 150 via the network based on the information.

Augmented reality system 1 as shown in FIG.
In 00, the information transmitted from the optical beacon 110 may include an address on the network of the service provider 150 capable of providing information related to the real world object, an identifier for identifying an information resource such as a URL, and the like. For example, it is possible to access the service provider 150 and receive a desired service by using the network connection function of the mobile terminal.

That is, the I built in the information terminal 130 is
The direction of the line of sight of the D recognition camera 131 is directed toward the desired real world object or its optical beacon 110, and the information terminal 130
When it is shown on the display unit 133 that the optical beacon is recognized, if the user wants to access the information, the service is activated by pressing the select button.

However, it suffices if the user can select the information terminal 130 with the ID recognition camera while holding the desired real world object (for example, a signboard of a building) in a desired posture, but the user reaches out to the object. In the posture, it is not possible to confirm which optical signal is actually detected. This is because when the terminal is moved to a place where the display screen of the terminal 130 is easy to see (for example, in front of the body),
This is because the optical beacon 110 has already disappeared outside the field of view of the ID recognition camera 131.

Further, when the user photographs the information terminal 130 with the ID recognition camera by holding it in the desired line-of-sight direction, a plurality of optical beacons may be captured in the same image frame. In such a case, it is necessary to select a desired optical beacon from the captured image or perform a selection confirmation operation, but a complicated terminal operation cannot be performed with the posture of holding the camera. Also, if the terminal screen is moved to a place where it is easy to see, the optical signal will be shaken off the screen frame, and it will not be possible to select it accurately.

Therefore, the information terminal 130 with the ID recognition camera according to the present embodiment is provided with the capture button and the image & ID storage section 135, and the ID recognition camera 1
The configuration is such that the selection operation of the optical beacon can be performed even when 31 is not held over the optical beacon. The user points the information terminal 130 toward the desired real-world object,
When the desired optical beacon is captured by the ID recognition camera 131, that is, when the display unit 133 of the information terminal 130 indicates that the optical beacon is recognized, the capture button is pressed. On the information terminal 130 side, in response to the pressing operation, the captured image and the I detected from the image are detected.
Image & ID of D (information received from optical beacon 110)
It is recorded in the storage unit 135. When the photographing of the optical beacon is completed in this way, the user can use the photographed image displayed on the display unit 133 again to select the optical beacon in a comfortable posture.

It is also possible to record many real world landscapes here and there, and slowly view the photographed image to access the information of the desired optical beacon when time is available.

When the code of the optical beacon of the optical beacon installed on the monitor changes according to the contents of the monitor, the information of the optical beacon changes while the user confirms and selects the optical beacon. It is also possible that the user may find it difficult to use. On the other hand, according to the present embodiment, even in such a case, since the optical beacon can be easily recorded by pressing the capture button, it is possible to reliably acquire the information of the optical beacon. The potency is great.

FIG. 14 illustrates a state of an information processing service realized based on an optical signal emitted from the optical beacon 110 in the real world using the information terminal 130 with the ID recognition camera. However, in the illustrated example, three optical beacons 110-1 to 110-installed on a building.
It is assumed that 3 is captured at the same time. Further, FIG. 15 shows a processing procedure of an information processing service using the optical beacon 110 in the form of a flowchart. This information processing service is actually realized by the CPU in the computer unit 132 executing a predetermined program code.

Hereinafter, referring to FIG. 14 and FIG.
An information processing service using the optical beacon 110 will be described.

First, I installed in the information terminal 130
The D recognition camera 131 photographs a real world landscape including buildings and advertising signs, and the optical beacon 1 included in the photographed image.
Identification information is acquired from 10-1 to 110-3 (step S1). The identification information referred to here is the optical beacon 110.
The transmission data decoded from the blinking pattern of the light transmitted by and the position where the optical beacon is detected on the captured image (represented by the XY coordinate system).

Then, the display unit 133 displays and outputs the real world landscape image photographed by the ID recognition camera 131. At this time, the identification information acquired in the preceding step S1 is overlay-displayed on the place where the optical beacons 110-1 to 110-3 are detected or in the vicinity thereof (step S2). The display form of the identification information is not particularly limited to characters, icons, and the like. For example, a “balloon” may be used as shown in FIG.

Here, the user selects the capture button 13
When 3A is pressed (step S3), in response to this, the captured image of the real world by the ID recognition camera 131 is recorded in the image & ID storage unit 135 in association with the acquired identification information (step S4). .

If the capture button 133A is not pressed, the process returns to step S1 and, as described above, I
The D recognition camera 131 photographs the real world landscape, analyzes the identification information emitted from the optical beacon 110 included in the photographed image, and displays the analysis result on the photographed image on the display screen of the display unit 133.

By recording the photographed image and the ID (information received from the optical beacon 110) detected from the image in the image & ID storage unit 135 in this way, step S
After 5, after the user finished shooting the optical beacon, the user
By using the captured image displayed on the display unit 133 again, it is possible to select the optical beacon in a comfortable posture. It is also possible to record many real world landscapes here and there and slowly view the captured image to access the desired optical beacon information when time is available.

FIG. 16 shows an example of the configuration of the extended service screen presented on the display screen of the display unit 133.

A desired photographed image is taken out from the image & ID storage unit 135, and as shown in FIG. 16 (A), the photographed image and the identification information obtained from the optical beacons 110-1 to 110-3 included therein are stored. An overlay is displayed (step S5). On the screen shown, each optical beacon 110-1
At the display positions of 110 to 110-3, link destination information such as a URL represented by the identification information and anchors for starting other information processing services are embedded. In the example shown,
Each of the optical beacons 110-1 to 110-3 corresponds to a menu of record store A, restaurant B, and area information C, respectively.

The user can select a desired menu by moving up and down using the cursor button 134D on such a display screen. The menu in the selected state is switched to, for example, highlighted display (or reverse display). In FIG. 16A, the restaurant B is selected, and in FIG. 16B, the area store information C is selected by moving the cursor.

If the menu of interest cannot be found, the EXIT button 134E is pressed to terminate the processing of the real-world scenery image being displayed (step S6). In this case, returning to step S1, the ID recognition camera 131 photographs the next real world landscape, analyzes the identification information emitted from the optical beacon 110 included in the photographed image, and displays it on the display unit, as described above. 1
The analysis result is overlaid on the captured image on the display screen 33. Alternatively, the extended service itself may be terminated and the home screen (or top screen: not shown) may be returned by operating the EXIT button 134E.

On the other hand, by pressing the select button 134B instead of the EXIT button 134E, the selection of the menu (optical beacon) in the selected state is confirmed (step S7).

In response to this selecting operation, the connecting operation to the link destination indicated by the selected identification information is started (step S8). At this time, inside the information terminal 130, a connection is made to a connection destination whose identification information is represented by a URL, an IP address, or the like via the network interface unit 136, and a corresponding server on the computer network (for example, the service provider 135). Download information content from.

Since the process of downloading from the service provider 135 takes time, the display unit 133 displays and outputs a process screen as shown in FIG. 16C in order to visually indicate that the download is being executed. May be.

After that, when the download processing from the service provider 135 ends successfully, the received information is output on the screen of the display unit 133. FIG. 16D shows a screen configuration example at this time.

The display information provided by the service provider 135 is, for example, HTML (HyperText Markup Languag).
e) format. In the illustrated example, the area information includes restaurant information, event information, and surrounding areas.
There are menus or anchors linked to more detailed information such as maps. The user also performs information search processing on the computer network by further selecting a menu.

[Supplement] The present invention has been described in detail with reference to the specific embodiment. However, it is obvious that those skilled in the art can modify or substitute the embodiments without departing from the scope of the present invention. That is, the present invention has been disclosed in the form of exemplification, and the contents of this specification should not be construed in a limited manner. In order to determine the gist of the present invention, the section of the claims described at the beginning should be taken into consideration.

[0131]

As described above in detail, according to the present invention,
It is possible to provide an excellent information processing apparatus and information processing method, a storage medium, and a computer program capable of accepting a user's selection operation via a display screen and suitably activating a predetermined process.

Further, according to the present invention, an excellent information processing apparatus and information processing method, a storage medium, which can provide a suitable user operation environment on a photographed image of a real world landscape,
Also, a computer program can be provided.

Further, according to the present invention, it is possible to provide a suitable operating environment for selecting desired identification information from a photographed image of a real-world landscape in which visibility identification information is arranged. Processing device, information processing method, storage medium,
Also, a computer program can be provided.

The portable information terminal that realizes the present invention is provided with, for example, an “image & ID storage section” and a “capture button”. Then, when the user points the mobile terminal at the desired optical beacon and the display screen of the mobile terminal indicates that the optical beacon is recognized,
Press the button to record the image and ID (ie, the information received from the optical beacon) in the "image & ID storage". After that, when the captured image is displayed on the screen, the user can select the optical beacon in a comfortable posture.

[Brief description of drawings]

FIG. 1 is a data communication system 10 using an optical beacon.
It is the figure which showed the structure of.

FIG. 2 is a diagram schematically showing an internal configuration of a transmitter 20.

FIG. 3 is a diagram showing an internal configuration of a data processing unit 22 in the transmitter 20.

FIG. 4 is a diagram schematically showing a configuration of a light receiving block 51 of a receiver 50.

5 is a diagram schematically showing an internal configuration of one light receiving block 51 in the receiver 50. FIG.

FIG. 6 is a diagram schematically showing an internal configuration of a data processing unit 60 in the receiver 50.

FIG. 7 is a flowchart showing a data transmission processing procedure on the transmitter 20 side.

FIG. 8 is a data communication system 10 using optical signals.
FIG. 6 is a diagram showing a format of a data frame used for data transmission in FIG.

9 is a flowchart showing a detailed processing procedure of data frame generation in step S3 of the data transmission processing shown in FIG.

FIG. 10 is a flowchart showing a procedure for receiving processing on the receiver 50 side of data transmitted from the pattern display section 23 in the form of an optically distinguishable pattern.

FIG. 11 shows reception data provided in the working memory 54.
It is the figure which showed the structure of the buffer typically.

FIG. 12 is an augmented reality system 100 according to this embodiment.
It is the figure which showed the structure of.

FIG. 13 is a diagram showing an internal configuration of an information terminal with an ID recognition camera 130.

FIG. 14 is a diagram showing a state of an information processing service realized based on an optical signal emitted from an optical beacon 110 in the real world using an ID recognition camera-equipped information terminal 130.

FIG. 15 is a flowchart showing a processing procedure of an information processing service using the optical beacon 110.

16 is a diagram showing a configuration example of an extended service screen presented on the display screen of the display unit 133. FIG.

[Explanation of symbols] 10 ... Data communication system 20 ... Transmitter 21 ... LED 22 ... Data processing unit 23 ... Transmission data storage memory 24 ... Working memory 50 ... Receiver 51 ... Light receiving block 60 ... Data processing unit 61 ... Control unit 62 ... Memory for storing transfer data 63 ... Transfer data processing memory 64 ... Frame memory 70 ... Application Department 81 ... Light receiving element 82 ... Band pass filter 83 ... Phase lock loop 84 ... A / D converter 85 ... Memory for storing imaging data 86 ... Memory for storing binarized data 87 ... Memory for storing transfer data 88 ... Control unit 100 ... Augmented Reality System 110 ... Optical beacon 130 ... Information terminal with ID recognition camera 131 ... ID recognition camera 132 ... Computer section 133 ... Display unit 134 ... User input section 135 ... Image & ID storage section 136 ... Network interface section 150 ... Service Provider

   ─────────────────────────────────────────────────── ─── Continued front page    (72) Inventor Hidenori Koyanazu             1-14-10 Higashi Gotanda, Shinagawa-ku, Tokyo Stock             Ceremony company Sony Kihara Laboratory F-term (reference) 5B057 AA20 BA02 CA12 CA16 CE08                       DA07 DA16 DB02                 5E501 AA04 AB03 AB15 AC25 BA03                       BA05 BA08 BA09 BA17 CA04                       CA05 CB02 CB14 EB05 FA02                       FA05 FB44

Claims (16)

[Claims] 1. An information processing apparatus for starting a predetermined process in response to a user operation, comprising image input means for inputting a landscape image in the real world including visibility identification information, and an image input means included in the input image. Decoding means for decoding the visible identification information, storage means for storing the input image and the decoding result in association with each other, image display means for displaying the input image read from the storage means, and the image User input means for receiving a user selection operation for the visibility identification information appearing on the display screen of the display means, and service execution means for activating a processing service corresponding to the visibility identification information selected by the user. An information processing device characterized by: 2. The information processing apparatus according to claim 1, wherein the visibility identification information is displayed as one or more patterns capable of optically discriminating byte strings forming data. . 3. The information processing apparatus according to claim 1, wherein the image input means inputs a landscape image in the real world in response to a capture instruction. 4. The information processing apparatus according to claim 1, wherein said image input means inputs a landscape image in the real world in response to detection of visibility identifying information. 5. The information processing apparatus according to claim 1, wherein the storage means is capable of simultaneously storing two or more real world landscape images including visibility identification information. 6. The information processing apparatus according to claim 1, wherein the image display means overlay-displays the visibility identification information detected from the input image or the decoding result thereof on the input image. 7. The network connection means is further provided, and the service execution means performs information search via the network connection means based on an information resource identifier on the network represented by the visibility identification information. The information processing apparatus according to claim 1, which is characterized in that. 8. An information processing method for activating a predetermined process in response to a user operation, comprising an image input step of inputting a landscape image in the real world including visibility identification information, and an image input step included in the input image. A decoding step of decoding visible identification information, a storage step of storing the input image and the decoding result in association with each other, an image display step of displaying the input image stored in the storage step, and the image display A user input step of accepting a user selection operation for the visibility identification information appearing on the screen displayed in the step, and a service execution step of activating a processing service corresponding to the visibility identification information selected by the user. An information processing method comprising: 9. The information processing method according to claim 8, wherein the visibility identification information is displayed as one or more patterns capable of optically discriminating byte strings forming data. . 10. The information processing method according to claim 8, wherein in the image input step, a landscape image in the real world is input in response to a capture instruction. 11. The information processing method according to claim 8, wherein in the image input step, a landscape image in the real world is input in response to detection of the visibility identification information. 12. The information processing method according to claim 8, wherein in the storing step, two or more real world landscape images including visibility identification information can be stored at the same time. 13. The information processing method according to claim 8, wherein in the image displaying step, the visibility identification information detected from the input image or a decoding result thereof is overlay-displayed on the input image. 14. An information search is performed based on an information resource identifier on a network represented by the visibility identification information in the service execution step.
Information processing method described in. 15. A storage medium that physically stores computer software in a computer-readable format that is described to execute a process for activating a predetermined operation in response to a user operation on a computer system. ,
The computer software includes an image input step of inputting a landscape image in the real world including visibility identification information, a decoding step of decoding the visibility identification information included in the input image, and an input image, A storage step of associating and storing the decoding result, an image display step of displaying the input image stored in the storage step, and a user selection for visibility identification information appearing on the screen displayed in the image display step. A storage medium comprising: a user input step of accepting an operation; and a service execution step of activating a processing service corresponding to the visibility identification information selected by the user. 16. A computer program written in a computer-readable format so as to execute a process for activating a predetermined operation in response to a user operation on a computer system, the information including visibility identification information. An image input step of inputting a landscape image in the real world, a decoding step of decoding visibility identification information included in the input image, a storage step of storing the input image and the decoding result in association with each other, An image display step of displaying the input image stored in the storage step; a user input step of accepting a user selection operation for the visibility identification information appearing on the screen displayed in the image display step; A service execution step of activating a processing service corresponding to the visibility identification information. Computer programs and features.