JP2026037133A - system - Google Patents

Abstract

A system is provided.
A means for acquiring visual information of the outside world;
A means for acquiring auditory information from the outside world;
means for converting the visual information into text information;
means for converting the auditory information into text information;
a means for searching for related information on the web based on the converted text information;
means for displaying the retrieved information in an augmented reality (AR) format;
a means for collecting user interest-based data;
means for analyzing the collected data;
A system including:
[Selected Figure] Figure 1

Description

The technology disclosed herein relates to a system.

Patent document 1 discloses a persona chatbot control method executed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot's character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Japanese Patent Application Laid-Open No. 2022-180282

Traditional information acquisition systems required users to search for and acquire information themselves, making it difficult to effectively provide necessary information in real time. Furthermore, because information was not provided based on the user's interests, it was not possible to provide information that was optimal for each individual user. Furthermore, there was a lack of effective ways to utilize the acquired visual and auditory information, making it difficult to provide an interactive experience based on that information.

The present invention provides a system that acquires visual and auditory information from the outside world in real time and converts it into text information. Specifically, the system includes means for acquiring visual information from the outside world, means for acquiring auditory information from the outside world, means for converting the acquired visual information into text information, means for converting the acquired auditory information into text information, means for searching for related information on the web based on the converted text information, means for displaying the searched information in an augmented reality (AR) format, means for collecting data based on the user's interests, and means for analyzing the collected data. This system allows users to automatically acquire information from the outside world and obtain related information in real time. Furthermore, it makes it possible to efficiently provide information based on the user's interests and concerns, significantly improving the user experience.

"Visual information" refers to information about physical images or footage of the surroundings captured by an imaging device such as a camera.

"Auditory information" refers to information about surrounding sounds and conversations acquired by a sound collection device such as a microphone.

"Text information" is information expressed in character codes converted from visual and auditory information.

"Conversion means" refers to a device or software that analyzes visual or auditory information and converts it into corresponding text information.

A "search tool" is a device or software that locates relevant information on the web based on specific keywords or queries.

Augmented reality (AR) is a technology that overlays virtual information and images onto the real environment.

"Display means" refers to a device or software that presents information obtained and searched using AR technology in a visible form to the user.

"Interest data" is information that records the interests and concerns expressed by the user through their gaze, voice, etc.

"Collection Instrument" means any device or software used to record and collect interest data expressed by users.

"Analysis means" refers to a device or software that analyzes collected interest data to understand trends in users' interests and concerns.

1 is a conceptual diagram illustrating an example of the configuration of a data processing system according to a first embodiment. 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. FIG. 10 is a conceptual diagram illustrating an example of the configuration of a data processing system according to a second embodiment. FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. FIG. 10 is a conceptual diagram illustrating an example of the configuration of a data processing system according to a third embodiment. FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. 1 shows an emotion map onto which multiple emotions are mapped. 1 shows an emotion map onto which multiple emotions are mapped. FIG. 2 is a sequence diagram illustrating a processing flow of the data processing system according to the first embodiment. FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in application example 1. FIG. 10 is a sequence diagram showing the flow of processing of the data processing system in the second embodiment when an emotion engine is combined. FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined.

Below, an example of an embodiment of a system relating to the technology disclosed herein will be described with reference to the accompanying drawings.

First, let me explain the terminology used in the following explanation.

In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Furthermore, a processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose Computing on Graphics Processing Units), an APU (Accelerated Processing Unit), etc.

In the following embodiments, coded random access memory (RAM) is memory in which information is temporarily stored and is used as work memory by the processor.

In the following embodiments, the coded storage refers to one or more non-volatile storage devices that store various programs, parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

In the following embodiments, a communication I/F (Interface) with a symbol is an interface that includes a communication processor, an antenna, etc. The communication I/F controls communication between multiple computers. Examples of communication standards that can be applied to the communication I/F include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

In the following embodiments, "A and/or B" is synonymous with "at least one of A and B." In other words, "A and/or B" means that it may be just A, just B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and/or B" also applies when three or more things are expressed connected by "and/or."

[First embodiment]

Figure 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

As shown in FIG. 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

The data processing device 12 includes a computer 22, a database 24, and a communication I/F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and communication I/F 26 are also connected to the bus 34. The communication I/F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and/or a LAN (Local Area Network).

The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I/F 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

The reception device 38 is equipped with a touch panel 38A, a microphone 38B, etc., and receives user input. The touch panel 38A detects contact with an indicator (e.g., a pen or finger) to receive user input via the indicator. The microphone 38B detects the user's voice to receive user input via voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

The output device 40 is equipped with a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible by the user 20 (e.g., audio and/or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

The communication I/F 44 is connected to the network 54. The communication I/Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54.

Figure 2 shows an example of the main functions of the data processing device 12 and smart device 14.

As shown in FIG. 2, in the data processing device 12, specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

Storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

Next, we will explain the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following explanation, the data processing device 12 will be referred to as the "server" and the smart device 14 will be referred to as the "terminal."

This invention is a system that acquires visual and auditory information from the outside world in real time, searches for related information on the web based on that information, and displays it to the user in AR format. Specific embodiments of this system are described below.

First, the smart glasses, which serve as the device, acquire visual and auditory information from the outside world. Specifically, the camera installed in the smart glasses captures images, and the microphone records audio. This data is sent in real time to a processing unit within the device.

The device then uses OCR (optical character recognition) technology to convert the acquired visual information into text information. For example, it can recognize a cafe sign and convert it into text information such as "Cafe ABC." Similarly, auditory information is converted into text information using voice recognition technology. For example, it can recognize the conversation of a passerby and generate text information such as "This cafe is delicious."

The converted text information is sent from the device to the server. The server uses the received text information to search for related information on the web. Specifically, it uses a web search engine to obtain reviews, menus, location information, etc. for "Cafe ABC." The search results are then sent back to the device.

The device organizes the search results and displays them in AR format in the user's field of view. For example, reviews and popular menu items for "Cafe ABC" could be overlaid on the smartglasses' display. This allows users to access relevant information in real time through their smartglasses even while walking around town.

Furthermore, the device collects data on the user's interests. Specifically, when the user looks at specific information or issues a voice command such as "I'm interested in this menu item," that information is recorded. The collected interest data is sent to a server, which analyzes it to understand the user's interest trends. The results of this analysis are reflected in subsequent searches and information provision, making it possible to provide the most appropriate information tailored to the user's preferences.

For example, if a user is walking down the street and spots a sign that says "Cafe XYZ," the camera in the smart glasses will recognize the sign and convert it into text information as "Cafe XYZ" using OCR technology. Furthermore, if the user says, "I'm interested in the cake at this cafe," speech recognition technology will convert this statement into text. The server will perform a web search based on the keywords "Cafe XYZ" and "cake" to retrieve related reviews and menu information. This information will be sent to the device and displayed in AR format on the smart glasses' display. The user will be able to check reviews and photos of the cake on the spot.

In this manner, the present invention enables users to obtain and provide information in real time, improving the user experience. Furthermore, by collecting and analyzing data based on the user's interests, it becomes possible to provide even more personalized information.

The processing flow is explained below.

Step 1:

The device acquires visual and auditory information from the outside world. Specifically, the smart glasses' built-in camera captures video and a microphone records audio. This data is sent in real time to a processing unit within the device.

Step 2:

Visual information acquired by the device is converted into text information using OCR technology. For example, the device recognizes a sign that reads "Cafe ABC" and converts the image into text. Similarly, auditory information acquired is converted into text information using voice recognition technology. For example, a passerby's speech, "This cafe is delicious," is converted into text.

Step 3:

The device sends the converted text information to the server. Specifically, the text data generated by OCR or voice recognition is uploaded to the server via the Internet.

Step 4:

The server searches for related information on the web based on the text information it receives. The server uses a search engine to obtain reviews, menus, location information, etc. for "Cafe ABC." It is also possible to gather more information by using multiple search engines.

Step 5:

The server organizes the search results and selects the most relevant information. For example, it prioritizes reviews with high user ratings and information from official websites.

Step 6:

The server sends the organized information to the device. Specifically, it sends data summarizing related reviews and menu information to the device via the Internet.

Step 7:

The information received by the device is displayed in AR format. Reviews and popular menu items from Cafe ABC are overlaid on the smartglasses display.

Step 8:

When a user shows interest in specific information through their gaze or voice, the device records that data. For example, if a user stares intently at a particular review page or says, "I'm interested in this menu item."

Step 9:

The device sends the collected interest data to the server. The interest data includes the user's gaze information and voice commands.

Step 10:

The server analyzes the collected interest data to understand trends in user interests. The results of this analysis are reflected in future search results and information provision, enabling the server to provide users with the most appropriate information.

(Example 1)

Next, we will explain Example 1. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

In modern society, users are expected to obtain the information they need quickly and efficiently. It is particularly important to improve the user experience by obtaining and utilizing information in real time, even while on the move or while active. However, existing technology lacks sufficient methods for appropriately obtaining and providing information that interests users in real time, and the information acquisition process is cumbersome. Furthermore, there are still challenges in identifying users' interest trends and providing personalized information based on them. Against this background, there is a need for technology that can obtain and analyze users' visual and auditory information and provide relevant information in real time based on that information.

The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

In this invention, the server includes means for acquiring visual information from the outside world, means for acquiring auditory information from the outside world, means for converting the visual information into text information, means for converting the auditory information into text information, means for searching for related information on the web based on the converted text information, means for displaying the searched information in an augmented reality (AR) format, means for collecting data based on the user's interests in real time, and means for analyzing the collected data and reflecting it in subsequent search results. This enables the user to quickly and efficiently obtain the information they need while on the move or while active, and further enables the provision of personalized information based on the user's interests and preferences.

"Visual information from the external world" refers to visual information such as objects and letters that exist in the surrounding environment.

"External auditory information" refers to auditory information such as sounds and conversations that exist in the surrounding environment.

"Means for converting visual information into text information" refers to technology that converts acquired visual information into text data, specifically the use of OCR (optical character recognition) technology.

"Means for converting auditory information into text information" refers to technology that converts acquired auditory information into text data, specifically using voice recognition technology.

"Means for searching for related information on the web" refers to technology that searches for and retrieves related information on the Internet based on converted text information.

"Augmented reality (AR) display" refers to technology that displays acquired information overlaid on the real world, allowing users to simultaneously view images of the real world and additional information.

"Means for collecting data based on user interests in real time" refers to technology that records and collects trends in interest in real time when a user shows interest in specific information.

"Means of analyzing collected data and reflecting it in future search results" refers to technology that analyzes collected user interest data and personalizes the next information provided based on the results of that analysis.

This invention is a system that acquires visual and auditory information from the outside world in real time, searches for related information on the web based on that information, and displays it to the user in augmented reality (AR) format. Specific embodiments of this system are described below.

First, the smart glasses, which serve as the device, acquire visual and auditory information from the outside world. Specifically, the camera installed in the smart glasses captures images, and the microphone records audio. This data is sent in real time to a processing unit within the device.

The device then uses OCR (optical character recognition) technology to convert the acquired visual information into text information. For example, it can recognize a cafe sign and convert it into text information such as "Cafe ABC." Similarly, auditory information is converted into text information using voice recognition technology. For example, it can recognize the conversation of a passerby and generate text information such as "This cafe is delicious."

The converted text information is sent from the device to the server. The server uses the received text information to search for related information on the web. Specifically, it uses a web search engine to obtain reviews, menus, location information, etc. for "Cafe ABC." The search results are then sent back to the device.

The device organizes the search results and displays them in AR format in the user's field of view. For example, reviews and popular menu items for "Cafe ABC" could be overlaid on the smartglasses' display. This allows users to access relevant information in real time through their smartglasses even while walking around town.

Furthermore, the device collects user interest data. Specifically, when the user looks at specific information or issues a voice command such as "I'm interested in this menu item," that information is recorded. The collected interest data is sent to a server, which analyzes it to understand the user's interest trends. The results of this analysis are reflected in subsequent searches and information provision, making it possible to provide the most appropriate information tailored to the user's preferences.

As a concrete example, if a user is walking down the street and spots a sign for "Cafe XYZ," the camera in the smart glasses will recognize the sign and convert it into text information as "Cafe XYZ" using OCR technology. Furthermore, if the user says, "I'm interested in the cake at this cafe," speech recognition technology will convert this statement into text. The server will perform a web search based on the keywords "Cafe XYZ" and "cake" to retrieve related reviews and menu information. This information will be sent to the device and displayed in AR format on the smart glasses' display. The user will be able to check reviews and photos of the cake on the spot.

Example prompts are as follows:

"I saw a sign for Cafe XYZ and would like more information and reviews. I'd also like to know what people think of the cafe's cakes."

In this manner, the present invention enables users to obtain and provide information in real time, improving the user experience. Furthermore, by collecting and analyzing data based on the user's interests, it becomes possible to provide even more personalized information.

The flow of the identification process in Example 1 will be explained using Figure 11.

Step 1:

The device acquires visual and auditory information from the outside world.

Inputs include images of the outside world that appear in the user's field of vision and sounds from the environment.

Specifically, the camera built into the smart glasses captures video footage, and the microphone records audio.

As output, the captured video and audio data is sent to the processing unit inside the terminal.

Step 2:

The visual information acquired by the device is converted into text information.

The input is the video data acquired in step 1.

Specifically, the device uses OCR (optical character recognition) technology to analyze the characters in the video data and convert it into text data. For example, it would recognize a sign that reads "Cafe ABC" and convert it into text as "Cafe ABC."

The output is the converted text data.

Step 3:

The device converts the auditory information it acquires into text information.

The input is the audio data obtained in step 1.

Specifically, the device uses voice recognition technology to analyze voice data and convert it into text data. For example, a passerby's speech, "This cafe is delicious," can be converted into text data.

The output is the converted text data.

Step 4:

The device sends the converted text information to the server.

The input is the text data obtained in steps 2 and 3.

Specific operations include using a communications module within the device to send text data to a server via the Internet.

As output, text data arrives at the server.

Step 5:

The server searches for relevant information on the web based on the text information received.

The input is the text data that arrived at the server in step 4.

Specific operations include the server using a web search engine (e.g., Google (registered trademark) API) to search for reviews, menus, location information, etc. related to "Cafe ABC."

The output is the search results.

Step 6:

The server sends the search results to the device.

The input is the search results obtained in step 5.

Specifically, the server uses a communications module to send search results to the device. The data is sent in a lightweight data format such as JSON.

As output, the search results arrive on the terminal.

Step 7:

The device organizes the search results and displays them in AR format in the user's field of view.

The input is the search results that arrived on the terminal in step 6.

Specifically, the device will overlay reviews and popular menu items for "Cafe ABC" on the smartglasses' display. For example, it might show something like "Cafe ABC: Rating 4.5 stars, Popular menu item: Cheesecake."

As output, search results are displayed in AR format in the user's field of vision.

Step 8:

The device collects user interest data and sends it to the server.

Inputs include user behavior data such as gaze fixation and voice commands.

Specifically, the device detects the user's eye movements and records information when the user looks at specific information or issues a voice command such as "I'm interested in this menu item." This interest data is sent to a server via the Internet.

As output, the interest data arrives at the server.

Step 9:

The server analyzes the collected data and reflects it in future search results.

The input is the interest data that arrived at the server in step 8.

Specifically, the server analyzes interest data and models the user's interest trends. Based on this model, the server provides personalized results the next time the user searches or provides information.

The analysis results are obtained as output and will be reflected in subsequent search results.

(Application Example 1)

Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the smart device 14 will be referred to as the "terminal."

With conventional technology, users had limited means of obtaining real-time product information in physical stores, making it difficult to quickly obtain reviews and promotional information about specific products. Furthermore, there was a lack of provision of personalized information tailored to users' interests. This could result in a poor user experience and a decrease in purchasing motivation.

The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

In this invention, the server includes means for acquiring visual information from the outside world, means for acquiring auditory information from the outside world, means for converting the visual information into text information, means for converting the auditory information into text information, means for searching for related information on the web based on the converted text information, means for displaying the searched information in an augmented reality (AR) format, means for collecting data based on the user's interests, means for analyzing the collected data, and means for providing product information and reviews in AR format in real time while the user is walking. This allows users to quickly and intuitively obtain product information in a physical store, and the provision of personalized information can improve the shopping experience.

"Visual information from the outside world" refers to image information about the user's surrounding environment and objects, and is primarily obtained using imaging devices such as cameras.

"External auditory information" refers to the sounds and acoustic information around the user, and is primarily obtained by acoustic collection devices such as microphones.

"Means for converting visual information into text information" refers to means that use optical character recognition (OCR) technology to convert acquired visual information into text-format information such as letters and numbers.

"Means for converting auditory information into text information" refers to means that use voice recognition technology to convert acquired auditory information into text-format information such as letters or words.

"Means for searching for relevant information on the web" refers to means that include a search engine function for finding appropriate information on the Internet based on converted text information.

"Augmented reality (AR) display" refers to a technology that displays digital information overlaid on real-world images, and is a method that uses display devices such as smart glasses.

"Means for collecting data based on user interests" refers to means for collecting information about subjects that the user is interested in, such as the user's gaze or voice commands.

"Means for analyzing collected data" refers to means for analyzing collected user interest data and understanding user preferences and trends.

"A means of providing product information and reviews in real time in AR format while the user is walking" refers to a means of instantly displaying product information and user reviews in augmented reality format through a visual device while the user is moving around in a physical store.

"Means for detecting a user's gaze" refers to means that use gaze tracking devices or technology to detect and track the user's gaze direction and gaze point.

"Means for recognizing user voice commands" refers to means for using a voice recognition system to capture, understand, and process voice instructions given by the user.

This invention provides a system that allows users to obtain product information and reviews in real time while in a physical store. Specifically, the system uses smart glasses to obtain visual and auditory information from the outside world, generates text information based on that information, searches for related information on the web, and displays it in augmented reality (AR) format.

System Configuration

The system consists of the following main components:

Hardware

Smart glasses: Equipped with a camera, microphone, display, and processor. Worn by the user, they capture visual and auditory information from the outside world.

Cloud server: Responsible for large-volume data processing and storage.

Software

Optical character recognition (OCR) technology (e.g., Tesseract): converts captured visual information into text.

Speech recognition technology (e.g., Google Cloud Speech-to-Text): Converts captured auditory information into text format.

Web search engines (e.g., ElasticSearch (registered trademark)): Search for relevant information on the Internet based on converted text information.

Augmented reality (AR) technology (e.g., ARCore): Displays acquired information in AR format on the smart glasses display.

Eye tracking technology: Detects where the user is looking and identifies objects of interest.

Operation Overview

1. Acquisition and conversion of visual information: The smart glasses' camera captures products and signs in the store and converts them into text information using OCR technology.

2. Acquisition and conversion of auditory information: The smart glasses' microphone records the user's voice commands and converts them into text using voice recognition technology.

3. Search for related information: The converted text information is sent to a cloud server and related information is retrieved using a web search engine.

4. Displaying information in AR format: The acquired information is overlaid on the smart glasses display in AR format.

5. Collection and analysis of interest data: By recording the user's gaze and voice commands and analyzing them on a cloud server, we can understand the user's interests and concerns, and use this information to provide them in future visits.

Specific examples

For example, if a user is looking for a specific product in a store, the smart glasses' camera will recognize the product and convert it into text information such as "Product name XYZ." A web search will then be performed using the keywords "Product name XYZ review," and the retrieved review information will be displayed in AR on the smart glasses' display. Furthermore, if the user issues a voice command such as "I want to know the rating of this product," the voice will be converted into text, and additional review information will be searched for and displayed. If the user looks at specific information, that information will be recorded and used to personalize the next search results.

Example prompt

When a user is searching for a specific product in a store,

Capture product name XYZ with the smart glasses camera,

Use OCR to convert "Product Name XYZ" from camera footage into text information.

Then, do a web search using the keywords "Product Name XYZ Review"

Display the acquired review information in AR on the smart glasses display.

Furthermore, if the user issues a voice command such as "I want to know the rating of this product,"

Convert speech to text and re-search for additional review information.

When a user looks at specific information, record that information and use it to personalize your next search results.

The flow of the specific processing in Application Example 1 will be explained using Figure 12.

Step 1:

The camera in the smart glasses captures the environment in a physical store. The input is a video of the products in the user's field of view. The processor in the smart glasses captures this video data in real time and sends it to the processing unit as visual information. The output is the captured video data.

Step 2:

The processor in the smart glasses converts the captured video data into text information using OCR technology. The captured video data is used as input. OCR technology (e.g., Tesseract) extracts text information from product names and signs and converts it into text information such as "Product Name XYZ." The output is the converted text information.

Step 3:

The microphone in the smart glasses records voice commands given by the user. The input obtained here is the user's voice information. This voice information is converted into text information using voice recognition technology (e.g., Google Cloud Speech-to-Text). For example, the voice command "I would like to hear reviews of this product" is converted into text "I would like to hear reviews of this product." The output is the text information of the converted voice.

Step 4:

The device sends the converted visual and auditory information to a cloud server. The input is text information converted using OCR and speech recognition. The cloud server searches for related information on the web based on the received text information. Specifically, it uses a web search engine (e.g., Elasticsearch) to execute a search query such as "Product Name XYZ Reviews" to retrieve related reviews and menu information. The output is related information as search results.

Step 5:

The cloud server organizes the retrieved relevant information and sends it to the smart glasses. The relevant information obtained through the search is used as input. First, the relevant information is formalized and converted into a format suitable for the smart glasses' display. The output is data formatted for AR display.

Step 6:

The smart glasses display displays the acquired information in an AR overlay format. The input here is formatted data sent from the cloud server. The smart glasses display overlays product reviews and menu information in the user's field of view. The output is AR display information overlaid on the physical store environment.

Step 7:

The eye-tracking function of smart glasses detects the user's gaze and records information that the user shows interest in. The input is data on the direction of gaze and point of view. Eye-tracking technology analyzes in real time what information the user is focusing on, and the results are recorded as data. The output is interest data.

Step 8:

The cloud server analyzes the collected user interest data. The input is interest data recorded using eye-tracking technology and voice commands. The server analyzes this data to understand the user's preferences and trends. This information is used for future searches and information provision. The output is analyzed data on the user's preferences and trends.

Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

This invention combines a system that acquires visual and auditory information from the outside world in real time, searches for related information on the web based on that information, and displays it to the user in AR format with an emotion engine that recognizes the user's emotions. Specific embodiments of this system are described below.

First, the smart glasses, which serve as the device, acquire visual and auditory information from the outside world. Specifically, the camera installed in the smart glasses captures images, and the microphone records audio. This data is sent in real time to a processing unit within the device.

The device then converts the acquired visual information into text using OCR technology. For example, it can recognize a cafe sign and convert it into text such as "Cafe ABC." Similarly, it converts auditory information into text using voice recognition technology. For example, it can recognize the conversation of a passerby and generate text such as "This cafe is delicious."

The captured video and audio data is then analyzed by the device's built-in emotion engine to determine the user's emotions. The emotion engine uses machine learning algorithms to identify emotions from the user's facial expressions and tone of voice. For example, it can recognize whether the user is happy, interested, or surprised.

The converted text information and recognized emotion data are sent from the device to the server. The server then searches for related information on the web based on the received text information. For example, based on the keyword "cafe ABC," reviews, menus, and location details can be retrieved using a search engine such as Google. Furthermore, by taking emotion data into consideration, it is possible to prioritize retrieving information that is likely to interest the user.

The search results are then sent back to the device, which then displays the retrieved information in AR format. For example, reviews and popular menu items for "Cafe ABC" are overlaid on the smartglasses display. This allows users to access relevant information in real time through their smartglasses even while walking around town.

The device also collects interest data based on the user's gaze and voice commands. For example, if a user stares intently at a particular piece of information or says something like "I'm interested in this cake," the device will record it. By combining this with emotional data recognized by the emotion engine, more detailed interest analysis becomes possible.

The collected interest and emotion data is sent to the server for analysis. The server uses this data to understand the user's interest trends and reflects this in the next information provided. For example, if the user shows a strong interest in cafes, information about cafes can be displayed preferentially from the next time onwards.

As a specific example, if a user is walking down the street and spots a sign for "Cafe XYZ," the camera on the smart glasses will recognize the sign and convert it into text information as "Cafe XYZ" using OCR technology. Furthermore, if the user says, "I'm interested in the cake at this cafe," speech recognition technology will convert the statement into text. At the same time, the emotion engine will recognize positive emotions from the user's interested facial expression and tone of voice. This information is sent to the server, which will search for detailed information about "Cafe XYZ" and prioritize information that is likely to interest the user. This information is sent to the device and displayed in AR format on the smart glasses' display. The user can then check reviews and photos of the cake on the spot.

This type of service allows users to obtain and provide information in real time, significantly improving the user experience. Furthermore, by utilizing emotional data, even more personalized information can be provided.

The processing flow is explained below.

Step 1:

The device acquires visual and auditory information from the outside world. Specifically, the smart glasses' built-in camera captures video and the microphone records audio. This data is stored in the device's memory in real time.

Step 2:

Visual information acquired by the device is converted into text information using OCR technology. For example, if a sign that reads "Cafe XYZ" is captured, the image data is analyzed and the text data "Cafe XYZ" is generated. Similarly, auditory information acquired is converted into text information using voice recognition technology. For example, a conversation such as "This cafe is delicious" is converted into text format.

Step 3:

The device uses an emotion engine to identify the user's emotions from visual and auditory information. It uses machine learning algorithms to analyze the user's facial expressions and tone of voice to recognize emotions such as joy, interest, and surprise. For example, if the user smiles at a cafe sign, the device will identify the emotion of joy.

Step 4:

The device sends the converted text information and identified emotion data to the server, which then uploads the data to the server via the Internet.

Step 5:

The server searches for related information on the web based on the text information it receives. For example, you can perform a Google search using the keyword "cafe XYZ" to retrieve reviews and menu information for that cafe.

Step 6:

The server organizes the search results and selects the most relevant information. It takes into account emotional data and prioritizes information that is likely to interest the user. For example, if the user is expressing happiness, it will prioritize positive reviews of that cafe.

Step 7:

The server sends the organized information to the device. The selected data is then transferred to the device via the Internet.

Step 8:

The information received by the device is displayed in AR format. Set it up so that reviews and popular menu items from "Cafe XYZ" are overlaid on the smart glasses display.

Step 9:

If a user shows interest in specific information through their gaze or voice command, the device will record that data. For example, if a user looks at a particular review or says, "I'm interested in this cake," that information will be recorded.

Step 10:

The device sends the collected interest and emotion data to a server. The data is transmitted in real time via the Internet.

Step 11:

The server analyzes the collected data to understand the user's interest trends and emotional patterns. Based on the results of this analysis, the server reflects this in future information provision, providing the user with the most appropriate information. For example, from the next search result onwards, information about the user's favourite cafes will be displayed first.

(Example 2)

Next, we will explain Example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

In modern society, there is an increasing need for users to obtain information in real time and make quick decisions. However, current technology does not adequately provide real-time information based on visual and auditory information, limiting the user experience. In particular, there is a lack of personalized information provision that takes into account the user's emotions and interests, which reduces the efficiency of information gathering and user satisfaction. For this reason, there is a demand for a system that can obtain visual and auditory information from the outside world in real time and provide information that reflects the user's emotions and interests.

The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

In this invention, the server includes means for acquiring visual information from the outside world, means for acquiring auditory information from the outside world, means for converting the visual information into text information, means for converting the auditory information into text information, means for searching for related information on the web based on the converted text information and emotional data, means for displaying the searched information in an augmented reality format, means for recognizing the user's emotions, means for collecting data based on the user's interests, and means for analyzing the collected data and emotional data. This enables users to efficiently obtain personalized information based on their emotions and interests in real time.

"Visual information" refers to video and image data obtained from the external environment.

"Auditory information" refers to sound and acoustic data obtained from the external environment.

"Text information" is character string data obtained by analyzing and converting visual and auditory information.

"Emotional data" is data obtained as a result of analyzing the user's emotional state.

"Relevant information on the web" is information that is searched using the Internet and is relevant to the user's requirements and context.

"Augmented reality" is a technology that displays digital information overlaid on visual information from the real world.

"Means for recognizing user emotions" refers to an algorithm that analyzes video and audio data and identifies the user's emotions.

"Data based on user interests" is data that indicates a user's interests by analyzing input information such as the user's gaze and voice commands.

"Means for analyzing collected data and emotional data" refers to an algorithm that analyzes collected data based on the user's interests and emotions and uses it to provide information the next time.

This invention combines a system that acquires visual and auditory information from the outside world in real time, searches for related information on the web based on that information, and displays it to the user in augmented reality format with an emotion engine that recognizes the user's emotions.

First, the smart glasses, which serve as the terminal, use a camera to acquire visual information from the outside world and a microphone to acquire auditory information from the outside world. The video and audio data acquired at this point is sent in real time to a processing unit within the terminal.

The device then converts the visual information into text using OCR technology. For example, it recognizes a cafe sign and converts it into text such as "Cafe ABC." Similarly, it converts auditory information into text using voice recognition technology. For example, it recognizes the conversation of a passerby and converts it into text such as "This cafe is delicious."

The device also uses the captured video and audio data to analyze the user's emotions using an emotion engine, which can identify emotions such as whether the user is happy, interested, or surprised. The emotion engine uses machine learning algorithms to recognize emotions from facial expressions and tone of voice.

The converted text information and recognized emotion data are sent to the server. The server uses the received text information to search for related information on the web. For example, based on the keyword "cafe ABC," it can use a search engine to retrieve reviews, menus, and location details. Furthermore, by taking emotion data into consideration, it can prioritize retrieving information that is likely to interest the user.

The search results are then sent back to the device, which then displays the retrieved information in augmented reality (AR) format. For example, reviews and popular menu items for "Cafe ABC" may be overlaid on the real-world view on the smart glasses display, allowing users to obtain relevant information in real time.

In addition, the device also collects interest data based on the user's gaze and voice commands. If the user stares intently at a particular piece of information or says something like "I'm interested in this cake," the device records it. By combining this with emotional data recognized by the emotion engine, more detailed interest analysis becomes possible.

The collected interest and emotion data is sent to the server, which uses this data to understand the user's interest trends and reflect them in the next information provided. For example, if the user shows a strong interest in cafes, information about cafes can be displayed preferentially from the next time onwards.

Detailed examples

If a user is walking down the street and spots a sign for "Cafe XYZ," the camera on the smart glasses will recognize the sign and convert it into text using OCR technology: "Cafe XYZ." Furthermore, if the user says, "I'm interested in the cakes at this cafe," the speech recognition technology will convert the statement into text. At the same time, the emotion engine will recognize positive emotions from the user's interested facial expression and tone of voice. This information is sent to the server, which searches for detailed information about "Cafe XYZ" and prioritizes information that is likely to interest the user. This information is then sent to the device and displayed in augmented reality (AR) on the smart glasses' display. Users can then instantly check reviews and photos of the cakes.

Example prompt:

"When a user searches for information about a cafe, please explain the detailed processing flow of the system in which smart glasses acquire visual and auditory data, analyze the user's emotions using an emotion engine, and display information in AR format."

The flow of the identification process in Example 2 will be explained using Figure 13.

Step 1:

The smart glasses that serve as the device collect visual and auditory information from the outside world.

Input: Video and audio data from the outside world.

Processing: The smart glasses' camera captures video and the microphone records audio.

Output: Captured video data and recorded audio data.

Specific operation: As the user walks around town, the smart glasses take photos of buildings and signs, and record surrounding conversations and environmental sounds.

Step 2:

Visual information acquired by the device is converted into text information using OCR technology, and auditory information is converted into text using voice recognition technology.

Input: Video and audio data.

Processing: Use OCR technology to extract text from video data, and use voice recognition technology to convert audio data into text.

Output: Text information.

Specific operation: The smart glasses capture a sign that reads "Cafe XYZ" and convert it into text information: "Cafe XYZ." Similarly, they recognize the speech of a passerby, "The cake at this cafe is delicious," and convert it into text.

Step 3:

The device uses video and audio data to analyze the user's emotions using an emotion engine.

Input: Video and audio data.

Processing: Machine learning algorithms are used to analyze facial expressions from video data and tone of voice from audio data to identify emotions.

Output: Emotion data.

Specific behavior: Analyzes the user's facial expression and tone of voice when they look at a cafe sign to determine whether they are interested or pleased.

Step 4:

The device sends the converted text information and emotion data to the server.

Input: Text information and emotion data.

Processing: Packetize the data and send it to the server.

Output: Data sent to the server.

Specific operation: The device sends the text information "Cafe XYZ" and emotion data indicating "interest" to the server.

Step 5:

The server searches for relevant information on the web based on the text information and emotion data received.

Input: Text information and emotion data.

Processing: Use a search engine to search for relevant information and prioritize information that is likely to interest the user, taking into account emotional data.

Output: Search results for related information.

Specific operation: The server searches for reviews and menu information about "Cafe XYZ" and selects information that is likely to interest the user.

Step 6:

The server will send the search results back to the device.

Input: Search results for related information.

Processing: The search results are compiled into packets and sent to the terminal.

Output: Data sent to the terminal.

Specific operation: The server sends data such as reviews and popular menu items for "Cafe XYZ" to the device.

Step 7:

The information acquired by the device is displayed to the user in augmented reality (AR) format.

Input: Search results for related information.

Processing: Using augmented reality (AR) technology, digital information is overlaid on real-world scenes.

Output: Information displayed in AR format.

Specific operation: Reviews of Cafe XYZ and photos of cakes are overlaid on the real world on the smart glasses display.

Step 8:

The device collects interest data based on the user's gaze and voice commands.

Input: User gaze data and voice commands.

Processing: Uses an eye gaze sensor to detect where the user is looking, and analyzes voice commands using voice recognition technology.

Output: Interest data.

Specific behavior: If a user stares at a photo of a cake for a long time or says, "I'm interested in this cake," the device will collect that data.

Step 9:

The device sends the collected interest and emotion data to the server, which analyzes it.

Input: Interest data and emotion data.

Processing: The data is packaged into packets and sent to a server, where it is analyzed using machine learning algorithms.

Output: Analysis results.

Specific operations: The collected data is sent to a server, which analyzes it to understand trends in user interests.

Step 10:

The server will use the analysis results to provide information from the next time onwards.

Input: Analysis results.

Processing: The analysis results will be recorded in a database and used for the next information provision.

Output: Preparing for next information submission.

Specific behavior: If the user shows a strong interest in the cafe, prepare to display information about the cafe preferentially from the next time onwards.

(Application Example 2)

Next, we will explain Application Example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the smart device 14 will be referred to as the "terminal."

In autonomous vehicles, passengers and drivers have limited means to effectively grasp information about the outside world and obtain specific information in real time. Furthermore, there is a lack of information provided that reflects the user's emotions and interests, which presents a challenge in terms of the user experience not being sufficiently improved.

The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring visual information from the outside world, means for acquiring auditory information from the outside world, and means for converting the visual information into text information. This makes it possible to acquire related information based on visual and auditory information in real time and provide personalized information that reflects the user's emotions and interests.

"Visual information from the outside world" refers to video and image data of the surroundings acquired using cameras and sensors installed on smart devices.

"External auditory information" refers to surrounding voice and sound data acquired using a microphone installed on a smart device.

"Means for converting visual information into text information" refers to optical character recognition (OCR) technology for converting video and image data into text information.

"Means for converting auditory information into text information" refers to speech recognition technology for converting voice data into text information.

"Means for searching for related information on the web" refers to the function of searching for information on the Internet using a server or search engine and obtaining the necessary data.

"Means of displaying in augmented reality (AR) format" refers to technology for displaying acquired information by overlaying it on real-world images.

"Means for collecting data based on user interests" refers to a function that infers a user's interests from their gaze, voice commands, etc., and collects data related to those interests.

"Means for analyzing collected data" refers to algorithms and methods for analyzing collected data and understanding user interests and behavioral patterns.

"Means for analyzing user emotions" refers to machine learning algorithms that analyze a user's facial expressions and tone of voice to determine their emotional state.

"Smart devices installed in self-driving vehicles" refers to electronic devices with functions such as cameras, microphones, and displays that are used inside self-driving vehicles.

This invention is a system for improving the user experience in autonomous vehicles. It uses smart devices to acquire visual and auditory information from the outside world in real time, and uses this information to search for and display relevant information. It also features the ability to analyze the user's emotions and provide information based on their interests.

First, the smart device, which serves as the terminal, acquires visual and auditory information from the outside world. Specifically, the camera installed on the smart device captures video and the microphone records audio. This data is sent in real time to a processing unit within the terminal.

The device then converts the acquired visual information into text using OCR technology. For example, if an autonomous vehicle recognizes a restaurant sign while driving through town, it will convert it into text such as "Restaurant ABC." Similarly, auditory information is converted into text using voice recognition technology. For example, if a passenger says, "I'm interested in the menu at this restaurant," this will be recorded as text.

Furthermore, the captured video and audio data is analyzed for the user's emotions by the device's built-in emotion engine. The emotion engine uses machine learning algorithms to identify emotions from the user's facial expressions and tone of voice. For example, it can recognize whether the user is interested, happy, or surprised.

The converted text information and recognized emotion data are sent from the device to a cloud server. The cloud server then searches for related information on the web based on the received text information. For example, based on the keyword "restaurant ABC," reviews, menus, and location details can be retrieved using a search engine. Furthermore, by taking emotion data into consideration, it is possible to prioritize retrieving information that is likely to interest the user.

The search results are then sent back to the device, which then displays the retrieved information in AR format. For example, reviews and popular menu items for "Restaurant ABC" are overlaid on the smart device's display. This allows users to access relevant information in real time even while on the move.

The device also collects interest data based on the user's gaze and voice commands. For example, if a user stares intently at a particular piece of information or says something like "this dish looks delicious," that data is recorded. By combining this with emotional data recognized by the emotion engine, more detailed interest analysis becomes possible.

The collected interest and emotion data is sent to a cloud server for analysis. The cloud server uses this data to understand the user's interest trends and reflects this in the next information provided. For example, if the user shows a strong interest in restaurants, information about restaurants can be displayed preferentially from the next time onwards.

As a concrete example

As a specific example, if a user spots a sign for "Restaurant XYZ" while driving around town, the camera on the smart device will recognize the sign and convert it into text information as "Restaurant XYZ" using OCR technology. Furthermore, if the user says, "I'm interested in the menu at this restaurant," speech recognition technology will convert this statement into text. At the same time, an emotion engine will recognize positive emotions from the user's interested facial expression and tone of voice. This information is sent to a cloud server, which then searches for detailed information about "Restaurant XYZ" and prioritizes information that is likely to interest the user. This information is then sent to the device and displayed in AR format on the smart device's display. The user can then check reviews and photos of the menu on the spot.

Example prompt

An example of a prompt is shown below.

"When a user finds a restaurant sign, the smart device identifies the information, converts it into text, and prioritizes displaying reviews and menu information for that restaurant based on the user's sentiment. For example, if a user says, 'I'm interested in the menu at this restaurant,' the device converts that speech into text and retrieves and displays relevant information in real time."

The flow of the specific processing in Application Example 2 will be explained using Figure 14.

Step 1:

The terminal, a smart device, acquires visual and auditory information from the outside world. As the user moves around town, the camera captures video of the surroundings and the microphone records audio from the surroundings. This data is sent to a processing unit in real time. The input is camera video data and audio data, and the output is raw data sent to the processing unit.

Step 2:

The device converts the acquired visual information into text using OCR technology. For example, if a smart device's camera captures a restaurant sign, the video data is converted into text, generating the text information "Restaurant XYZ." The input is video data, and the output is text data.

Step 3:

The device uses voice recognition technology to convert the acquired voice information into text information. For example, if a user says, "I'm interested in the menu at this restaurant," the voice data is converted into text information and related information is generated. The input is voice data, and the output is text data.

Step 4:

The device uses an emotion engine to analyze the user's emotions from the captured video and audio data. For example, it can recognize the emotion "interesting" from the user's facial expression and tone of voice. The input is video and audio data, and the output is emotional data.

Step 5:

The converted text information and recognized emotion data are sent from the device to a cloud server. The server searches for related information on the web based on the received text information. For example, based on the keyword "restaurant XYZ," it retrieves reviews and menu information. The input is text data and emotion data, and the output is related information data.

Step 6:

The cloud server takes emotional data into consideration and prioritizes obtaining information that is likely to interest the user, and then sends the results back to the device. For example, if the user is expressing positive emotions, it will prioritize sending highly rated restaurant reviews. The input is text data and emotional data, and the output is related information data.

Step 7:

The device displays the acquired information in AR format. For example, reviews and popular menu items for "Restaurant XYZ" are overlaid on the smart device's display, allowing the user to view them. The input is related information data, and the output is the displayed AR data.

Step 8:

The device continues to collect interest data based on the user's gaze and voice commands. For example, if the user stares at a particular piece of information or says, "I'm interested in this dish," that data is recorded. The input is gaze data and voice data, and the output is interest data.

Step 9:

The collected interest and emotion data is sent to a cloud server for detailed analysis. The server uses this data to understand the user's interest trends and reflects this in the next information provided. For example, if there is a strong interest in restaurants, information about restaurants will be displayed preferentially from the next time onwards. The input is interest data and emotion data, and the output is analysis data for the next information provided.

The specific processing unit 290 transmits the results of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the results of the specific processing. The microphone 38B acquires audio indicating the user input regarding the results of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

Data generation model 58 is what is known as generative AI (artificial intelligence). Examples of data generation model 58 include generative AI such as ChatGPT (registered trademark) (Internet search <URL: https://openai.com/blog/chatgpt>) and Gemini (registered trademark) (Internet search <URL: https://gemini.google.com/?hl=ja>). Data generation model 58 is obtained by performing deep learning on a neural network. A prompt containing an instruction is input to data generation model 58, and inference data such as audio data indicating speech, text data indicating text, and image data indicating an image is also input. Data generation model 58 performs inference on the input inference data in accordance with the instructions indicated by the prompt and outputs the inference results in the form of data such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and/or summarization.

In the above embodiment, an example was given in which the specific processing was performed by the data processing device 12, but the technology disclosed herein is not limited to this, and the specific processing may also be performed by the smart device 14.

[Second embodiment]

Figure 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

As shown in FIG. 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

The data processing device 12 includes a computer 22, a database 24, and a communication I/F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and communication I/F 26 are also connected to the bus 34. The communication I/F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and/or a LAN (Local Area Network).

The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I/F 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the user 20's surroundings (e.g., an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

The communication I/F 44 is connected to the network 54. The communication I/Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I/Fs 44 and 26 is carried out in a secure manner.

Figure 4 shows an example of the main functions of the data processing device 12 and smart glasses 214. As shown in Figure 4, in the data processing device 12, specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32.

The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as the specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

Storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

Next, we will explain the identification process performed by the identification processing unit 290 of the data processing device 12. In the following explanation, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

This invention is a system that acquires visual and auditory information from the outside world in real time, searches for related information on the web based on that information, and displays it to the user in AR format. Specific embodiments of this system are described below.

First, the smart glasses, which serve as the device, acquire visual and auditory information from the outside world. Specifically, the camera installed in the smart glasses captures images, and the microphone records audio. This data is sent in real time to a processing unit within the device.

The device then uses OCR (optical character recognition) technology to convert the acquired visual information into text information. For example, it can recognize a cafe sign and convert it into text information such as "Cafe ABC." Similarly, auditory information is converted into text information using voice recognition technology. For example, it can recognize the conversation of a passerby and generate text information such as "This cafe is delicious."

The converted text information is sent from the device to the server. The server uses the received text information to search for related information on the web. Specifically, it uses a web search engine to obtain reviews, menus, location information, etc. for "Cafe ABC." The search results are then sent back to the device.

The device organizes the search results and displays them in AR format in the user's field of view. For example, reviews and popular menu items for "Cafe ABC" could be overlaid on the smartglasses' display. This allows users to access relevant information in real time through their smartglasses even while walking around town.

Furthermore, the device collects data on the user's interests. Specifically, when the user looks at specific information or issues a voice command such as "I'm interested in this menu item," that information is recorded. The collected interest data is sent to a server, which analyzes it to understand the user's interest trends. The results of this analysis are reflected in subsequent searches and information provision, making it possible to provide the most appropriate information tailored to the user's preferences.

For example, if a user is walking down the street and spots a sign that says "Cafe XYZ," the camera in the smart glasses will recognize the sign and convert it into text information as "Cafe XYZ" using OCR technology. Furthermore, if the user says, "I'm interested in the cake at this cafe," speech recognition technology will convert this statement into text. The server will perform a web search based on the keywords "Cafe XYZ" and "cake" to retrieve related reviews and menu information. This information will be sent to the device and displayed in AR format on the smart glasses' display. The user will be able to check reviews and photos of the cake on the spot.

In this manner, the present invention enables users to obtain and provide information in real time, improving the user experience. Furthermore, by collecting and analyzing data based on the user's interests, it becomes possible to provide even more personalized information.

The processing flow is explained below.

Step 1:

The device acquires visual and auditory information from the outside world. Specifically, the smart glasses' built-in camera captures video and a microphone records audio. This data is sent in real time to a processing unit within the device.

Step 2:

Visual information acquired by the device is converted into text information using OCR technology. For example, the device recognizes a sign that reads "Cafe ABC" and converts the image into text. Similarly, auditory information acquired is converted into text information using voice recognition technology. For example, a passerby's speech, "This cafe is delicious," is converted into text.

Step 3:

The device sends the converted text information to the server. Specifically, the text data generated by OCR or voice recognition is uploaded to the server via the Internet.

Step 4:

The server searches for related information on the web based on the text information it receives. The server uses a search engine to obtain reviews, menus, location information, etc. for "Cafe ABC." It is also possible to gather more information by using multiple search engines.

Step 5:

The server organizes the search results and selects the most relevant information. For example, it prioritizes reviews with high user ratings and information from official websites.

Step 6:

The server sends the organized information to the device. Specifically, it sends data summarizing related reviews and menu information to the device via the Internet.

Step 7:

The information received by the device is displayed in AR format. Reviews and popular menu items from Cafe ABC are overlaid on the smartglasses display.

Step 8:

When a user shows interest in specific information through their gaze or voice, the device records that data. For example, if a user stares intently at a particular review page or says, "I'm interested in this menu item."

Step 9:

The device sends the collected interest data to the server. The interest data includes the user's gaze information and voice commands.

Step 10:

The server analyzes the collected interest data to understand trends in user interests. The results of this analysis are reflected in future search results and information provision, enabling the server to provide users with the most appropriate information.

(Example 1)

Next, we will explain Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

In modern society, users are expected to obtain the information they need quickly and efficiently. It is particularly important to improve the user experience by obtaining and utilizing information in real time, even while on the move or while active. However, existing technology lacks sufficient methods for appropriately obtaining and providing information that interests users in real time, and the information acquisition process is cumbersome. Furthermore, there are still challenges in identifying users' interest trends and providing personalized information based on them. Against this background, there is a need for technology that can obtain and analyze users' visual and auditory information and provide relevant information in real time based on that information.

The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

In this invention, the server includes means for acquiring visual information from the outside world, means for acquiring auditory information from the outside world, means for converting the visual information into text information, means for converting the auditory information into text information, means for searching for related information on the web based on the converted text information, means for displaying the searched information in an augmented reality (AR) format, means for collecting data based on the user's interests in real time, and means for analyzing the collected data and reflecting it in subsequent search results. This enables the user to quickly and efficiently obtain the information they need while on the move or while active, and further enables the provision of personalized information based on the user's interests and preferences.

"Visual information from the external world" refers to visual information such as objects and letters that exist in the surrounding environment.

"External auditory information" refers to auditory information such as sounds and conversations that exist in the surrounding environment.

"Means for converting visual information into text information" refers to technology that converts acquired visual information into text data, specifically the use of OCR (optical character recognition) technology.

"Means for converting auditory information into text information" refers to technology that converts acquired auditory information into text data, specifically using voice recognition technology.

"Means for searching for related information on the web" refers to technology that searches for and retrieves related information on the Internet based on converted text information.

"Augmented reality (AR) display" refers to technology that displays acquired information overlaid on the real world, allowing users to simultaneously view images of the real world and additional information.

"Means for collecting data based on user interests in real time" refers to technology that records and collects trends in interest in real time when a user shows interest in specific information.

"Means of analyzing collected data and reflecting it in future search results" refers to technology that analyzes collected user interest data and personalizes the next information provided based on the results of that analysis.

This invention is a system that acquires visual and auditory information from the outside world in real time, searches for related information on the web based on that information, and displays it to the user in augmented reality (AR) format. Specific embodiments of this system are described below.

First, the smart glasses, which serve as the device, acquire visual and auditory information from the outside world. Specifically, the camera installed in the smart glasses captures images, and the microphone records audio. This data is sent in real time to a processing unit within the device.

The device then uses OCR (optical character recognition) technology to convert the acquired visual information into text information. For example, it can recognize a cafe sign and convert it into text information such as "Cafe ABC." Similarly, auditory information is converted into text information using voice recognition technology. For example, it can recognize the conversation of a passerby and generate text information such as "This cafe is delicious."

The converted text information is sent from the device to the server. The server uses the received text information to search for related information on the web. Specifically, it uses a web search engine to obtain reviews, menus, location information, etc. for "Cafe ABC." The search results are then sent back to the device.

The device organizes the search results and displays them in AR format in the user's field of view. For example, reviews and popular menu items for "Cafe ABC" could be overlaid on the smartglasses' display. This allows users to access relevant information in real time through their smartglasses even while walking around town.

Furthermore, the device collects data on the user's interests. Specifically, when the user looks at specific information or issues a voice command such as "I'm interested in this menu item," that information is recorded. The collected interest data is sent to a server, which analyzes it to understand the user's interest trends. The results of this analysis are reflected in subsequent searches and information provision, making it possible to provide the most appropriate information tailored to the user's preferences.

As a concrete example, if a user is walking down the street and spots a sign for "Cafe XYZ," the camera in the smart glasses will recognize the sign and convert it into text information as "Cafe XYZ" using OCR technology. Furthermore, if the user says, "I'm interested in the cake at this cafe," speech recognition technology will convert the comment into text. The server will perform a web search based on the keywords "Cafe XYZ" and "cake" to retrieve related reviews and menu information. This information will be sent to the device and displayed in AR format on the smart glasses' display. The user will be able to check reviews and photos of the cake on the spot.

Example prompts are as follows:

"I saw a sign for Cafe XYZ and would like more information and reviews. I'd also like to know what people think of the cafe's cakes."

In this manner, the present invention enables users to obtain and provide information in real time, improving the user experience. Furthermore, by collecting and analyzing data based on the user's interests, it becomes possible to provide even more personalized information.

The flow of the identification process in Example 1 will be explained using Figure 11.

Step 1:

The device acquires visual and auditory information from the outside world.

Inputs include images of the outside world that appear in the user's field of vision and sounds from the environment.

Specifically, the camera built into the smart glasses captures video footage, and the microphone records audio.

As output, the captured video and audio data is sent to the processing unit inside the terminal.

Step 2:

The visual information acquired by the device is converted into text information.

The input is the video data acquired in step 1.

Specifically, the device uses OCR (optical character recognition) technology to analyze the characters in the video data and convert it into text data. For example, it would recognize a sign that reads "Cafe ABC" and convert it into text as "Cafe ABC."

The output is the converted text data.

Step 3:

The device converts the auditory information it acquires into text information.

The input is the audio data obtained in step 1.

Specifically, the device uses voice recognition technology to analyze voice data and convert it into text data. For example, a passerby's speech, "This cafe is delicious," can be converted into text data.

The output is the converted text data.

Step 4:

The device sends the converted text information to the server.

The input is the text data obtained in steps 2 and 3.

Specific operations include using a communications module within the device to send text data to a server via the Internet.

As output, text data arrives at the server.

Step 5:

The server searches for relevant information on the web based on the text information received.

The input is the text data that arrived at the server in step 4.

Specifically, the server uses a web search engine (e.g., Google's API) to search for reviews, menus, location information, etc. related to "Cafe ABC."

The output is the search results.

Step 6:

The server sends the search results to the device.

The input is the search results obtained in step 5.

Specifically, the server uses a communications module to send search results to the device. The data is sent in a lightweight data format such as JSON.

As output, the search results arrive on the terminal.

Step 7:

The device organizes the search results and displays them in AR format in the user's field of view.

The input is the search results that arrived on the terminal in step 6.

Specifically, the device will overlay reviews and popular menu items for "Cafe ABC" on the smartglasses' display. For example, it might show something like "Cafe ABC: Rating 4.5 stars, Popular menu item: Cheesecake."

As output, search results are displayed in AR format in the user's field of vision.

Step 8:

The device collects user interest data and sends it to the server.

Inputs include user behavior data such as gaze fixation and voice commands.

Specifically, the device detects the user's eye movements and records information when the user looks at specific information or issues a voice command such as "I'm interested in this menu item." This interest data is sent to a server via the Internet.

As output, the interest data arrives at the server.

Step 9:

The server analyzes the collected data and reflects it in future search results.

The input is the interest data that arrived at the server in step 8.

Specifically, the server analyzes interest data and models the user's interest trends. Based on this model, the server provides personalized results the next time the user searches or provides information.

The analysis results are obtained as output and will be reflected in subsequent search results.

(Application Example 1)

Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

With conventional technology, users had limited means of obtaining real-time product information in physical stores, making it difficult to quickly obtain reviews and promotional information about specific products. Furthermore, there was a lack of provision of personalized information tailored to users' interests. This could result in a poor user experience and a decrease in purchasing motivation.

The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

In this invention, the server includes means for acquiring visual information from the outside world, means for acquiring auditory information from the outside world, means for converting the visual information into text information, means for converting the auditory information into text information, means for searching for related information on the web based on the converted text information, means for displaying the searched information in an augmented reality (AR) format, means for collecting data based on the user's interests, means for analyzing the collected data, and means for providing product information and reviews in AR format in real time while the user is walking. This allows users to quickly and intuitively obtain product information in a physical store, and the provision of personalized information can improve the shopping experience.

"Visual information from the outside world" refers to image information about the user's surrounding environment and objects, and is primarily obtained using imaging devices such as cameras.

"External auditory information" refers to the sounds and acoustic information around the user, and is primarily obtained by acoustic collection devices such as microphones.

"Means for converting visual information into text information" refers to means that use optical character recognition (OCR) technology to convert acquired visual information into text-format information such as letters and numbers.

"Means for converting auditory information into text information" refers to means that use voice recognition technology to convert acquired auditory information into text-format information such as letters or words.

"Means for searching for relevant information on the web" refers to means that include a search engine function for finding appropriate information on the Internet based on converted text information.

"Augmented reality (AR) display" refers to a technology that displays digital information overlaid on real-world images, and is a method that uses display devices such as smart glasses.

"Means for collecting data based on user interests" refers to means for collecting information about subjects that the user is interested in, such as the user's gaze or voice commands.

"Means for analyzing collected data" refers to means for analyzing collected user interest data and understanding user preferences and trends.

"A means of providing product information and reviews in real time in AR format while the user is walking" refers to a means of instantly displaying product information and user reviews in augmented reality format through a visual device while the user is moving around in a physical store.

"Means for detecting a user's gaze" refers to means that use gaze tracking devices or technology to detect and track the user's gaze direction and gaze point.

"Means for recognizing user voice commands" refers to means for using a voice recognition system to capture, understand, and process voice instructions given by the user.

This invention provides a system that allows users to obtain product information and reviews in real time while in a physical store. Specifically, the system uses smart glasses to obtain visual and auditory information from the outside world, generates text information based on that information, searches for related information on the web, and displays it in augmented reality (AR) format.

System Configuration

The system consists of the following main components:

Hardware

Smart glasses: Equipped with a camera, microphone, display, and processor. Worn by the user, they capture visual and auditory information from the outside world.

Cloud server: Responsible for large-volume data processing and storage.

Software

Optical character recognition (OCR) technology (e.g., Tesseract): converts captured visual information into text.

Speech recognition technology (e.g., Google Cloud Speech-to-Text): Converts captured auditory information into text format.

Web search engines (e.g., Elasticsearch): Search for relevant information on the Internet based on converted text information.

Augmented reality (AR) technology (e.g., ARCore): Displays acquired information in AR format on the smart glasses display.

Eye tracking technology: Detects where the user is looking and identifies objects of interest.

Operation Overview

1. Acquisition and conversion of visual information: The smart glasses' camera captures products and signs in the store and converts them into text information using OCR technology.

2. Acquiring and converting auditory information: The smart glasses' microphone records the user's voice commands and converts them into text using voice recognition technology.

3. Search for related information: The converted text information is sent to a cloud server and related information is retrieved using a web search engine.

4. Displaying information in AR format: The acquired information is overlaid on the smart glasses display in AR format.

5. Collection and analysis of interest data: By recording the user's gaze and voice commands and analyzing them on a cloud server, we can understand the user's interests and concerns, and use this information to provide them in future visits.

Specific examples

For example, if a user is looking for a specific product in a store, the smart glasses' camera will recognize the product and convert it into text information such as "Product name XYZ." A web search will then be performed using the keywords "Product name XYZ review," and the retrieved review information will be displayed in AR on the smart glasses' display. Furthermore, if the user issues a voice command such as "I want to know the rating of this product," the voice will be converted into text, and additional review information will be searched for and displayed. If the user looks at specific information, that information will be recorded and used to personalize the next search results.

Example prompt

When a user is searching for a specific product in a store,

Capture product name XYZ with the smart glasses camera,

Use OCR to convert "Product Name XYZ" from camera footage into text information.

Then, do a web search using the keywords "Product Name XYZ Review"

Display the acquired review information in AR on the smart glasses display.

Furthermore, if the user issues a voice command such as "I want to know the rating of this product,"

Convert speech to text and re-search for additional review information.

When a user looks at specific information, record that information and use it to personalize your next search results.

The flow of the identification process in Application Example 1 will be explained using Figure 12.

Step 1:

The camera in the smart glasses captures the environment in a physical store. The input is a video of the products in the user's field of view. The processor in the smart glasses captures this video data in real time and sends it to the processing unit as visual information. The output is the captured video data.

Step 2:

The processor in the smart glasses converts the captured video data into text information using OCR technology. The captured video data is used as input. OCR technology (e.g., Tesseract) extracts text information from product names and signs and converts it into text information such as "Product Name XYZ." The output is the converted text information.

Step 3:

The microphone in the smart glasses records voice commands given by the user. The input obtained here is the user's voice information. This voice information is converted into text information using voice recognition technology (e.g., Google Cloud Speech-to-Text). For example, the voice command "I would like to hear reviews of this product" is converted into text "I would like to hear reviews of this product." The output is the text information of the converted voice.

Step 4:

The device sends the converted visual and auditory information to a cloud server. The input is text information converted using OCR and speech recognition. The cloud server searches for related information on the web based on the received text information. Specifically, it uses a web search engine (e.g., Elasticsearch) to execute a search query such as "Product Name XYZ Reviews" to retrieve related reviews and menu information. The output is related information as search results.

Step 5:

The cloud server organizes the retrieved relevant information and sends it to the smart glasses. The relevant information obtained through the search is used as input. First, the relevant information is formalized and converted into a format suitable for the smart glasses' display. The output is data formatted for AR display.

Step 6:

The smart glasses display displays the acquired information in an AR overlay format. The input here is formatted data sent from the cloud server. The smart glasses display overlays product reviews and menu information in the user's field of view. The output is AR display information overlaid on the physical store environment.

Step 7:

The eye-tracking function of smart glasses detects the user's gaze and records information that shows interest. The input is data on the direction of gaze and point of view. Eye-tracking technology analyzes in real time what information the user is focusing on, and the results are recorded as data. The output is interest data.

Step 8:

The cloud server analyzes the collected user interest data. The input is interest data recorded using eye-tracking technology and voice commands. The server analyzes this data to understand the user's preferences and trends. This information is used for future searches and information provision. The output is analyzed data on the user's preferences and trends.

It is also possible to further combine an emotion engine that estimates the user's emotion. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

This invention combines a system that acquires visual and auditory information from the outside world in real time, searches for related information on the web based on that information, and displays it to the user in AR format with an emotion engine that recognizes the user's emotions. Specific embodiments of this system are described below.

First, the smart glasses, which serve as the device, acquire visual and auditory information from the outside world. Specifically, the camera installed in the smart glasses captures images, and the microphone records audio. This data is sent in real time to a processing unit within the device.

The device then converts the acquired visual information into text using OCR technology. For example, it can recognize a cafe sign and convert it into text such as "Cafe ABC." Similarly, it converts auditory information into text using voice recognition technology. For example, it can recognize the conversation of a passerby and generate text such as "This cafe is delicious."

The captured video and audio data is then analyzed by the device's built-in emotion engine to determine the user's emotions. The emotion engine uses machine learning algorithms to identify emotions from the user's facial expressions and tone of voice. For example, it can recognize whether the user is happy, interested, or surprised.

The converted text information and recognized emotion data are sent from the device to the server. The server then searches for related information on the web based on the received text information. For example, based on the keyword "cafe ABC," reviews, menus, and location details can be retrieved using a search engine such as Google. Furthermore, by taking emotion data into consideration, it is possible to prioritize retrieving information that is likely to interest the user.

The search results are then sent back to the device, which then displays the retrieved information in AR format. For example, reviews and popular menu items for "Cafe ABC" are overlaid on the smartglasses display. This allows users to access relevant information in real time through their smartglasses even while walking around town.

The device also collects interest data based on the user's gaze and voice commands. For example, if a user stares intently at a particular piece of information or says something like "I'm interested in this cake," the device will record it. By combining this with emotional data recognized by the emotion engine, more detailed interest analysis becomes possible.

The collected interest and emotion data is sent to the server for analysis. The server uses this data to understand the user's interest trends and reflects this in the next information provided. For example, if the user shows a strong interest in cafes, information about cafes can be displayed preferentially from the next time onwards.

As a specific example, if a user is walking down the street and spots a sign for "Cafe XYZ," the camera on the smart glasses will recognize the sign and convert it into text information as "Cafe XYZ" using OCR technology. Furthermore, if the user says, "I'm interested in the cake at this cafe," speech recognition technology will convert the statement into text. At the same time, the emotion engine will recognize positive emotions from the user's interested facial expression and tone of voice. This information is sent to the server, which will search for detailed information about "Cafe XYZ" and prioritize information that is likely to interest the user. This information is sent to the device and displayed in AR format on the smart glasses' display. The user can then check reviews and photos of the cake on the spot.

This type of service allows users to obtain and provide information in real time, significantly improving the user experience. Furthermore, by utilizing emotional data, even more personalized information can be provided.

The processing flow is explained below.

Step 1:

The device acquires visual and auditory information from the outside world. Specifically, the smart glasses' built-in camera captures video and the microphone records audio. This data is stored in the device's memory in real time.

Step 2:

Visual information acquired by the device is converted into text information using OCR technology. For example, if a sign that reads "Cafe XYZ" is captured, the image data is analyzed and the text data "Cafe XYZ" is generated. Similarly, auditory information acquired is converted into text information using voice recognition technology. For example, a conversation such as "This cafe is delicious" is converted into text format.

Step 3:

The device uses an emotion engine to identify the user's emotions from visual and auditory information. It uses machine learning algorithms to analyze the user's facial expressions and tone of voice to recognize emotions such as joy, interest, and surprise. For example, if the user smiles at a cafe sign, the device will identify the emotion of joy.

Step 4:

The device sends the converted text information and identified emotion data to the server, which then uploads the data to the server via the Internet.

Step 5:

The server searches for related information on the web based on the text information it receives. For example, you can perform a Google search using the keyword "cafe XYZ" to retrieve reviews and menu information for that cafe.

Step 6:

The server organizes the search results and selects the most relevant information. It takes into account emotional data and prioritizes information that is likely to interest the user. For example, if the user is expressing happiness, it will prioritize positive reviews of that cafe.

Step 7:

The server sends the organized information to the device. The selected data is then transferred to the device via the Internet.

Step 8:

The information received by the device is displayed in AR format. Set it up so that reviews and popular menu items from "Cafe XYZ" are overlaid on the smart glasses display.

Step 9:

If a user shows interest in specific information through their gaze or voice command, the device will record that data. For example, if a user looks at a particular review or says, "I'm interested in this cake," that information will be recorded.

Step 10:

The device sends the collected interest and emotion data to a server. The data is transmitted in real time via the Internet.

Step 11:

The server analyzes the collected data to understand the user's interest trends and emotional patterns. Based on the results of this analysis, the server reflects this in future information provision, providing the user with the most appropriate information. For example, from the next search result onwards, information about the user's favourite cafes will be displayed first.

(Example 2)

Next, we will explain Example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

In modern society, there is an increasing need for users to obtain information in real time and make quick decisions. However, current technology does not adequately provide real-time information based on visual and auditory information, limiting the user experience. In particular, there is a lack of personalized information provision that takes into account the user's emotions and interests, which reduces the efficiency of information gathering and user satisfaction. For this reason, there is a demand for a system that can obtain visual and auditory information from the outside world in real time and provide information that reflects the user's emotions and interests.

The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

In this invention, the server includes means for acquiring visual information from the outside world, means for acquiring auditory information from the outside world, means for converting the visual information into text information, means for converting the auditory information into text information, means for searching for related information on the web based on the converted text information and emotional data, means for displaying the searched information in an augmented reality format, means for recognizing the user's emotions, means for collecting data based on the user's interests, and means for analyzing the collected data and emotional data. This enables users to efficiently obtain personalized information based on their emotions and interests in real time.

"Visual information" refers to video and image data obtained from the external environment.

"Auditory information" refers to sound and acoustic data obtained from the external environment.

"Text information" is character string data obtained by analyzing and converting visual and auditory information.

"Emotional data" is data obtained as a result of analyzing the user's emotional state.

"Relevant information on the web" is information that is searched using the Internet and is relevant to the user's requirements and context.

"Augmented reality" is a technology that displays digital information overlaid on visual information from the real world.

"Means for recognizing user emotions" refers to an algorithm that analyzes video and audio data and identifies the user's emotions.

"Data based on user interests" is data that indicates a user's interests by analyzing input information such as the user's gaze and voice commands.

"Means for analyzing collected data and emotional data" refers to an algorithm that analyzes collected data based on the user's interests and emotions and uses it to provide information the next time.

This invention combines a system that acquires visual and auditory information from the outside world in real time, searches for related information on the web based on that information, and displays it to the user in augmented reality format with an emotion engine that recognizes the user's emotions.

First, the smart glasses, which serve as the terminal, use a camera to acquire visual information from the outside world and a microphone to acquire auditory information from the outside world. The video and audio data acquired at this point is sent in real time to a processing unit within the terminal.

The device then converts the visual information into text using OCR technology. For example, it recognizes a cafe sign and converts it into text such as "Cafe ABC." Similarly, it converts auditory information into text using voice recognition technology. For example, it recognizes the conversation of a passerby and converts it into text such as "This cafe is delicious."

The device also uses the captured video and audio data to analyze the user's emotions using an emotion engine, which can identify emotions such as whether the user is happy, interested, or surprised. The emotion engine uses machine learning algorithms to recognize emotions from facial expressions and tone of voice.

The converted text information and recognized emotion data are sent to the server. The server uses the received text information to search for related information on the web. For example, based on the keyword "cafe ABC," it can use a search engine to retrieve reviews, menus, and location details. Furthermore, by taking emotion data into consideration, it can prioritize retrieving information that is likely to interest the user.

The search results are then sent back to the device, which then displays the retrieved information in augmented reality (AR) format. For example, reviews and popular menu items for "Cafe ABC" may be overlaid on the real-world view on the smart glasses display, allowing users to obtain relevant information in real time.

In addition, the device also collects interest data based on the user's gaze and voice commands. If the user stares intently at a particular piece of information or says something like "I'm interested in this cake," the device records it. By combining this with emotional data recognized by the emotion engine, more detailed interest analysis becomes possible.

The collected interest and emotion data is sent to the server, which uses this data to understand the user's interest trends and reflect them in the next information provided. For example, if the user shows a strong interest in cafes, information about cafes can be displayed preferentially from the next time onwards.

Detailed examples

If a user is walking down the street and spots a sign for "Cafe XYZ," the camera on the smart glasses will recognize the sign and convert it into text using OCR technology: "Cafe XYZ." Furthermore, if the user says, "I'm interested in the cakes at this cafe," the speech recognition technology will convert the statement into text. At the same time, the emotion engine will recognize positive emotions from the user's interested facial expression and tone of voice. This information is sent to the server, which searches for detailed information about "Cafe XYZ" and prioritizes information that is likely to interest the user. This information is then sent to the device and displayed in augmented reality (AR) on the smart glasses' display. Users can then instantly check reviews and photos of the cakes.

Example prompt:

"When a user searches for information about a cafe, please explain the detailed processing flow of the system in which smart glasses acquire visual and auditory data, analyze the user's emotions using an emotion engine, and display information in AR format."

The flow of the identification process in Example 2 will be explained using Figure 13.

Step 1:

The smart glasses that serve as the device collect visual and auditory information from the outside world.

Input: Video and audio data from the outside world.

Processing: The smart glasses' camera captures video and the microphone records audio.

Output: Captured video data and recorded audio data.

Specific operation: As the user walks around town, the smart glasses take photos of buildings and signs, and record surrounding conversations and environmental sounds.

Step 2:

Visual information acquired by the device is converted into text information using OCR technology, and auditory information is converted into text using voice recognition technology.

Input: Video and audio data.

Processing: Use OCR technology to extract text from video data, and use voice recognition technology to convert audio data into text.

Output: Text information.

Specific operation: The smart glasses capture a sign that reads "Cafe XYZ" and convert it into text information: "Cafe XYZ." Similarly, they recognize the speech of a passerby, "The cake at this cafe is delicious," and convert it into text.

Step 3:

The device uses video and audio data to analyze the user's emotions using an emotion engine.

Input: Video and audio data.

Processing: Machine learning algorithms are used to analyze facial expressions from video data and tone of voice from audio data to identify emotions.

Output: Emotion data.

Specific behavior: Analyzes the user's facial expression and tone of voice when they look at a cafe sign to determine whether they are interested or pleased.

Step 4:

The device sends the converted text information and emotion data to the server.

Input: Text information and emotion data.

Processing: Packetize the data and send it to the server.

Output: Data sent to the server.

Specific operation: The device sends the text information "Cafe XYZ" and emotion data indicating "interest" to the server.

Step 5:

The server searches for relevant information on the web based on the text information and emotion data received.

Input: Text information and emotion data.

Processing: Use a search engine to search for relevant information and prioritize information that is likely to interest the user, taking into account emotional data.

Output: Search results for related information.

Specific operation: The server searches for reviews and menu information about "Cafe XYZ" and selects information that is likely to interest the user.

Step 6:

The server will send the search results back to the device.

Input: Search results for related information.

Processing: The search results are compiled into packets and sent to the terminal.

Output: Data sent to the terminal.

Specific operation: The server sends data such as reviews and popular menu items for "Cafe XYZ" to the device.

Step 7:

The information acquired by the device is displayed to the user in augmented reality (AR) format.

Input: Search results for related information.

Processing: Using augmented reality (AR) technology, digital information is overlaid on real-world scenes.

Output: Information displayed in AR format.

Specific operation: Reviews of Cafe XYZ and photos of cakes are overlaid on the real world on the smart glasses display.

Step 8:

The device collects interest data based on the user's gaze and voice commands.

Input: User gaze data and voice commands.

Processing: Uses an eye gaze sensor to detect where the user is looking, and analyzes voice commands using voice recognition technology.

Output: Interest data.

Specific behavior: If a user stares at a photo of a cake for a long time or says, "I'm interested in this cake," the device will collect that data.

Step 9:

The device sends the collected interest and emotion data to the server, which analyzes it.

Input: Interest data and emotion data.

Processing: The data is packaged into packets and sent to a server, where it is analyzed using machine learning algorithms.

Output: Analysis results.

Specific operations: The collected data is sent to a server, which analyzes it to understand trends in user interests.

Step 10:

The server will use the analysis results to provide information from the next time onwards.

Input: Analysis results.

Processing: The analysis results will be recorded in a database and used for the next information provision.

Output: Preparing for next information submission.

Specific behavior: If the user shows a strong interest in the cafe, prepare to prioritize displaying information about the cafe from the next time onwards.

(Application Example 2)

Next, we will explain Application Example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

In autonomous vehicles, passengers and drivers have limited means to effectively grasp information about the outside world and obtain specific information in real time. Furthermore, there is a lack of information provided that reflects the user's emotions and interests, which presents a challenge in terms of the user experience not being sufficiently improved.

The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring visual information from the outside world, means for acquiring auditory information from the outside world, and means for converting the visual information into text information. This makes it possible to acquire related information based on visual and auditory information in real time and provide personalized information that reflects the user's emotions and interests.

"Visual information from the outside world" refers to video and image data of the surroundings acquired using cameras and sensors installed on smart devices.

"External auditory information" refers to surrounding voice and sound data acquired using a microphone installed on a smart device.

"Means for converting visual information into text information" refers to optical character recognition (OCR) technology for converting video and image data into text information.

"Means for converting auditory information into text information" refers to speech recognition technology for converting voice data into text information.

"Means for searching for related information on the web" refers to the function of searching for information on the Internet using a server or search engine and obtaining the necessary data.

"Means of displaying in augmented reality (AR) format" refers to technology for displaying acquired information by overlaying it on real-world images.

"Means for collecting data based on user interests" refers to a function that infers a user's interests from their gaze, voice commands, etc., and collects data related to those interests.

"Means for analyzing collected data" refers to algorithms and methods for analyzing collected data and understanding user interests and behavioral patterns.

"Means for analyzing user emotions" refers to machine learning algorithms that analyze a user's facial expressions and tone of voice to determine their emotional state.

"Smart devices installed in self-driving vehicles" refers to electronic devices with functions such as cameras, microphones, and displays that are used inside self-driving vehicles.

This invention is a system for improving the user experience in autonomous vehicles. It uses smart devices to acquire visual and auditory information from the outside world in real time, and uses this information to search for and display relevant information. It also features the ability to analyze the user's emotions and provide information based on their interests.

First, the smart device, which serves as the terminal, acquires visual and auditory information from the outside world. Specifically, the camera installed on the smart device captures video and the microphone records audio. This data is sent in real time to a processing unit within the terminal.

The device then converts the acquired visual information into text using OCR technology. For example, if an autonomous vehicle recognizes a restaurant sign while driving through town, it will convert it into text such as "Restaurant ABC." Similarly, auditory information is converted into text using voice recognition technology. For example, if a passenger says, "I'm interested in the menu at this restaurant," this will be recorded as text.

Furthermore, the captured video and audio data is analyzed for the user's emotions by the device's built-in emotion engine. The emotion engine uses machine learning algorithms to identify emotions from the user's facial expressions and tone of voice. For example, it can recognize whether the user is interested, happy, or surprised.

The converted text information and recognized emotion data are sent from the device to a cloud server. The cloud server then searches for related information on the web based on the received text information. For example, based on the keyword "restaurant ABC," reviews, menus, and location details can be retrieved using a search engine. Furthermore, by taking emotion data into consideration, it is possible to prioritize retrieving information that is likely to interest the user.

The search results are then sent back to the device, which then displays the retrieved information in AR format. For example, reviews and popular menu items for "Restaurant ABC" are overlaid on the smart device's display. This allows users to access relevant information in real time even while on the move.

The device also collects interest data based on the user's gaze and voice commands. For example, if a user stares intently at a particular piece of information or says something like "this dish looks delicious," that data is recorded. By combining this with emotional data recognized by the emotion engine, more detailed interest analysis becomes possible.

The collected interest and emotion data is sent to a cloud server for analysis. The cloud server uses this data to understand the user's interest trends and reflects this in the next information provided. For example, if the user shows a strong interest in restaurants, information about restaurants can be displayed preferentially from the next time onwards.

As a concrete example

As a specific example, if a user spots a sign for "Restaurant XYZ" while driving around town, the camera on the smart device will recognize the sign and convert it into text information as "Restaurant XYZ" using OCR technology. Furthermore, if the user says, "I'm interested in the menu at this restaurant," speech recognition technology will convert this statement into text. At the same time, an emotion engine will recognize positive emotions from the user's interested facial expression and tone of voice. This information is sent to a cloud server, which then searches for detailed information about "Restaurant XYZ" and prioritizes information that is likely to interest the user. This information is then sent to the device and displayed in AR format on the smart device's display. The user can then check reviews and photos of the menu on the spot.

Example prompt

An example of a prompt is shown below.

"When a user finds a restaurant sign, the smart device identifies the information, converts it into text, and prioritizes displaying reviews and menu information for that restaurant based on the user's sentiment. For example, if a user says, 'I'm interested in the menu at this restaurant,' the device converts that speech into text and retrieves and displays relevant information in real time."

The flow of the specific processing in Application Example 2 will be explained using Figure 14.

Step 1:

The terminal, a smart device, acquires visual and auditory information from the outside world. As the user moves around town, the camera captures video of the surroundings and the microphone records audio from the surroundings. This data is sent to a processing unit in real time. The input is camera video data and audio data, and the output is raw data sent to the processing unit.

Step 2:

The device converts the acquired visual information into text using OCR technology. For example, if a smart device's camera captures a restaurant sign, the video data is converted into text, generating the text information "Restaurant XYZ." The input is video data, and the output is text data.

Step 3:

The device converts the acquired voice information into text information using voice recognition technology. For example, if a user says, "I'm interested in the menu at this restaurant," the voice data is converted into text information and related information is generated. The input is voice data, and the output is text data.

Step 4:

The device uses an emotion engine to analyze the user's emotions from the captured video and audio data. For example, it can recognize the emotion "interesting" from the user's facial expression and tone of voice. The input is video and audio data, and the output is emotional data.

Step 5:

The converted text information and recognized emotion data are sent from the device to a cloud server. The server searches for related information on the web based on the received text information. For example, based on the keyword "restaurant XYZ," it retrieves reviews and menu information. The input is text data and emotion data, and the output is related information data.

Step 6:

The cloud server takes emotional data into consideration and prioritizes obtaining information that is likely to interest the user, and then sends the results back to the device. For example, if the user is expressing positive emotions, it will prioritize sending highly rated restaurant reviews. The input is text data and emotional data, and the output is related information data.

Step 7:

The device displays the acquired information in AR format. For example, reviews and popular menu items for "Restaurant XYZ" are overlaid on the smart device's display, allowing the user to view them. The input is related information data, and the output is the displayed AR data.

Step 8:

The device continues to collect interest data based on the user's gaze and voice commands. For example, if the user stares at a particular piece of information or says, "I'm interested in this dish," that data is recorded. The input is gaze data and voice data, and the output is interest data.

Step 9:

The collected interest and emotion data is sent to a cloud server for detailed analysis. The server uses this data to understand the user's interest trends and reflects this in the next information provided. For example, if there is a strong interest in restaurants, information about restaurants will be displayed preferentially from the next time onwards. The input is interest data and emotion data, and the output is analysis data for the next information provided.

The specific processing unit 290 transmits the results of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the results of the specific processing. The microphone 238 acquires audio indicating the user input regarding the results of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

Data generation model 58 is what is known as generative AI (artificial intelligence). Examples of data generation model 58 include generative AI such as ChatGPT (Internet search <URL: https://openai.com/blog/chatgpt>) and Gemini (Internet search <URL: https://gemini.google.com/?hl=ja>). Data generation model 58 is obtained by performing deep learning on a neural network. A prompt containing an instruction is input to data generation model 58, and inference data such as voice data indicating speech, text data indicating text, and image data indicating an image is also input. Data generation model 58 performs inference on the input inference data in accordance with the instructions indicated by the prompt, and outputs the inference results in the form of data such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and/or summarization.

In the above embodiment, an example was given in which the specific processing was performed by the data processing device 12, but the technology disclosed herein is not limited to this, and the specific processing may also be performed by the smart glasses 214.

[Third embodiment]

Figure 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

As shown in FIG. 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

The data processing device 12 includes a computer 22, a database 24, and a communication I/F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and communication I/F 26 are also connected to the bus 34. The communication I/F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and/or a LAN (Local Area Network).

The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I/F 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the user 20's surroundings (e.g., an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

The communication I/F 44 is connected to the network 54. The communication I/Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I/Fs 44 and 26 is carried out in a secure manner.

Figure 6 shows an example of the main functions of the data processing device 12 and headset terminal 314. As shown in Figure 6, in the data processing device 12, specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32.

The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as the specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

Storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

In the headset terminal 314, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

Next, we will explain the identification process performed by the identification processing unit 290 of the data processing device 12. In the following explanation, the data processing device 12 will be referred to as the "server" and the headset-type terminal 314 will be referred to as the "terminal."

This invention is a system that acquires visual and auditory information from the outside world in real time, searches for related information on the web based on that information, and displays it to the user in AR format. Specific embodiments of this system are described below.

First, the smart glasses, which serve as the device, acquire visual and auditory information from the outside world. Specifically, the camera installed in the smart glasses captures images, and the microphone records audio. This data is sent in real time to a processing unit within the device.

The device then uses OCR (optical character recognition) technology to convert the acquired visual information into text information. For example, it can recognize a cafe sign and convert it into text information such as "Cafe ABC." Similarly, auditory information is converted into text information using voice recognition technology. For example, it can recognize the conversation of a passerby and generate text information such as "This cafe is delicious."

The converted text information is sent from the device to the server. The server uses the received text information to search for related information on the web. Specifically, it uses a web search engine to obtain reviews, menus, location information, etc. for "Cafe ABC." The search results are then sent back to the device.

The device organizes the search results and displays them in AR format in the user's field of view. For example, reviews and popular menu items for "Cafe ABC" could be overlaid on the smartglasses' display. This allows users to access relevant information in real time through their smartglasses even while walking around town.

Furthermore, the device collects data on the user's interests. Specifically, when the user looks at specific information or issues a voice command such as "I'm interested in this menu item," that information is recorded. The collected interest data is sent to a server, which analyzes it to understand the user's interest trends. The results of this analysis are reflected in subsequent searches and information provision, making it possible to provide the most appropriate information tailored to the user's preferences.

For example, if a user is walking down the street and spots a sign that says "Cafe XYZ," the camera in the smart glasses will recognize the sign and convert it into text information as "Cafe XYZ" using OCR technology. Furthermore, if the user says, "I'm interested in the cake at this cafe," speech recognition technology will convert this statement into text. The server will perform a web search based on the keywords "Cafe XYZ" and "cake" to retrieve related reviews and menu information. This information will be sent to the device and displayed in AR format on the smart glasses' display. The user will be able to check reviews and photos of the cake on the spot.

In this manner, the present invention enables users to obtain and provide information in real time, improving the user experience. Furthermore, by collecting and analyzing data based on the user's interests, it becomes possible to provide even more personalized information.

The processing flow is explained below.

Step 1:

The device acquires visual and auditory information from the outside world. Specifically, the smart glasses' built-in camera captures video and a microphone records audio. This data is sent in real time to a processing unit within the device.

Step 2:

Visual information acquired by the device is converted into text information using OCR technology. For example, the device recognizes a sign that reads "Cafe ABC" and converts the image into text. Similarly, auditory information acquired is converted into text information using voice recognition technology. For example, a passerby's speech, "This cafe is delicious," is converted into text.

Step 3:

The device sends the converted text information to the server. Specifically, the text data generated by OCR or voice recognition is uploaded to the server via the Internet.

Step 4:

The server searches for related information on the web based on the text information it receives. The server uses a search engine to obtain reviews, menus, location information, etc. for "Cafe ABC." It is also possible to gather more information by using multiple search engines.

Step 5:

The server organizes the search results and selects the most relevant information. For example, it prioritizes reviews with high user ratings and information from official websites.

Step 6:

The server sends the organized information to the device. Specifically, it sends data summarizing related reviews and menu information to the device via the Internet.

Step 7:

The information received by the device is displayed in AR format. Reviews and popular menu items from Cafe ABC are overlaid on the smartglasses display.

Step 8:

When a user shows interest in specific information through their gaze or voice, the device records that data. For example, if a user stares intently at a particular review page or says, "I'm interested in this menu item."

Step 9:

The device sends the collected interest data to the server. The interest data includes the user's gaze information and voice commands.

Step 10:

The server analyzes the collected interest data to understand trends in user interests. The results of this analysis are reflected in future search results and information provision, enabling the server to provide users with the most appropriate information.

(Example 1)

Next, Example 1 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset-type terminal 314 will be referred to as the "terminal."

In modern society, users are expected to obtain the information they need quickly and efficiently. It is particularly important to improve the user experience by obtaining and utilizing information in real time, even while on the move or while active. However, existing technology lacks sufficient methods for appropriately obtaining and providing information that interests users in real time, and the information acquisition process is cumbersome. Furthermore, there are still challenges in identifying users' interest trends and providing personalized information based on them. Against this background, there is a need for technology that can obtain and analyze users' visual and auditory information and provide relevant information in real time based on that information.

The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

In this invention, the server includes means for acquiring visual information from the outside world, means for acquiring auditory information from the outside world, means for converting the visual information into text information, means for converting the auditory information into text information, means for searching for related information on the web based on the converted text information, means for displaying the searched information in an augmented reality (AR) format, means for collecting data based on the user's interests in real time, and means for analyzing the collected data and reflecting it in subsequent search results. This enables the user to quickly and efficiently obtain the information they need while on the move or while active, and further enables the provision of personalized information based on the user's interests and preferences.

"Visual information from the external world" refers to visual information such as objects and letters that exist in the surrounding environment.

"External auditory information" refers to auditory information such as sounds and conversations that exist in the surrounding environment.

"Means for converting visual information into text information" refers to technology that converts acquired visual information into text data, specifically the use of OCR (optical character recognition) technology.

"Means for converting auditory information into text information" refers to technology that converts acquired auditory information into text data, specifically using voice recognition technology.

"Means for searching for related information on the web" refers to technology that searches for and retrieves related information on the Internet based on converted text information.

"Augmented reality (AR) display" refers to technology that displays acquired information overlaid on the real world, allowing users to simultaneously view images of the real world and additional information.

"Means for collecting data based on user interests in real time" refers to technology that records and collects trends in interest in real time when a user shows interest in specific information.

"Means of analyzing collected data and reflecting it in future search results" refers to technology that analyzes collected user interest data and personalizes the next information provided based on the results of that analysis.

This invention is a system that acquires visual and auditory information from the outside world in real time, searches for related information on the web based on that information, and displays it to the user in augmented reality (AR) format. Specific embodiments of this system are described below.

First, the smart glasses, which serve as the device, acquire visual and auditory information from the outside world. Specifically, the camera installed in the smart glasses captures images, and the microphone records audio. This data is sent in real time to a processing unit within the device.

The device then uses OCR (optical character recognition) technology to convert the acquired visual information into text information. For example, it can recognize a cafe sign and convert it into text information such as "Cafe ABC." Similarly, auditory information is converted into text information using voice recognition technology. For example, it can recognize the conversation of a passerby and generate text information such as "This cafe is delicious."

The converted text information is sent from the device to the server. The server uses the received text information to search for related information on the web. Specifically, it uses a web search engine to obtain reviews, menus, location information, etc. for "Cafe ABC." The search results are then sent back to the device.

The device organizes the search results and displays them in AR format in the user's field of view. For example, reviews and popular menu items for "Cafe ABC" could be overlaid on the smartglasses' display. This allows users to access relevant information in real time through their smartglasses even while walking around town.

Furthermore, the device collects data on the user's interests. Specifically, when the user looks at specific information or issues a voice command such as "I'm interested in this menu item," that information is recorded. The collected interest data is sent to a server, which analyzes it to understand the user's interest trends. The results of this analysis are reflected in subsequent searches and information provision, making it possible to provide the most appropriate information tailored to the user's preferences.

As a concrete example, if a user is walking down the street and spots a sign for "Cafe XYZ," the camera in the smart glasses will recognize the sign and convert it into text information as "Cafe XYZ" using OCR technology. Furthermore, if the user says, "I'm interested in the cake at this cafe," speech recognition technology will convert the comment into text. The server will perform a web search based on the keywords "Cafe XYZ" and "cake" to retrieve related reviews and menu information. This information will be sent to the device and displayed in AR format on the smart glasses' display. The user will be able to check reviews and photos of the cake on the spot.

Example prompts are as follows:

"I saw a sign for Cafe XYZ and would like more information and reviews. I'd also like to know what people think of the cafe's cakes."

In this manner, the present invention enables users to obtain and provide information in real time, improving the user experience. Furthermore, by collecting and analyzing data based on the user's interests, it becomes possible to provide even more personalized information.

The flow of the identification process in Example 1 will be explained using Figure 11.

Step 1:

The device acquires visual and auditory information from the outside world.

Inputs include images of the outside world that appear in the user's field of vision and sounds from the environment.

Specifically, the camera built into the smart glasses captures video footage, and the microphone records audio.

As output, the captured video and audio data is sent to the processing unit inside the terminal.

Step 2:

The visual information acquired by the device is converted into text information.

The input is the video data acquired in step 1.

Specifically, the device uses OCR (optical character recognition) technology to analyze the characters in the video data and convert it into text data. For example, it would recognize a sign that reads "Cafe ABC" and convert it into text as "Cafe ABC."

The output is the converted text data.

Step 3:

The device converts the auditory information it acquires into text information.

The input is the audio data obtained in step 1.

Specifically, the device uses voice recognition technology to analyze voice data and convert it into text data. For example, a passerby's speech, "This cafe is delicious," can be converted into text data.

The output is the converted text data.

Step 4:

The device sends the converted text information to the server.

The input is the text data obtained in steps 2 and 3.

Specific operations include using a communications module within the device to send text data to a server via the Internet.

As output, text data arrives at the server.

Step 5:

The server searches for relevant information on the web based on the text information received.

The input is the text data that arrived at the server in step 4.

Specifically, the server uses a web search engine (e.g., Google's API) to search for reviews, menus, location information, etc. related to "Cafe ABC."

The output is the search results.

Step 6:

The server sends the search results to the device.

The input is the search results obtained in step 5.

Specifically, the server uses a communications module to send search results to the device. The data is sent in a lightweight data format such as JSON.

As output, the search results arrive on the terminal.

Step 7:

The device organizes the search results and displays them in AR format in the user's field of vision.

The input is the search results that arrived on the terminal in step 6.

Specifically, the device will overlay reviews and popular menu items for "Cafe ABC" on the smartglasses' display. For example, it might show something like "Cafe ABC: Rating 4.5 stars, Popular menu item: Cheesecake."

As output, search results are displayed in AR format in the user's field of vision.

Step 8:

The device collects user interest data and sends it to the server.

Inputs include user behavior data such as gaze fixation and voice commands.

Specifically, the device detects the user's eye movements and records information when the user looks at specific information or issues a voice command such as "I'm interested in this menu item." This interest data is sent to a server via the Internet.

As output, the interest data arrives at the server.

Step 9:

The server analyzes the collected data and reflects it in future search results.

As input, there is the interest data that arrived at the server in step 8.

Specifically, the server analyzes interest data and models the user's interest trends. Based on this model, the server provides personalized results the next time the user searches or provides information.

The analysis results are obtained as output and will be reflected in future search results.

(Application Example 1)

Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the headset-type terminal 314 will be referred to as the "terminal."

With conventional technology, users had limited means of obtaining real-time product information in physical stores, making it difficult to quickly obtain reviews and promotional information about specific products. Furthermore, there was a lack of provision of personalized information tailored to users' interests. This could result in a poor user experience and a decrease in purchasing motivation.

The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

In this invention, the server includes means for acquiring visual information from the outside world, means for acquiring auditory information from the outside world, means for converting the visual information into text information, means for converting the auditory information into text information, means for searching for related information on the web based on the converted text information, means for displaying the searched information in an augmented reality (AR) format, means for collecting data based on the user's interests, means for analyzing the collected data, and means for providing product information and reviews in AR format in real time while the user is walking. This allows users to quickly and intuitively obtain product information in a physical store, and the provision of personalized information can improve the shopping experience.

"Visual information from the outside world" refers to image information about the user's surrounding environment and objects, and is primarily obtained using imaging devices such as cameras.

"External auditory information" refers to the sounds and acoustic information around the user, and is primarily obtained by acoustic collection devices such as microphones.

"Means for converting visual information into text information" refers to means that use optical character recognition (OCR) technology to convert acquired visual information into text-format information such as letters and numbers.

"Means for converting auditory information into text information" refers to means that use voice recognition technology to convert acquired auditory information into text-format information such as letters or words.

"Means for searching for relevant information on the web" refers to means that include a search engine function for finding appropriate information on the Internet based on converted text information.

"Augmented reality (AR) display" refers to a technology that displays digital information overlaid on real-world images, and is a method that uses display devices such as smart glasses.

"Means for collecting data based on user interests" refers to means for collecting information about subjects that the user is interested in, such as the user's gaze or voice commands.

"Means for analyzing collected data" refers to means for analyzing collected user interest data and understanding user preferences and trends.

"A means of providing product information and reviews in real time in AR format while the user is walking" refers to a means of instantly displaying product information and user reviews in augmented reality format through a visual device while the user is moving around in a physical store.

"Means for detecting a user's gaze" refers to means that use gaze tracking devices or technology to detect and track the user's gaze direction and gaze point.

"Means for recognizing user voice commands" refers to means for using a voice recognition system to capture, understand, and process voice instructions given by the user.

This invention provides a system that allows users to obtain product information and reviews in real time while in a physical store. Specifically, the system uses smart glasses to obtain visual and auditory information from the outside world, generates text information based on that information, searches for related information on the web, and displays it in augmented reality (AR) format.

System Configuration

The system consists of the following main components:

Hardware

Smart glasses: Equipped with a camera, microphone, display, and processor. Worn by the user, they capture visual and auditory information from the outside world.

Cloud server: Responsible for large-volume data processing and storage.

Software

Optical character recognition (OCR) technology (e.g., Tesseract): converts captured visual information into text.

Speech recognition technology (e.g., Google Cloud Speech-to-Text): Converts captured auditory information into text format.

Web search engines (e.g., Elasticsearch): Search for relevant information on the Internet based on converted text information.

Augmented reality (AR) technology (e.g., ARCore): Displays acquired information in AR format on the smart glasses display.

Eye tracking technology: Detects where the user is looking and identifies objects of interest.

Operation Overview

1. Acquisition and conversion of visual information: The smart glasses' camera captures products and signs in the store and converts them into text information using OCR technology.

2. Acquisition and conversion of auditory information: The smart glasses' microphone records the user's voice commands and converts them into text using voice recognition technology.

3. Search for related information: The converted text information is sent to a cloud server and related information is retrieved using a web search engine.

4. Displaying information in AR format: The acquired information is overlaid on the smart glasses display in AR format.

5. Collection and analysis of interest data: By recording the user's gaze and voice commands and analyzing them on a cloud server, we can understand the user's interests and concerns, and use this information to provide them in future visits.

Specific examples

For example, if a user is looking for a specific product in a store, the smart glasses' camera will recognize the product and convert it into text information such as "Product name XYZ." A web search will then be performed using the keywords "Product name XYZ review," and the retrieved review information will be displayed in AR on the smart glasses' display. Furthermore, if the user issues a voice command such as "I want to know the rating of this product," the voice will be converted into text, and additional review information will be searched for and displayed. If the user looks at specific information, that information will be recorded and used to personalize the next search results.

Example prompt

When a user is searching for a specific product in a store,

Capture product name XYZ with the smart glasses camera,

Use OCR to convert "Product Name XYZ" from camera footage into text information.

Then, do a web search using the keywords "Product Name XYZ Review"

Display the acquired review information in AR on the smart glasses display.

Furthermore, if the user issues a voice command such as "I want to know the rating of this product,"

Convert speech to text and re-search for additional review information.

When a user looks at specific information, record that information and use it to personalize your next search results.

The flow of the specific processing in Application Example 1 will be explained using Figure 12.

Step 1:

The camera in the smart glasses captures the environment in a physical store. The input is a video of the products in the user's field of view. The processor in the smart glasses captures this video data in real time and sends it to the processing unit as visual information. The output is the captured video data.

Step 2:

The processor in the smart glasses converts the captured video data into text information using OCR technology. The captured video data is used as input. OCR technology (e.g., Tesseract) extracts text information from product names and signs and converts it into text information such as "Product Name XYZ." The output is the converted text information.

Step 3:

The microphone in the smart glasses records voice commands given by the user. The input obtained here is the user's voice information. This voice information is converted into text information using voice recognition technology (e.g., Google Cloud Speech-to-Text). For example, the voice command "I would like to hear reviews of this product" is converted into text "I would like to hear reviews of this product." The output is the text information of the converted voice.

Step 4:

The device sends the converted visual and auditory information to a cloud server. The input is text information converted using OCR and speech recognition. The cloud server searches for related information on the web based on the received text information. Specifically, it uses a web search engine (e.g., Elasticsearch) to execute a search query such as "Product Name XYZ Reviews" to retrieve related reviews and menu information. The output is related information as search results.

Step 5:

The cloud server organizes the retrieved relevant information and sends it to the smart glasses. The relevant information obtained through the search is used as input. First, the relevant information is formalized and converted into a format suitable for the smart glasses' display. The output is data formatted for AR display.

Step 6:

The smart glasses display displays the acquired information in an AR overlay format. The input here is formatted data sent from the cloud server. The smart glasses display overlays product reviews and menu information in the user's field of view. The output is AR display information overlaid on the physical store environment.

Step 7:

The eye-tracking function of smart glasses detects the user's gaze and records information that the user shows interest in. The input is data on the direction of gaze and point of view. Eye-tracking technology analyzes in real time what information the user is focusing on, and the results are recorded as data. The output is interest data.

Step 8:

The cloud server analyzes the collected user interest data. The input is interest data recorded using eye-tracking technology and voice commands. The server analyzes this data to understand the user's preferences and trends. This information is used for future searches and information provision. The output is analyzed data on the user's preferences and trends.

It is also possible to further combine an emotion engine that estimates the user's emotion. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

This invention combines a system that acquires visual and auditory information from the outside world in real time, searches for related information on the web based on that information, and displays it to the user in AR format with an emotion engine that recognizes the user's emotions. Specific embodiments of this system are described below.

First, the smart glasses, which serve as the device, acquire visual and auditory information from the outside world. Specifically, the camera installed in the smart glasses captures images, and the microphone records audio. This data is sent in real time to a processing unit within the device.

The device then converts the acquired visual information into text using OCR technology. For example, it can recognize a cafe sign and convert it into text such as "Cafe ABC." Similarly, it converts auditory information into text using voice recognition technology. For example, it can recognize the conversation of a passerby and generate text such as "This cafe is delicious."

The captured video and audio data is then analyzed by the device's built-in emotion engine to determine the user's emotions. The emotion engine uses machine learning algorithms to identify emotions from the user's facial expressions and tone of voice. For example, it can recognize whether the user is happy, interested, or surprised.

The converted text information and recognized emotion data are sent from the device to the server. The server then searches for related information on the web based on the received text information. For example, based on the keyword "cafe ABC," reviews, menus, and location details can be retrieved using a search engine such as Google. Furthermore, by taking emotion data into consideration, it is possible to prioritize retrieving information that is likely to interest the user.

The search results are then sent back to the device, which then displays the retrieved information in AR format. For example, reviews and popular menu items for "Cafe ABC" are overlaid on the smartglasses display. This allows users to access relevant information in real time through their smartglasses even while walking around town.

The device also collects interest data based on the user's gaze and voice commands. For example, if a user stares intently at a particular piece of information or says something like "I'm interested in this cake," the device will record it. By combining this with emotional data recognized by the emotion engine, more detailed interest analysis becomes possible.

The collected interest and emotion data is sent to the server for analysis. The server uses this data to understand the user's interest trends and reflects this in the next information provided. For example, if the user shows a strong interest in cafes, information about cafes can be displayed preferentially from the next time onwards.

As a specific example, if a user is walking down the street and spots a sign for "Cafe XYZ," the camera on the smart glasses will recognize the sign and convert it into text information as "Cafe XYZ" using OCR technology. Furthermore, if the user says, "I'm interested in the cake at this cafe," speech recognition technology will convert the statement into text. At the same time, the emotion engine will recognize positive emotions from the user's interested facial expression and tone of voice. This information is sent to the server, which will search for detailed information about "Cafe XYZ" and prioritize information that is likely to interest the user. This information is sent to the device and displayed in AR format on the smart glasses' display. The user can then check reviews and photos of the cake on the spot.

This type of service allows users to obtain and provide information in real time, significantly improving the user experience. Furthermore, by utilizing emotional data, even more personalized information can be provided.

The processing flow is explained below.

Step 1:

The device acquires visual and auditory information from the outside world. Specifically, the smart glasses' built-in camera captures video and the microphone records audio. This data is stored in the device's memory in real time.

Step 2:

Visual information acquired by the device is converted into text information using OCR technology. For example, if a sign that reads "Cafe XYZ" is captured, the image data is analyzed and the text data "Cafe XYZ" is generated. Similarly, auditory information acquired is converted into text information using voice recognition technology. For example, a conversation such as "This cafe is delicious" is converted into text format.

Step 3:

The device uses an emotion engine to identify the user's emotions from visual and auditory information. It uses machine learning algorithms to analyze the user's facial expressions and tone of voice to recognize emotions such as joy, interest, and surprise. For example, if the user smiles at a cafe sign, the device will identify the emotion of joy.

Step 4:

The device sends the converted text information and identified emotion data to the server, which then uploads the data to the server via the Internet.

Step 5:

The server searches for related information on the web based on the text information it receives. For example, you can perform a Google search using the keyword "cafe XYZ" to retrieve reviews and menu information for that cafe.

Step 6:

The server organizes the search results and selects the most relevant information. It takes into account emotional data and prioritizes information that is likely to interest the user. For example, if the user is expressing happiness, it will prioritize positive reviews of that cafe.

Step 7:

The server sends the organized information to the device. The selected data is then transferred to the device via the Internet.

Step 8:

The information received by the device is displayed in AR format. Set it up so that reviews and popular menu items from "Cafe XYZ" are overlaid on the smart glasses display.

Step 9:

If a user shows interest in specific information through their gaze or voice command, the device will record that data. For example, if a user looks at a particular review or says, "I'm interested in this cake," that information will be recorded.

Step 10:

The device sends the collected interest and emotion data to a server. The data is transmitted in real time via the Internet.

Step 11:

The server analyzes the collected data to understand the user's interest trends and emotional patterns. Based on the results of this analysis, the server reflects this in future information provision, providing the user with the most appropriate information. For example, from the next search result onwards, information about the user's favourite cafes will be displayed first.

(Example 2)

Next, Example 2 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset-type terminal 314 will be referred to as the "terminal."

In modern society, there is an increasing need for users to obtain information in real time and make quick decisions. However, current technology does not adequately provide real-time information based on visual and auditory information, limiting the user experience. In particular, there is a lack of personalized information provision that takes into account the user's emotions and interests, which reduces the efficiency of information gathering and user satisfaction. For this reason, there is a demand for a system that can obtain visual and auditory information from the outside world in real time and provide information that reflects the user's emotions and interests.

The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

In this invention, the server includes means for acquiring visual information from the outside world, means for acquiring auditory information from the outside world, means for converting the visual information into text information, means for converting the auditory information into text information, means for searching for related information on the web based on the converted text information and emotional data, means for displaying the searched information in an augmented reality format, means for recognizing the user's emotions, means for collecting data based on the user's interests, and means for analyzing the collected data and emotional data. This enables users to efficiently obtain personalized information based on their emotions and interests in real time.

"Visual information" refers to video and image data obtained from the external environment.

"Auditory information" refers to sound and acoustic data obtained from the external environment.

"Text information" is character string data obtained by analyzing and converting visual and auditory information.

"Emotional data" is data obtained as a result of analyzing the user's emotional state.

"Relevant information on the web" is information that is searched using the Internet and is relevant to the user's requirements and context.

"Augmented reality" is a technology that displays digital information overlaid on visual information from the real world.

"Means for recognizing user emotions" refers to an algorithm that analyzes video and audio data and identifies the user's emotions.

"Data based on user interests" is data that indicates a user's interests by analyzing input information such as the user's gaze and voice commands.

"Means for analyzing collected data and emotional data" refers to an algorithm that analyzes collected data based on the user's interests and emotions and uses it to provide information the next time.

This invention combines a system that acquires visual and auditory information from the outside world in real time, searches for related information on the web based on that information, and displays it to the user in augmented reality format with an emotion engine that recognizes the user's emotions.

First, the smart glasses, which serve as the terminal, use a camera to acquire visual information from the outside world and a microphone to acquire auditory information from the outside world. The video and audio data acquired at this point is sent in real time to a processing unit within the terminal.

The device then converts the visual information into text using OCR technology. For example, it recognizes a cafe sign and converts it into text such as "Cafe ABC." Similarly, it converts auditory information into text using voice recognition technology. For example, it recognizes the conversation of a passerby and converts it into text such as "This cafe is delicious."

The device also uses the captured video and audio data to analyze the user's emotions using an emotion engine, which can identify emotions such as whether the user is happy, interested, or surprised. The emotion engine uses machine learning algorithms to recognize emotions from facial expressions and tone of voice.

The converted text information and recognized emotion data are sent to the server. The server uses the received text information to search for related information on the web. For example, based on the keyword "cafe ABC," it can use a search engine to retrieve reviews, menus, and location details. Furthermore, by taking emotion data into consideration, it can prioritize retrieving information that is likely to interest the user.

The search results are then sent back to the device, which then displays the retrieved information in augmented reality (AR) format. For example, reviews and popular menu items for "Cafe ABC" may be overlaid on the real-world view on the smart glasses display, allowing users to obtain relevant information in real time.

In addition, the device also collects interest data based on the user's gaze and voice commands. If the user stares intently at a particular piece of information or says something like "I'm interested in this cake," the device records it. By combining this with emotional data recognized by the emotion engine, more detailed interest analysis becomes possible.

The collected interest and emotion data is sent to the server, which uses this data to understand the user's interest trends and reflect them in the next information provided. For example, if the user shows a strong interest in cafes, information about cafes can be displayed preferentially from the next time onwards.

Detailed examples

If a user is walking down the street and spots a sign for "Cafe XYZ," the camera on the smart glasses will recognize the sign and convert it into text using OCR technology: "Cafe XYZ." Furthermore, if the user says, "I'm interested in the cakes at this cafe," the speech recognition technology will convert the statement into text. At the same time, the emotion engine will recognize positive emotions from the user's interested facial expression and tone of voice. This information is sent to the server, which searches for detailed information about "Cafe XYZ" and prioritizes information that is likely to interest the user. This information is then sent to the device and displayed in augmented reality (AR) on the smart glasses' display. Users can then instantly check reviews and photos of the cakes.

Example prompt:

"When a user searches for information about a cafe, please explain the detailed processing flow of the system in which smart glasses acquire visual and auditory data, analyze the user's emotions using an emotion engine, and display information in AR format."

The flow of the identification process in Example 2 will be explained using Figure 13.

Step 1:

The smart glasses that serve as the device collect visual and auditory information from the outside world.

Input: Video and audio data from the outside world.

Processing: The smart glasses' camera captures video and the microphone records audio.

Output: Captured video data and recorded audio data.

Specific operation: As the user walks around town, the smart glasses take photos of buildings and signs, and record surrounding conversations and environmental sounds.

Step 2:

Visual information acquired by the device is converted into text information using OCR technology, and auditory information is converted into text using voice recognition technology.

Input: Video and audio data.

Processing: Use OCR technology to extract text from video data, and use voice recognition technology to convert audio data into text.

Output: Text information.

Specific operation: The smart glasses capture a sign that reads "Cafe XYZ" and convert it into text information: "Cafe XYZ." Similarly, they recognize the speech of a passerby, "The cake at this cafe is delicious," and convert it into text.

Step 3:

The device uses video and audio data to analyze the user's emotions using an emotion engine.

Input: Video and audio data.

Processing: Machine learning algorithms are used to analyze facial expressions from video data and tone of voice from audio data to identify emotions.

Output: Emotion data.

Specific behavior: Analyzes the user's facial expression and tone of voice when they look at a cafe sign to determine whether they are interested or pleased.

Step 4:

The device sends the converted text information and emotion data to the server.

Input: Text information and emotion data.

Processing: Packetize the data and send it to the server.

Output: Data sent to the server.

Specific operation: The device sends the text information "Cafe XYZ" and emotion data indicating "interest" to the server.

Step 5:

The server searches for relevant information on the web based on the text information and emotion data received.

Input: Text information and emotion data.

Processing: Use a search engine to search for relevant information and prioritize information that is likely to interest the user, taking into account emotional data.

Output: Search results for related information.

Specific operation: The server searches for reviews and menu information about "Cafe XYZ" and selects information that is likely to interest the user.

Step 6:

The server will send the search results back to the device.

Input: Search results for related information.

Processing: The search results are compiled into packets and sent to the terminal.

Output: Data sent to the terminal.

Specific operation: The server sends data such as reviews and popular menu items for "Cafe XYZ" to the device.

Step 7:

The information acquired by the device is displayed to the user in augmented reality (AR) format.

Input: Search results for related information.

Processing: Using augmented reality (AR) technology, digital information is overlaid on real-world scenes.

Output: Information displayed in AR format.

Specific operation: Reviews of Cafe XYZ and photos of cakes are overlaid on the real world on the smart glasses display.

Step 8:

The device collects interest data based on the user's gaze and voice commands.

Input: User gaze data and voice commands.

Processing: Uses an eye gaze sensor to detect where the user is looking, and analyzes voice commands using voice recognition technology.

Output: Interest data.

Specific behavior: If a user stares at a photo of a cake for a long time or says, "I'm interested in this cake," the device will collect that data.

Step 9:

The device sends the collected interest and emotion data to the server, which analyzes it.

Input: Interest data and emotion data.

Processing: The data is packaged into packets and sent to a server, where it is analyzed using machine learning algorithms.

Output: Analysis results.

Specific operations: The collected data is sent to a server, which analyzes it to understand trends in user interests.

Step 10:

The server will use the analysis results to provide information from the next time onwards.

Input: Analysis results.

Processing: The analysis results will be recorded in a database and used for the next information provision.

Output: Preparing for next information submission.

Specific behavior: If the user shows a strong interest in the cafe, prepare to prioritize displaying information about the cafe from the next time onwards.

(Application Example 2)

Next, we will explain Application Example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the headset-type terminal 314 will be referred to as the "terminal."

In autonomous vehicles, passengers and drivers have limited means to effectively grasp information about the outside world and obtain specific information in real time. Furthermore, there is a lack of information provided that reflects the user's emotions and interests, which presents a challenge in terms of the user experience not being sufficiently improved.

The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring visual information from the outside world, means for acquiring auditory information from the outside world, and means for converting the visual information into text information. This makes it possible to acquire related information based on visual and auditory information in real time and provide personalized information that reflects the user's emotions and interests.

"Visual information from the outside world" refers to video and image data of the surroundings acquired using cameras and sensors installed on smart devices.

"External auditory information" refers to surrounding voice and sound data acquired using a microphone installed on a smart device.

"Means for converting visual information into text information" refers to optical character recognition (OCR) technology for converting video and image data into text information.

"Means for converting auditory information into text information" refers to speech recognition technology for converting voice data into text information.

"Means for searching for related information on the web" refers to the function of searching for information on the Internet using a server or search engine and obtaining the necessary data.

"Means of displaying in augmented reality (AR) format" refers to technology for displaying acquired information by overlaying it on real-world images.

"Means for collecting data based on user interests" refers to a function that infers a user's interests from their gaze, voice commands, etc., and collects data related to those interests.

"Means for analyzing collected data" refers to algorithms and methods for analyzing collected data and understanding user interests and behavioral patterns.

"Means for analyzing user emotions" refers to machine learning algorithms that analyze a user's facial expressions and tone of voice to determine their emotional state.

"Smart devices installed in self-driving vehicles" refers to electronic devices with functions such as cameras, microphones, and displays that are used inside self-driving vehicles.

This invention is a system for improving the user experience in autonomous vehicles. It uses smart devices to acquire visual and auditory information from the outside world in real time, and uses this information to search for and display relevant information. It also features the ability to analyze the user's emotions and provide information based on their interests.

First, the smart device, which serves as the terminal, acquires visual and auditory information from the outside world. Specifically, the camera installed on the smart device captures video and the microphone records audio. This data is sent in real time to a processing unit within the terminal.

The device then converts the acquired visual information into text using OCR technology. For example, if an autonomous vehicle recognizes a restaurant sign while driving through town, it will convert it into text such as "Restaurant ABC." Similarly, auditory information is converted into text using voice recognition technology. For example, if a passenger says, "I'm interested in the menu at this restaurant," this will be recorded as text.

Furthermore, the captured video and audio data is analyzed for the user's emotions by the device's built-in emotion engine. The emotion engine uses machine learning algorithms to identify emotions from the user's facial expressions and tone of voice. For example, it can recognize whether the user is interested, happy, or surprised.

The converted text information and recognized emotion data are sent from the device to a cloud server. The cloud server then searches for related information on the web based on the received text information. For example, based on the keyword "restaurant ABC," reviews, menus, and location details can be retrieved using a search engine. Furthermore, by taking emotion data into consideration, it is possible to prioritize retrieving information that is likely to interest the user.

The search results are then sent back to the device, which then displays the retrieved information in AR format. For example, reviews and popular menu items for "Restaurant ABC" are overlaid on the smart device's display. This allows users to access relevant information in real time even while on the move.

The device also collects interest data based on the user's gaze and voice commands. For example, if a user stares intently at a particular piece of information or says something like "this dish looks delicious," that data is recorded. By combining this with emotional data recognized by the emotion engine, more detailed interest analysis becomes possible.

The collected interest and emotion data is sent to a cloud server for analysis. The cloud server uses this data to understand the user's interest trends and reflects this in the next information provided. For example, if the user shows a strong interest in restaurants, information about restaurants can be displayed preferentially from the next time onwards.

As a concrete example

As a specific example, if a user spots a sign for "Restaurant XYZ" while driving around town, the camera on the smart device will recognize the sign and convert it into text information as "Restaurant XYZ" using OCR technology. Furthermore, if the user says, "I'm interested in the menu at this restaurant," speech recognition technology will convert this statement into text. At the same time, an emotion engine will recognize positive emotions from the user's interested facial expression and tone of voice. This information is sent to a cloud server, which then searches for detailed information about "Restaurant XYZ" and prioritizes information that is likely to interest the user. This information is then sent to the device and displayed in AR format on the smart device's display. The user can then check reviews and photos of the menu on the spot.

Example prompt

An example of a prompt is shown below.

"When a user finds a restaurant sign, the smart device identifies the information, converts it into text, and prioritizes displaying reviews and menu information for that restaurant based on the user's sentiment. For example, if a user says, 'I'm interested in the menu at this restaurant,' the device converts that speech into text and retrieves and displays relevant information in real time."

The flow of the specific processing in Application Example 2 will be explained using Figure 14.

Step 1:

The terminal, a smart device, acquires visual and auditory information from the outside world. As the user moves around town, the camera captures video of the surroundings and the microphone records audio from the surroundings. This data is sent to a processing unit in real time. The input is camera video data and audio data, and the output is raw data sent to the processing unit.

Step 2:

The device converts the acquired visual information into text using OCR technology. For example, if a smart device's camera captures a restaurant sign, the video data is converted into text, generating the text information "Restaurant XYZ." The input is video data, and the output is text data.

Step 3:

The device uses voice recognition technology to convert the acquired voice information into text information. For example, if a user says, "I'm interested in the menu at this restaurant," the voice data is converted into text information and related information is generated. The input is voice data, and the output is text data.

Step 4:

The device uses an emotion engine to analyze the user's emotions from the captured video and audio data. For example, it can recognize the emotion "interesting" from the user's facial expression and tone of voice. The input is video and audio data, and the output is emotional data.

Step 5:

The converted text information and recognized emotion data are sent from the device to a cloud server. The server searches for related information on the web based on the received text information. For example, based on the keyword "restaurant XYZ," it retrieves reviews and menu information. The input is text data and emotion data, and the output is related information data.

Step 6:

The cloud server takes emotional data into consideration and prioritizes obtaining information that is likely to interest the user, and then sends the results back to the device. For example, if the user is expressing positive emotions, it will prioritize sending highly rated restaurant reviews. The input is text data and emotional data, and the output is related information data.

Step 7:

The device displays the acquired information in AR format. For example, reviews and popular menu items for "Restaurant XYZ" are overlaid on the smart device's display, allowing the user to view them. The input is related information data, and the output is the displayed AR data.

Step 8:

The device continues to collect interest data based on the user's gaze and voice commands. For example, if the user stares at a particular piece of information or says, "I'm interested in this dish," that data is recorded. The input is gaze data and voice data, and the output is interest data.

Step 9:

The collected interest and emotion data is sent to a cloud server for detailed analysis. The server uses this data to understand the user's interest trends and reflects this in the next information provided. For example, if there is a strong interest in restaurants, information about restaurants will be displayed preferentially from the next time onwards. The input is interest data and emotion data, and the output is analysis data for the next information provided.

The specific processing unit 290 transmits the results of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the results of the specific processing. The microphone 238 acquires audio indicating the user input regarding the results of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

Data generation model 58 is what is known as generative AI (artificial intelligence). Examples of data generation model 58 include generative AI such as ChatGPT (Internet search <URL: https://openai.com/blog/chatgpt>) and Gemini (Internet search <URL: https://gemini.google.com/?hl=ja>). Data generation model 58 is obtained by performing deep learning on a neural network. A prompt containing an instruction is input to data generation model 58, and inference data such as voice data indicating speech, text data indicating text, and image data indicating an image is also input. Data generation model 58 performs inference on the input inference data in accordance with the instructions indicated by the prompt, and outputs the inference results in the form of data such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and/or summarization.

In the above embodiment, an example was given in which the specific processing was performed by the data processing device 12, but the technology disclosed herein is not limited to this, and the specific processing may also be performed by the headset-type terminal 314.

[Fourth embodiment]

Figure 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

As shown in FIG. 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

The data processing device 12 includes a computer 22, a database 24, and a communication I/F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and communication I/F 26 are also connected to the bus 34. The communication I/F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and/or a LAN (Local Area Network).

The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I/F 44, and a control target 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and control target 443 are also connected to the bus 52.

The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the user 20's surroundings (e.g., an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

The communication I/F 44 is connected to the network 54. The communication I/Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I/Fs 44 and 26 is carried out in a secure manner.

The control object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, in the data processing device 12, specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32.

The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as the specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

Storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

In the robot 414, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

Next, we will explain the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 will be referred to as the "terminal."

This invention is a system that acquires visual and auditory information from the outside world in real time, searches for related information on the web based on that information, and displays it to the user in AR format. Specific embodiments of this system are described below.

First, the smart glasses, which serve as the device, acquire visual and auditory information from the outside world. Specifically, the camera installed in the smart glasses captures images, and the microphone records audio. This data is sent in real time to a processing unit within the device.

The device then uses OCR (optical character recognition) technology to convert the acquired visual information into text information. For example, it can recognize a cafe sign and convert it into text information such as "Cafe ABC." Similarly, auditory information is converted into text information using voice recognition technology. For example, it can recognize the conversation of a passerby and generate text information such as "This cafe is delicious."

The converted text information is sent from the device to the server. The server uses the received text information to search for related information on the web. Specifically, it uses a web search engine to obtain reviews, menus, location information, etc. for "Cafe ABC." The search results are then sent back to the device.

The device organizes the search results and displays them in AR format in the user's field of view. For example, reviews and popular menu items for "Cafe ABC" could be overlaid on the smartglasses' display. This allows users to access relevant information in real time through their smartglasses even while walking around town.

Furthermore, the device collects data on the user's interests. Specifically, when the user looks at specific information or issues a voice command such as "I'm interested in this menu item," that information is recorded. The collected interest data is sent to a server, which analyzes it to understand the user's interest trends. The results of this analysis are reflected in subsequent searches and information provision, making it possible to provide the most appropriate information tailored to the user's preferences.

For example, if a user is walking down the street and spots a sign that says "Cafe XYZ," the camera in the smart glasses will recognize the sign and convert it into text information as "Cafe XYZ" using OCR technology. Furthermore, if the user says, "I'm interested in the cake at this cafe," speech recognition technology will convert this statement into text. The server will perform a web search based on the keywords "Cafe XYZ" and "cake" to retrieve related reviews and menu information. This information will be sent to the device and displayed in AR format on the smart glasses' display. The user will be able to check reviews and photos of the cake on the spot.

In this manner, the present invention enables users to obtain and provide information in real time, improving the user experience. Furthermore, by collecting and analyzing data based on the user's interests, it becomes possible to provide even more personalized information.

The processing flow is explained below.

Step 1:

The device acquires visual and auditory information from the outside world. Specifically, the smart glasses' built-in camera captures video and a microphone records audio. This data is sent in real time to a processing unit within the device.

Step 2:

Visual information acquired by the device is converted into text information using OCR technology. For example, the device recognizes a sign that reads "Cafe ABC" and converts the image into text. Similarly, auditory information acquired is converted into text information using voice recognition technology. For example, a passerby's speech, "This cafe is delicious," is converted into text.

Step 3:

The device sends the converted text information to the server. Specifically, the text data generated by OCR or voice recognition is uploaded to the server via the Internet.

Step 4:

The server searches for related information on the web based on the text information it receives. The server uses a search engine to obtain reviews, menus, location information, etc. for "Cafe ABC." It is also possible to gather more information by using multiple search engines.

Step 5:

The server organizes the search results and selects the most relevant information. For example, it prioritizes reviews with high user ratings and information from official websites.

Step 6:

The server sends the organized information to the device. Specifically, it sends data summarizing related reviews and menu information to the device via the Internet.

Step 7:

The information received by the device is displayed in AR format. Reviews and popular menu items from Cafe ABC are overlaid on the smartglasses display.

Step 8:

When a user shows interest in specific information through their gaze or voice, the device records that data. For example, if a user stares intently at a particular review page or says, "I'm interested in this menu item."

Step 9:

The device sends the collected interest data to the server. The interest data includes the user's gaze information and voice commands.

Step 10:

The server analyzes the collected interest data to understand trends in user interests. The results of this analysis are reflected in future search results and information provision, enabling the server to provide users with the most appropriate information.

(Example 1)

Next, Example 1 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 will be referred to as the "terminal."

In modern society, users are expected to obtain the information they need quickly and efficiently. It is particularly important to improve the user experience by obtaining and utilizing information in real time, even while on the move or while active. However, existing technology lacks sufficient methods for appropriately obtaining and providing information that interests users in real time, and the information acquisition process is cumbersome. Furthermore, there are still challenges in identifying users' interest trends and providing personalized information based on them. Against this background, there is a need for technology that can obtain and analyze users' visual and auditory information and provide relevant information in real time based on that information.

The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

In this invention, the server includes means for acquiring visual information from the outside world, means for acquiring auditory information from the outside world, means for converting the visual information into text information, means for converting the auditory information into text information, means for searching for related information on the web based on the converted text information, means for displaying the searched information in an augmented reality (AR) format, means for collecting data based on the user's interests in real time, and means for analyzing the collected data and reflecting it in subsequent search results. This enables the user to quickly and efficiently obtain the information they need while on the move or while active, and further enables the provision of personalized information based on the user's interests and preferences.

"Visual information from the external world" refers to visual information such as objects and letters that exist in the surrounding environment.

"External auditory information" refers to auditory information such as sounds and conversations that exist in the surrounding environment.

"Means for converting visual information into text information" refers to technology that converts acquired visual information into text data, specifically the use of OCR (optical character recognition) technology.

"Means for converting auditory information into text information" refers to technology that converts acquired auditory information into text data, specifically using voice recognition technology.

"Means for searching for related information on the web" refers to technology that searches for and retrieves related information on the Internet based on converted text information.

"Augmented reality (AR) display" refers to technology that displays acquired information overlaid on the real world, allowing users to simultaneously view images of the real world and additional information.

"Means for collecting data based on user interests in real time" refers to technology that records and collects trends in interest in real time when a user shows interest in specific information.

"Means of analyzing collected data and reflecting it in future search results" refers to technology that analyzes collected user interest data and personalizes the next information provided based on the results of that analysis.

This invention is a system that acquires visual and auditory information from the outside world in real time, searches for related information on the web based on that information, and displays it to the user in augmented reality (AR) format. Specific embodiments of this system are described below.

First, the smart glasses, which serve as the device, acquire visual and auditory information from the outside world. Specifically, the camera installed in the smart glasses captures images, and the microphone records audio. This data is sent in real time to a processing unit within the device.

The device then uses OCR (optical character recognition) technology to convert the acquired visual information into text information. For example, it can recognize a cafe sign and convert it into text information such as "Cafe ABC." Similarly, auditory information is converted into text information using voice recognition technology. For example, it can recognize the conversation of a passerby and generate text information such as "This cafe is delicious."

The converted text information is sent from the device to the server. The server uses the received text information to search for related information on the web. Specifically, it uses a web search engine to obtain reviews, menus, location information, etc. for "Cafe ABC." The search results are then sent back to the device.

The device organizes the search results and displays them in AR format in the user's field of view. For example, reviews and popular menu items for "Cafe ABC" could be overlaid on the smartglasses' display. This allows users to access relevant information in real time through their smartglasses even while walking around town.

Furthermore, the device collects data on the user's interests. Specifically, when the user looks at specific information or issues a voice command such as "I'm interested in this menu item," that information is recorded. The collected interest data is sent to a server, which analyzes it to understand the user's interest trends. The results of this analysis are reflected in subsequent searches and information provision, making it possible to provide the most appropriate information tailored to the user's preferences.

As a concrete example, if a user is walking down the street and spots a sign for "Cafe XYZ," the camera in the smart glasses will recognize the sign and convert it into text information as "Cafe XYZ" using OCR technology. Furthermore, if the user says, "I'm interested in the cake at this cafe," speech recognition technology will convert the comment into text. The server will perform a web search based on the keywords "Cafe XYZ" and "cake" to retrieve related reviews and menu information. This information will be sent to the device and displayed in AR format on the smart glasses' display. The user will be able to check reviews and photos of the cake on the spot.

Example prompts are as follows:

"I saw a sign for Cafe XYZ and would like more information and reviews. I'd also like to know what people think of the cafe's cakes."

In this manner, the present invention enables users to obtain and provide information in real time, improving the user experience. Furthermore, by collecting and analyzing data based on the user's interests, it becomes possible to provide even more personalized information.

The flow of the identification process in Example 1 will be explained using Figure 11.

Step 1:

The device acquires visual and auditory information from the outside world.

Inputs include images of the outside world that appear in the user's field of vision and sounds from the environment.

Specifically, the camera built into the smart glasses captures video footage, and the microphone records audio.

As output, the captured video and audio data is sent to the processing unit inside the terminal.

Step 2:

The visual information acquired by the device is converted into text information.

The input is the video data acquired in step 1.

Specifically, the device uses OCR (optical character recognition) technology to analyze the characters in the video data and convert it into text data. For example, it would recognize a sign that reads "Cafe ABC" and convert it into text as "Cafe ABC."

The output is the converted text data.

Step 3:

The device converts the auditory information it acquires into text information.

The input is the audio data obtained in step 1.

Specifically, the device uses voice recognition technology to analyze voice data and convert it into text data. For example, a passerby's speech, "This cafe is delicious," can be converted into text data.

The output is the converted text data.

Step 4:

The device sends the converted text information to the server.

The input is the text data obtained in steps 2 and 3.

Specific operations include using a communications module within the device to send text data to a server via the Internet.

As output, text data arrives at the server.

Step 5:

The server searches for relevant information on the web based on the text information received.

The input is the text data that arrived at the server in step 4.

Specifically, the server uses a web search engine (e.g., Google's API) to search for reviews, menus, location information, etc. related to "Cafe ABC."

The output is the search results.

Step 6:

The server sends the search results to the device.

The input is the search results obtained in step 5.

Specifically, the server uses a communications module to send search results to the device. The data is sent in a lightweight data format such as JSON.

As output, the search results arrive on the terminal.

Step 7:

The device organizes the search results and displays them in AR format in the user's field of view.

The input is the search results that arrived on the terminal in step 6.

Specifically, the device will overlay reviews and popular menu items for "Cafe ABC" on the smartglasses' display. For example, it might show something like "Cafe ABC: Rating 4.5 stars, Popular menu item: Cheesecake."

As output, search results are displayed in AR format in the user's field of vision.

Step 8:

The device collects user interest data and sends it to the server.

Inputs include user behavior data such as gaze fixation and voice commands.

Specifically, the device detects the user's eye movements and records information when the user looks at specific information or issues a voice command such as "I'm interested in this menu item." This interest data is sent to a server via the Internet.

As output, the interest data arrives at the server.

Step 9:

The server analyzes the collected data and reflects it in future search results.

The input is the interest data that arrived at the server in step 8.

Specifically, the server analyzes interest data and models the user's interest trends. Based on this model, the server provides personalized results the next time the user searches or provides information.

The analysis results are obtained as output and will be reflected in subsequent search results.

(Application Example 1)

Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 will be referred to as the "terminal."

With conventional technology, users had limited means of obtaining real-time product information in physical stores, making it difficult to quickly obtain reviews and promotional information about specific products. Furthermore, there was a lack of provision of personalized information tailored to users' interests. This could result in a poor user experience and a decrease in purchasing motivation.

The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

In this invention, the server includes means for acquiring visual information from the outside world, means for acquiring auditory information from the outside world, means for converting the visual information into text information, means for converting the auditory information into text information, means for searching for related information on the web based on the converted text information, means for displaying the searched information in an augmented reality (AR) format, means for collecting data based on the user's interests, means for analyzing the collected data, and means for providing product information and reviews in AR format in real time while the user is walking. This allows users to quickly and intuitively obtain product information in a physical store, and the provision of personalized information can improve the shopping experience.

"Visual information from the outside world" refers to image information about the user's surrounding environment and objects, and is primarily obtained using imaging devices such as cameras.

"External auditory information" refers to the sounds and acoustic information around the user, and is primarily obtained by acoustic collection devices such as microphones.

"Means for converting visual information into text information" refers to means that use optical character recognition (OCR) technology to convert acquired visual information into text-format information such as letters and numbers.

"Means for converting auditory information into text information" refers to means that use voice recognition technology to convert acquired auditory information into text-format information such as letters or words.

"Means for searching for relevant information on the web" refers to means that include a search engine function for finding appropriate information on the Internet based on converted text information.

"Augmented reality (AR) display" refers to a technology that displays digital information overlaid on real-world images, and is a method that uses display devices such as smart glasses.

"Means for collecting data based on user interests" refers to means for collecting information about subjects that the user is interested in, such as the user's gaze or voice commands.

"Means for analyzing collected data" refers to means for analyzing collected user interest data and understanding user preferences and trends.

"A means of providing product information and reviews in real time in AR format while the user is walking" refers to a means of instantly displaying product information and user reviews in augmented reality format through a visual device while the user is moving around in a physical store.

"Means for detecting a user's gaze" refers to means that use gaze tracking devices or technology to detect and track the user's gaze direction and gaze point.

"Means for recognizing user voice commands" refers to means for using a voice recognition system to capture, understand, and process voice instructions given by the user.

This invention provides a system that allows users to obtain product information and reviews in real time while in a physical store. Specifically, the system uses smart glasses to obtain visual and auditory information from the outside world, generates text information based on that information, searches for related information on the web, and displays it in augmented reality (AR) format.

System Configuration

The system consists of the following main components:

Hardware

Smart glasses: Equipped with a camera, microphone, display, and processor. Worn by the user, they capture visual and auditory information from the outside world.

Cloud server: Responsible for large-volume data processing and storage.

Software

Optical character recognition (OCR) technology (e.g., Tesseract): converts captured visual information into text.

Speech recognition technology (e.g., Google Cloud Speech-to-Text): Converts captured auditory information into text format.

Web search engines (e.g., Elasticsearch): Search for relevant information on the Internet based on converted text information.

Augmented reality (AR) technology (e.g., ARCore): Displays acquired information in AR format on the smart glasses display.

Eye tracking technology: Detects where the user is looking and identifies objects of interest.

Operation Overview

1. Acquisition and conversion of visual information: The smart glasses' camera captures products and signs in the store and converts them into text information using OCR technology.

2. Acquisition and conversion of auditory information: The smart glasses' microphone records the user's voice commands and converts them into text using voice recognition technology.

3. Search for related information: The converted text information is sent to a cloud server and related information is retrieved using a web search engine.

4. Displaying information in AR format: The acquired information is overlaid on the smart glasses display in AR format.

5. Collection and analysis of interest data: By recording the user's gaze and voice commands and analyzing them on a cloud server, we can understand the user's interests and concerns, and use this information to provide them in future visits.

Specific examples

For example, if a user is looking for a specific product in a store, the smart glasses' camera will recognize the product and convert it into text information such as "Product name XYZ." A web search will then be performed using the keywords "Product name XYZ review," and the retrieved review information will be displayed in AR on the smart glasses' display. Furthermore, if the user issues a voice command such as "I want to know the rating of this product," the voice will be converted into text, and additional review information will be searched for and displayed. If the user looks at specific information, that information will be recorded and used to personalize the next search results.

Example prompt

When a user is searching for a specific product in a store,

Capture product name XYZ with the smart glasses camera,

Use OCR to convert "Product Name XYZ" from camera footage into text information.

Then, do a web search using the keywords "Product Name XYZ Review"

Display the acquired review information in AR on the smart glasses display.

Furthermore, if the user issues a voice command such as "I want to know the rating of this product,"

Convert speech to text and re-search for additional review information.

When a user looks at specific information, record that information and use it to personalize your next search results.

The flow of the specific processing in Application Example 1 will be explained using Figure 12.

Step 1:

The camera in the smart glasses captures the environment in a physical store. The input is a video of the products in the user's field of view. The processor in the smart glasses captures this video data in real time and sends it to the processing unit as visual information. The output is the captured video data.

Step 2:

The processor in the smart glasses converts the captured video data into text information using OCR technology. The captured video data is used as input. OCR technology (e.g., Tesseract) extracts text information from product names and signs and converts it into text information such as "Product Name XYZ." The output is the converted text information.

Step 3:

The microphone in the smart glasses records voice commands given by the user. The input obtained here is the user's voice information. This voice information is converted into text information using voice recognition technology (e.g., Google Cloud Speech-to-Text). For example, the voice command "I would like to hear reviews of this product" is converted into text "I would like to hear reviews of this product." The output is the text information of the converted voice.

Step 4:

The device sends the converted visual and auditory information to a cloud server. The input is text information converted using OCR and speech recognition. The cloud server searches for related information on the web based on the received text information. Specifically, it uses a web search engine (e.g., Elasticsearch) to execute a search query such as "Product Name XYZ Reviews" to retrieve related reviews and menu information. The output is related information as search results.

Step 5:

The cloud server organizes the retrieved relevant information and sends it to the smart glasses. The relevant information obtained through the search is used as input. First, the relevant information is formalized and converted into a format suitable for the smart glasses' display. The output is data formatted for AR display.

Step 6:

The smart glasses display displays the acquired information in an AR overlay format. The input here is formatted data sent from the cloud server. The smart glasses display overlays product reviews and menu information in the user's field of view. The output is AR display information overlaid on the physical store environment.

Step 7:

The eye-tracking function of smart glasses detects the user's gaze and records information that the user shows interest in. The input is data on the direction of gaze and point of view. Eye-tracking technology analyzes in real time what information the user is focusing on, and the results are recorded as data. The output is interest data.

Step 8:

The cloud server analyzes the collected user interest data. The input is interest data recorded using eye-tracking technology and voice commands. The server analyzes this data to understand the user's preferences and trends. This information is used for future searches and information provision. The output is analyzed data on the user's preferences and trends.

It is also possible to further combine an emotion engine that estimates the user's emotion. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

This invention combines a system that acquires visual and auditory information from the outside world in real time, searches for related information on the web based on that information, and displays it to the user in AR format with an emotion engine that recognizes the user's emotions. Specific embodiments of this system are described below.

First, the smart glasses, which serve as the device, acquire visual and auditory information from the outside world. Specifically, the camera installed in the smart glasses captures images, and the microphone records audio. This data is sent in real time to a processing unit within the device.

The device then converts the acquired visual information into text using OCR technology. For example, it can recognize a cafe sign and convert it into text such as "Cafe ABC." Similarly, it converts auditory information into text using voice recognition technology. For example, it can recognize the conversation of a passerby and generate text such as "This cafe is delicious."

The captured video and audio data is then analyzed by the device's built-in emotion engine to determine the user's emotions. The emotion engine uses machine learning algorithms to identify emotions from the user's facial expressions and tone of voice. For example, it can recognize whether the user is happy, interested, or surprised.

The converted text information and recognized emotion data are sent from the device to the server. The server then searches for related information on the web based on the received text information. For example, based on the keyword "cafe ABC," reviews, menus, and location details can be retrieved using a search engine such as Google. Furthermore, by taking emotion data into consideration, it is possible to prioritize retrieving information that is likely to interest the user.

The search results are then sent back to the device, which then displays the retrieved information in AR format. For example, reviews and popular menu items for "Cafe ABC" are overlaid on the smartglasses display. This allows users to access relevant information in real time through their smartglasses even while walking around town.

The device also collects interest data based on the user's gaze and voice commands. For example, if a user stares intently at a particular piece of information or says something like "I'm interested in this cake," the device will record it. By combining this with emotional data recognized by the emotion engine, more detailed interest analysis becomes possible.

The collected interest and emotion data is sent to the server for analysis. The server uses this data to understand the user's interest trends and reflects this in the next information provided. For example, if the user shows a strong interest in cafes, information about cafes can be displayed preferentially from the next time onwards.

As a specific example, if a user is walking down the street and spots a sign for "Cafe XYZ," the camera on the smart glasses will recognize the sign and convert it into text information as "Cafe XYZ" using OCR technology. Furthermore, if the user says, "I'm interested in the cake at this cafe," speech recognition technology will convert the statement into text. At the same time, the emotion engine will recognize positive emotions from the user's interested facial expression and tone of voice. This information is sent to the server, which will search for detailed information about "Cafe XYZ" and prioritize information that is likely to interest the user. This information is sent to the device and displayed in AR format on the smart glasses' display. The user can then check reviews and photos of the cake on the spot.

This type of service allows users to obtain and provide information in real time, significantly improving the user experience. Furthermore, by utilizing emotional data, even more personalized information can be provided.

The processing flow is explained below.

Step 1:

The device acquires visual and auditory information from the outside world. Specifically, the smart glasses' built-in camera captures video and the microphone records audio. This data is stored in the device's memory in real time.

Step 2:

Visual information acquired by the device is converted into text information using OCR technology. For example, if a sign that reads "Cafe XYZ" is captured, the image data is analyzed and the text data "Cafe XYZ" is generated. Similarly, auditory information acquired is converted into text information using voice recognition technology. For example, a conversation such as "This cafe is delicious" is converted into text format.

Step 3:

The device uses an emotion engine to identify the user's emotions from visual and auditory information. It uses machine learning algorithms to analyze the user's facial expressions and tone of voice to recognize emotions such as joy, interest, and surprise. For example, if the user smiles at a cafe sign, the device will identify the emotion of joy.

Step 4:

The device sends the converted text information and identified emotion data to the server, which then uploads the data to the server via the Internet.

Step 5:

The server searches for related information on the web based on the text information it receives. For example, you can perform a Google search using the keyword "cafe XYZ" to retrieve reviews and menu information for that cafe.

Step 6:

The server organizes the search results and selects the most relevant information. It takes into account emotional data and prioritizes information that is likely to interest the user. For example, if the user is expressing happiness, it will prioritize positive reviews of that cafe.

Step 7:

The server sends the organized information to the device. The selected data is then transferred to the device via the Internet.

Step 8:

The information received by the device is displayed in AR format. Set it up so that reviews and popular menu items from "Cafe XYZ" are overlaid on the smart glasses display.

Step 9:

If a user shows interest in specific information through their gaze or voice command, the device will record that data. For example, if a user looks at a particular review or says, "I'm interested in this cake," that information will be recorded.

Step 10:

The device sends the collected interest and emotion data to a server. The data is transmitted in real time via the Internet.

Step 11:

The server analyzes the collected data to understand the user's interest trends and emotional patterns. Based on the results of this analysis, the server reflects this in future information provision, providing the user with the most appropriate information. For example, from the next search result onwards, information about the user's favourite cafes will be displayed first.

(Example 2)

Next, Example 2 will be described. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

In modern society, there is an increasing need for users to obtain information in real time and make quick decisions. However, current technology does not adequately provide real-time information based on visual and auditory information, limiting the user experience. In particular, there is a lack of personalized information provision that takes into account the user's emotions and interests, which reduces the efficiency of information gathering and user satisfaction. For this reason, there is a demand for a system that can obtain visual and auditory information from the outside world in real time and provide information that reflects the user's emotions and interests.

The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

In this invention, the server includes means for acquiring visual information from the outside world, means for acquiring auditory information from the outside world, means for converting the visual information into text information, means for converting the auditory information into text information, means for searching for related information on the web based on the converted text information and emotional data, means for displaying the searched information in an augmented reality format, means for recognizing the user's emotions, means for collecting data based on the user's interests, and means for analyzing the collected data and emotional data. This enables users to efficiently obtain personalized information based on their emotions and interests in real time.

"Visual information" refers to video and image data obtained from the external environment.

"Auditory information" refers to sound and acoustic data obtained from the external environment.

"Text information" is character string data obtained by analyzing and converting visual and auditory information.

"Emotional data" is data obtained as a result of analyzing the user's emotional state.

"Relevant information on the web" is information that is searched using the Internet and is relevant to the user's requirements and context.

"Augmented reality" is a technology that displays digital information overlaid on visual information from the real world.

"Means for recognizing user emotions" refers to an algorithm that analyzes video and audio data and identifies the user's emotions.

"Data based on user interests" is data that indicates a user's interests by analyzing input information such as the user's gaze and voice commands.

"Means for analyzing collected data and emotional data" refers to an algorithm that analyzes collected data based on the user's interests and emotions and uses it to provide information the next time.

This invention combines a system that acquires visual and auditory information from the outside world in real time, searches for related information on the web based on that information, and displays it to the user in augmented reality format with an emotion engine that recognizes the user's emotions.

First, the smart glasses, which serve as the terminal, use a camera to acquire visual information from the outside world and a microphone to acquire auditory information from the outside world. The video and audio data acquired at this point is sent in real time to a processing unit within the terminal.

The device then converts the visual information into text using OCR technology. For example, it recognizes a cafe sign and converts it into text such as "Cafe ABC." Similarly, it converts auditory information into text using voice recognition technology. For example, it recognizes the conversation of a passerby and converts it into text such as "This cafe is delicious."

The device also uses the captured video and audio data to analyze the user's emotions using an emotion engine, which can identify emotions such as whether the user is happy, interested, or surprised. The emotion engine uses machine learning algorithms to recognize emotions from facial expressions and tone of voice.

The converted text information and recognized emotion data are sent to the server. The server uses the received text information to search for related information on the web. For example, based on the keyword "cafe ABC," it can use a search engine to retrieve reviews, menus, and location details. Furthermore, by taking emotion data into consideration, it can prioritize retrieving information that is likely to interest the user.

The search results are then sent back to the device, which then displays the retrieved information in augmented reality (AR) format. For example, reviews and popular menu items for "Cafe ABC" may be overlaid on the real-world view on the smart glasses display, allowing users to obtain relevant information in real time.

In addition, the device also collects interest data based on the user's gaze and voice commands. If the user stares intently at a particular piece of information or says something like "I'm interested in this cake," the device records it. By combining this with emotional data recognized by the emotion engine, more detailed interest analysis becomes possible.

The collected interest and emotion data is sent to the server, which uses this data to understand the user's interest trends and reflect them in the next information provided. For example, if the user shows a strong interest in cafes, information about cafes can be displayed preferentially from the next time onwards.

Detailed examples

If a user is walking down the street and spots a sign for "Cafe XYZ," the camera on the smart glasses will recognize the sign and convert it into text using OCR technology: "Cafe XYZ." Furthermore, if the user says, "I'm interested in the cakes at this cafe," the speech recognition technology will convert the statement into text. At the same time, the emotion engine will recognize positive emotions from the user's interested facial expression and tone of voice. This information is sent to the server, which searches for detailed information about "Cafe XYZ" and prioritizes information that is likely to interest the user. This information is then sent to the device and displayed in augmented reality (AR) on the smart glasses' display. Users can then instantly check reviews and photos of the cakes.

Example prompt:

"When a user searches for information about a cafe, please explain the detailed processing flow of the system in which smart glasses acquire visual and auditory data, analyze the user's emotions using an emotion engine, and display information in AR format."

The flow of the identification process in Example 2 will be explained using Figure 13.

Step 1:

The smart glasses that serve as the device collect visual and auditory information from the outside world.

Input: Video and audio data from the outside world.

Processing: The smart glasses' camera captures video and the microphone records audio.

Output: Captured video data and recorded audio data.

Specific operation: As the user walks around town, the smart glasses take photos of buildings and signs, and record surrounding conversations and environmental sounds.

Step 2:

Visual information acquired by the device is converted into text information using OCR technology, and auditory information is converted into text using voice recognition technology.

Input: Video and audio data.

Processing: Use OCR technology to extract text from video data, and use voice recognition technology to convert audio data into text.

Output: Text information.

Specific operation: The smart glasses capture a sign that reads "Cafe XYZ" and convert it into text information: "Cafe XYZ." Similarly, they recognize the speech of a passerby, "The cake at this cafe is delicious," and convert it into text.

Step 3:

The device uses video and audio data to analyze the user's emotions using an emotion engine.

Input: Video and audio data.

Processing: Machine learning algorithms are used to analyze facial expressions from video data and tone of voice from audio data to identify emotions.

Output: Emotion data.

Specific behavior: Analyzes the user's facial expression and tone of voice when they look at a cafe sign to determine whether they are interested or pleased.

Step 4:

The device sends the converted text information and emotion data to the server.

Input: Text information and emotion data.

Processing: Packetize the data and send it to the server.

Output: Data sent to the server.

Specific operation: The device sends the text information "Cafe XYZ" and emotion data indicating "interest" to the server.

Step 5:

The server searches for relevant information on the web based on the text information and emotion data received.

Input: Text information and emotion data.

Processing: Use a search engine to search for relevant information and prioritize information that is likely to interest the user, taking into account emotional data.

Output: Search results for related information.

Specific operation: The server searches for reviews and menu information about "Cafe XYZ" and selects information that is likely to interest the user.

Step 6:

The server will send the search results back to the device.

Input: Search results for related information.

Processing: The search results are compiled into packets and sent to the terminal.

Output: Data sent to the terminal.

Specific operation: The server sends data such as reviews and popular menu items for "Cafe XYZ" to the device.

Step 7:

The information acquired by the device is displayed to the user in augmented reality (AR) format.

Input: Search results for related information.

Processing: Using augmented reality (AR) technology, digital information is overlaid on real-world scenes.

Output: Information displayed in AR format.

Specific operation: Reviews of Cafe XYZ and photos of cakes are overlaid on the real world on the smart glasses display.

Step 8:

The device collects interest data based on the user's gaze and voice commands.

Input: User gaze data and voice commands.

Processing: Uses an eye gaze sensor to detect where the user is looking, and analyzes voice commands using voice recognition technology.

Output: Interest data.

Specific behavior: If a user stares at a photo of a cake for a long time or says, "I'm interested in this cake," the device will collect that data.

Step 9:

The device sends the collected interest and emotion data to the server, which analyzes it.

Input: Interest data and emotion data.

Processing: The data is packaged into packets and sent to a server, where it is analyzed using machine learning algorithms.

Output: Analysis results.

Specific operations: The collected data is sent to a server, which analyzes it to understand trends in user interests.

Step 10:

The server will use the analysis results to provide information from the next time onwards.

Input: Analysis results.

Processing: The analysis results will be recorded in a database and used for the next information provision.

Output: Preparing for next information submission.

Specific behavior: If the user shows a strong interest in the cafe, prepare to prioritize displaying information about the cafe from the next time onwards.

(Application Example 2)

Next, we will explain Application Example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 will be referred to as the "terminal."

In autonomous vehicles, passengers and drivers have limited means to effectively grasp information about the outside world and obtain specific information in real time. Furthermore, there is a lack of information provided that reflects the user's emotions and interests, which presents a challenge in terms of the user experience not being sufficiently improved.

The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring visual information from the outside world, means for acquiring auditory information from the outside world, and means for converting the visual information into text information. This makes it possible to acquire related information based on visual and auditory information in real time and provide personalized information that reflects the user's emotions and interests.

"Visual information from the outside world" refers to video and image data of the surroundings acquired using cameras and sensors installed on smart devices.

"External auditory information" refers to surrounding voice and sound data acquired using a microphone installed on a smart device.

"Means for converting visual information into text information" refers to optical character recognition (OCR) technology for converting video and image data into text information.

"Means for converting auditory information into text information" refers to speech recognition technology for converting voice data into text information.

"Means for searching for related information on the web" refers to the function of searching for information on the Internet using a server or search engine and obtaining the necessary data.

"Means of displaying in augmented reality (AR) format" refers to technology for displaying acquired information by overlaying it on real-world images.

"Means for collecting data based on user interests" refers to a function that infers a user's interests from their gaze, voice commands, etc., and collects data related to those interests.

"Means for analyzing collected data" refers to algorithms and methods for analyzing collected data and understanding user interests and behavioral patterns.

"Means for analyzing user emotions" refers to machine learning algorithms that analyze a user's facial expressions and tone of voice to determine their emotional state.

"Smart devices installed in self-driving vehicles" refers to electronic devices with functions such as cameras, microphones, and displays that are used inside self-driving vehicles.

This invention is a system for improving the user experience in autonomous vehicles. It uses smart devices to acquire visual and auditory information from the outside world in real time, and uses this information to search for and display relevant information. It also features the ability to analyze the user's emotions and provide information based on their interests.

First, the smart device, which serves as the terminal, acquires visual and auditory information from the outside world. Specifically, the camera installed on the smart device captures video and the microphone records audio. This data is sent in real time to a processing unit within the terminal.

The device then converts the acquired visual information into text using OCR technology. For example, if an autonomous vehicle recognizes a restaurant sign while driving through town, it will convert it into text such as "Restaurant ABC." Similarly, auditory information is converted into text using voice recognition technology. For example, if a passenger says, "I'm interested in the menu at this restaurant," this will be recorded as text.

Furthermore, the captured video and audio data is analyzed for the user's emotions by the device's built-in emotion engine. The emotion engine uses machine learning algorithms to identify emotions from the user's facial expressions and tone of voice. For example, it can recognize whether the user is interested, happy, or surprised.

The converted text information and recognized emotion data are sent from the device to a cloud server. The cloud server then searches for related information on the web based on the received text information. For example, based on the keyword "restaurant ABC," reviews, menus, and location details can be retrieved using a search engine. Furthermore, by taking emotion data into consideration, it is possible to prioritize retrieving information that is likely to interest the user.

The search results are then sent back to the device, which then displays the retrieved information in AR format. For example, reviews and popular menu items for "Restaurant ABC" are overlaid on the smart device's display. This allows users to access relevant information in real time even while on the move.

The device also collects interest data based on the user's gaze and voice commands. For example, if a user stares intently at a particular piece of information or says something like "this dish looks delicious," that data is recorded. By combining this with emotional data recognized by the emotion engine, more detailed interest analysis becomes possible.

The collected interest and emotion data is sent to a cloud server for analysis. The cloud server uses this data to understand the user's interest trends and reflects this in the next information provided. For example, if the user shows a strong interest in restaurants, information about restaurants can be displayed preferentially from the next time onwards.

As a concrete example

As a specific example, if a user spots a sign for "Restaurant XYZ" while driving around town, the camera on the smart device will recognize the sign and convert it into text information as "Restaurant XYZ" using OCR technology. Furthermore, if the user says, "I'm interested in the menu at this restaurant," speech recognition technology will convert this statement into text. At the same time, an emotion engine will recognize positive emotions from the user's interested facial expression and tone of voice. This information is sent to a cloud server, which then searches for detailed information about "Restaurant XYZ" and prioritizes information that is likely to interest the user. This information is then sent to the device and displayed in AR format on the smart device's display. The user can then check reviews and photos of the menu on the spot.

Example prompt

An example of a prompt is shown below.

"When a user finds a restaurant sign, the smart device identifies the information, converts it into text, and prioritizes displaying reviews and menu information for that restaurant based on the user's sentiment. For example, if a user says, 'I'm interested in the menu at this restaurant,' the device converts that speech into text and retrieves and displays relevant information in real time."

The flow of the specific processing in Application Example 2 will be explained using Figure 14.

Step 1:

The terminal, a smart device, acquires visual and auditory information from the outside world. As the user moves around town, the camera captures video of the surroundings and the microphone records audio from the surroundings. This data is sent to a processing unit in real time. The input is camera video data and audio data, and the output is raw data sent to the processing unit.

Step 2:

The device converts the acquired visual information into text using OCR technology. For example, if a smart device's camera captures a restaurant sign, the video data is converted into text, generating the text information "Restaurant XYZ." The input is video data, and the output is text data.

Step 3:

The device uses voice recognition technology to convert the acquired voice information into text information. For example, if a user says, "I'm interested in the menu at this restaurant," the voice data is converted into text information and related information is generated. The input is voice data, and the output is text data.

Step 4:

The device uses an emotion engine to analyze the user's emotions from the captured video and audio data. For example, it can recognize the emotion "interesting" from the user's facial expression and tone of voice. The input is video and audio data, and the output is emotional data.

Step 5:

The converted text information and recognized emotion data are sent from the device to a cloud server. The server searches for related information on the web based on the received text information. For example, based on the keyword "restaurant XYZ," it retrieves reviews and menu information. The input is text data and emotion data, and the output is related information data.

Step 6:

The cloud server takes emotional data into consideration and prioritizes obtaining information that is likely to interest the user, and then sends the results back to the device. For example, if the user is expressing positive emotions, it will prioritize sending highly rated restaurant reviews. The input is text data and emotional data, and the output is related information data.

Step 7:

The device displays the acquired information in AR format. For example, reviews and popular menu items for "Restaurant XYZ" are overlaid on the smart device's display, allowing the user to view them. The input is related information data, and the output is the displayed AR data.

Step 8:

The device continues to collect interest data based on the user's gaze and voice commands. For example, if the user stares at a particular piece of information or says, "I'm interested in this dish," that data is recorded. The input is gaze data and voice data, and the output is interest data.

Step 9:

The collected interest and emotion data is sent to a cloud server for detailed analysis. The server uses this data to understand the user's interest trends and reflects this in the next information provided. For example, if there is a strong interest in restaurants, information about restaurants will be displayed preferentially from the next time onwards. The input is interest data and emotion data, and the output is analysis data for the next information provided.

The specific processing unit 290 transmits the results of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the results of the specific processing. The microphone 238 acquires audio indicating the user input regarding the results of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

Data generation model 58 is what is known as generative AI (artificial intelligence). Examples of data generation model 58 include generative AI such as ChatGPT (Internet search <URL: https://openai.com/blog/chatgpt>) and Gemini (Internet search <URL: https://gemini.google.com/?hl=ja>). Data generation model 58 is obtained by performing deep learning on a neural network. A prompt containing an instruction is input to data generation model 58, and inference data such as voice data indicating speech, text data indicating text, and image data indicating an image is also input. Data generation model 58 performs inference on the input inference data in accordance with the instructions indicated by the prompt, and outputs the inference results in the form of data such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and/or summarization.

In the above embodiment, an example was given in which the specific processing was performed by the data processing device 12, but the technology disclosed herein is not limited to this, and the specific processing may also be performed by the robot 414.

The emotion identification model 59, which serves as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see Figure 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

Figure 9 shows an emotion map 400 on which multiple emotions are mapped. In emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both emotions and mental states. Emotions that are generally generated from reactions that occur in the brain are arranged on the left side of the concentric circles. Emotions that are generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions that are generally generated from reactions that occur in the brain and are induced by situational judgment are arranged above and below the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the top side of the concentric circles, and the emotion of "discomfort" is arranged on the bottom side. In this way, emotion map 400 maps multiple emotions based on the structure by which emotions are generated, with emotions that tend to occur simultaneously being mapped close together.

These emotions are distributed in the 3 o'clock direction on emotion map 400, and usually fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness takes precedence over internal sensations, resulting in a sense of calm.

The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further out you go on emotion map 400, the more visible (expressed in behavior) the emotion becomes.

Here, human emotions are based on various balances such as posture and blood sugar levels, and when these balances deviate from the ideal, it indicates discomfort, and when they approach the ideal, it indicates pleasure. Emotions can also be created for robots, cars, motorcycles, etc., based on various balances such as posture and remaining battery life, so that when these balances deviate from the ideal, it indicates discomfort, and when they approach the ideal, it indicates pleasure. Emotion maps may be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on speech emotion recognition and emotional brain physiological signal analysis systems, Tokushima University, doctoral dissertation: https://ci.nii.ac.jp/naid/500000375379). The left half of the emotion map is lined with emotions belonging to an area called "reaction," where sensation is dominant. The right half of the emotion map is lined with emotions belonging to an area called "situation," where situational awareness is dominant.

The emotion map defines two emotions that encourage learning. One is the negative emotion around the middle of "repentance" or "reflection" on the situation side. In other words, this is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the positive emotion around "desire" on the response side. In other words, this is when the robot experiences positive feelings such as "I want more" or "I want to know more."

The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are close to each other have similar values, as in the emotion map 900 shown in Figure 10. Figure 10 shows an example in which multiple emotions, such as "relieved," "calm," and "reassuring," have similar emotion values.

The system according to the present disclosure has been described above primarily in terms of the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may also be implemented as a general information processing system. The present disclosure may also be implemented, for example, as a software program that runs on a personal computer or an application that runs on a smartphone, etc. The method according to the present disclosure may also be provided to users in the form of SaaS (Software as a Service).

In the above embodiment, an example was given in which a specific process was performed by a single computer 22, but the technology of the present disclosure is not limited to this, and distributed processing of the specific process may be performed by multiple computers, including the computer 22. For example, the data generation model 58 may be provided in a device external to the data processing device 12, and data may be generated in that external device in response to input data.

In the above embodiment, an example was described in which the specific processing program 56 is stored in the storage 32, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

It is not necessary to store the entire specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entire specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

The following types of processors can be used as hardware resources for executing specific processes. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource for executing specific processes by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which are processors with a circuit configuration designed specifically for executing specific processes. All processors have built-in or connected memory, and all use the memory to execute specific processes.

The hardware resource that executes the specific processing may be composed of one of these various processors, or may be composed of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). The hardware resource that executes the specific processing may also be a single processor.

As an example of a configuration using a single processor, first, there is a configuration in which one processor is configured using a combination of one or more CPUs and software, and this processor functions as a hardware resource that executes specific processing. Second, there is a configuration in which a processor is used to realize the functions of an entire system, including multiple hardware resources that execute specific processing, on a single IC chip, as typified by SoC (System-on-a-chip). In this way, specific processing is realized using one or more of the various processors listed above as hardware resources.

More specifically, the hardware structure of these various processors can be an electrical circuit that combines circuit elements such as semiconductor devices. Furthermore, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps can be added, or the processing order can be rearranged, all within the scope of the spirit of the invention.

The above-described written content and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above explanation of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or substitutions may be made to the above-described written content and illustrations, as long as they do not deviate from the spirit of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described written content and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

All publications, patent applications, and technical standards mentioned in this specification are incorporated by reference herein to the same extent as if each individual publication, patent application, and technical standard was specifically and individually indicated to be incorporated by reference.

The following is further disclosed regarding the above embodiments.

(Claim 1)

A means of acquiring visual information from the outside world,

A means of acquiring auditory information from the outside world,

Means for converting the visual information into text information;

Means for converting the auditory information into text information;

Means for searching for related information on the web based on the converted text information;

Means for displaying the searched information in an augmented reality (AR) format;

Means of collecting data based on user interests,

Means for analyzing the collected data;

A system including

(Claim 2)

The system of claim 1, further comprising means for detecting the user's line of sight.

(Claim 3)

The system of claim 1, further comprising means for recognizing voice commands from the user.

"Example 1"

(Claim 1)

A means of acquiring visual information from the outside world,

A means of acquiring auditory information from the outside world,

Means for converting the visual information into text information;

Means for converting the auditory information into text information;

Means for searching for related information on the web based on the converted text information;

Means for displaying the searched information in an augmented reality (AR) format;

Means of collecting data based on user interests in real time,

Means for analyzing the collected data and reflecting it in subsequent search results;

A system including

(Claim 2)

The system of claim 1, further comprising means for detecting the user's gaze and collecting interest data.

(Claim 3)

The system of claim 1, further comprising means for recognizing the user's voice commands and collecting interest data.

"Application Example 1"

(Claim 1)

A means of acquiring visual information from the outside world,

A means of acquiring auditory information from the outside world,

Means for converting the visual information into text information;

Means for converting the auditory information into text information;

Means for searching for related information on the web based on the converted text information;

Means for displaying the searched information in an augmented reality (AR) format;

Means of collecting data based on user interests,

Means for analyzing the collected data;

A means to provide product information and reviews in real time in AR format while users are walking,

A system including

(Claim 2)

The system of claim 1, further comprising means for detecting the user's line of sight.

(Claim 3)

The system of claim 1, further comprising means for recognizing voice commands from the user.

"Example 2: Combining Emotion Engines"

(Claim 1)

A means of acquiring visual information from the outside world,

A means of acquiring auditory information from the outside world,

Means for converting the visual information into text information;

Means for converting the auditory information into text information;

Means for searching for related information on the web based on the converted text information and emotion data;

Means for displaying the searched information in an augmented reality format;

Means for recognizing user emotions,

Means of collecting data based on user interests,

Means for analyzing the collected data and emotion data;

A system including

(Claim 2)

The system of claim 1, further comprising means for detecting a user's gaze.

(Claim 3)

The system of claim 1, further comprising means for recognizing a user's voice command.

"Application Example 2: Combining Emotion Engines"

(Claim 1)

A means of acquiring visual information from the outside world,

A means of acquiring auditory information from the outside world,

Means for converting the visual information into text information;

Means for converting the auditory information into text information;

Means for searching for related information on the web based on the converted text information;

Means for displaying the searched information in an augmented reality (AR) format;

Means of collecting data based on user interests,

Means for analyzing the collected data;

Means for analyzing user emotions,

A means for displaying relevant information on smart devices installed in autonomous vehicles;

A system including

(Claim 2)

The system of claim 1, further comprising means for detecting the user's line of sight.

(Claim 3)

The system of claim 1, further comprising means for recognizing voice commands from the user.

10, 210, 310, 410 Data processing system 12 Data processing device 14 Smart device 214 Smart glasses 314 Headset type terminal 414 Robot

Claims (3)

A means for acquiring visual information of the outside world;
A means for acquiring auditory information from the outside world;
means for converting the visual information into text information;
means for converting the auditory information into text information;
a means for searching for related information on the web based on the converted text information;
means for displaying the retrieved information in an augmented reality (AR) format;
a means for collecting user interest-based data;
means for analyzing the collected data;
A system including: The system of claim 1, further comprising means for detecting the user's line of sight. The system of claim 1, further comprising means for recognizing voice commands from the user.