JP2026024073A - system - Google Patents

Abstract

A system is provided.
[Solution] A means for learning the appearance and personality of a specific character or person using artificial intelligence technology;
A means of storing the learned data as a non-fungible token (NFT);
means for providing an interface for a user to select a conversation partner and converse with;
a means for conducting real-time voice conversations;
A means of collecting fees from users and distributing the revenue among the parties involved;
A system including:
[Selected Figure] Figure 1

Description

The technology disclosed herein relates to a system.

Patent document 1 discloses a persona chatbot control method executed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot's character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Japanese Patent Application Laid-Open No. 2022-180282

Direct interaction with characters and people is limited by time and geographical constraints, limiting opportunities for many fans. Also, hosting events and meet-and-greets with celebrities and characters requires high costs and manpower, meaning only a limited number of people can enjoy them. Furthermore, there is a risk of reputational damage due to fraudulent use such as deepfakes and impersonation. For this reason, there is a demand for a safe and easy way for fans to interact with their favorite characters and people in real time, 24 hours a day, 365 days a year.

This invention provides a system that uses artificial intelligence technology to learn the appearance and personality of specific characters or people, and stores that data as a non-fungible token (NFT). Specifically, it has a means for providing an interface where users can select a conversation partner and engage in real-time voice conversations. It also has a means for collecting usage fees and sharing revenue, and includes a function that uses the learned artificial intelligence data to generate responses based on the content of the user's conversation and convert them into voice data. This provides an environment where fans can safely and easily interact with their favorite characters or people in real time, 24 hours a day, 365 days a year.

A "character" is a fictional person, animal, or other fictional entity with a particular appearance and personality that appears in stories and media.

"Person" refers to a real person, especially a famous individual or idol with a particular appearance or personality.

"Artificial intelligence technology" refers to algorithms and software technologies used to simulate human intellectual activity.

"Learning" is the process of using data to adjust algorithms and find specific patterns and rules.

A "non-fungible token (NFT)" is a token that uses blockchain technology to guarantee the uniqueness of digital assets and is a means of establishing ownership and transferability.

"Interface" refers to the screen and operating tools that users use to interact with the system.

"Real-time" refers to a state in which processing and responses are carried out immediately without delay.

"Voice conversation" refers to interactive communication using voice.

"Usage Fee" means the fee paid for use of the Service.

"Revenue sharing" refers to the division of revenues among parties based on certain criteria.

"Response" refers to the reaction the system generates in response to input from the user.

1 is a conceptual diagram illustrating an example of the configuration of a data processing system according to a first embodiment. 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. FIG. 10 is a conceptual diagram illustrating an example of the configuration of a data processing system according to a second embodiment. FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. FIG. 10 is a conceptual diagram illustrating an example of the configuration of a data processing system according to a third embodiment. FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. 1 shows an emotion map onto which multiple emotions are mapped. 1 shows an emotion map onto which multiple emotions are mapped. FIG. 2 is a sequence diagram illustrating a processing flow of the data processing system according to the first embodiment. FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in application example 1. FIG. 10 is a sequence diagram showing the flow of processing of the data processing system in the second embodiment when an emotion engine is combined. FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined.

Below, an example of an embodiment of a system relating to the technology disclosed herein will be described with reference to the accompanying drawings.

First, let me explain the terminology used in the following explanation.

In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Furthermore, a processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose Computing on Graphics Processing Units), an APU (Accelerated Processing Unit), etc.

In the following embodiments, coded random access memory (RAM) is memory in which information is temporarily stored and is used by the processor as work memory.

In the following embodiments, the coded storage refers to one or more non-volatile storage devices that store various programs, parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

In the following embodiments, a communication I/F (Interface) with a symbol is an interface that includes a communication processor, an antenna, etc. The communication I/F controls communication between multiple computers. Examples of communication standards that can be applied to the communication I/F include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

In the following embodiments, "A and/or B" is synonymous with "at least one of A and B." In other words, "A and/or B" means that it may be just A, just B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and/or B" also applies when three or more things are expressed connected by "and/or."

[First embodiment]

Figure 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

As shown in FIG. 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

The data processing device 12 includes a computer 22, a database 24, and a communication I/F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and communication I/F 26 are also connected to the bus 34. The communication I/F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and/or a LAN (Local Area Network).

The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I/F 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

The reception device 38 is equipped with a touch panel 38A, a microphone 38B, etc., and receives user input. The touch panel 38A detects contact with an indicator (e.g., a pen or finger) to receive user input via the indicator. The microphone 38B detects the user's voice to receive user input via voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

The output device 40 is equipped with a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible by the user 20 (e.g., audio and/or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

The communication I/F 44 is connected to the network 54. The communication I/Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54.

Figure 2 shows an example of the main functions of the data processing device 12 and smart device 14.

As shown in FIG. 2, in the data processing device 12, specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

Storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

Next, we will explain the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following explanation, the data processing device 12 will be referred to as the "server" and the smart device 14 will be referred to as the "terminal."

This invention is a system that uses artificial intelligence technology to learn the appearance and personality of specific characters or people, stores that learning data as non-fungible tokens (NFTs), and enables real-time voice conversations with users 24 hours a day, 365 days a year. The system is configured as follows:

System Configuration

1. Learning with AI technology

The server collects data on the appearance, personality, and past behavior of specific characters or individuals, and uses artificial intelligence algorithms to learn this information.

The learning results include the character's specific behavioral and response patterns and are saved as a highly accurate model.

2. Save as an NFT

The server stores the learned AI data as an NFT, using blockchain technology to guarantee its uniqueness and ownership.

3. Providing an interface

The device provides the user with an interface for selecting a conversation partner. Users can log in and select their favorite character.

4. Real-time voice conversation

The user performs an operation to start a conversation with the selected character. The device captures the user's voice, converts it into text data, and sends it to the server.

Based on the text data received by the server, an appropriate response is generated using an AI model. This response is then converted back into audio data and sent to the device.

The device plays the audio data and the conversation takes place in real time.

5. Usage Fee Collection and Revenue Sharing

The server collects usage fees from users. Payment processing is carried out through the terminal.

Revenues will be shared among the parties involved, including copyright holders of characters and people, and service providers.

Specific examples

1. The user logs in and selects a character.

The user opens the app and logs in by entering "user123" and "password123" on the login screen. The device sends the entered data to the server and authentication is performed. If authentication is successful, the home screen is displayed on the device.

The user presses the "Select Character" button on the home screen and selects, for example, "Character A." The device sends the selected character information to the server, and the server retrieves the corresponding AI data and sends it to the device.

2. Initiating and conducting actual conversations

The user presses the "Start Conversation" button and speaks "Hello, Character A" into the device's microphone. The device converts the speech into text and sends it to the server.

The server analyzes the text data and generates an appropriate response based on the AI model. For example, the response generated might be, "Hello, I'm Character A. How's your day?"

The generated response is converted into audio data and sent to the device. The device plays the audio data, allowing the user to hear the response.

3. Payment of fees and revenue sharing

When a user pays a usage fee, the terminal captures the payment information and sends it to the server. The server processes the payment using a payment gateway and, after confirmation, distributes the revenue among the parties.

This system allows fans to safely and easily interact with their favorite characters and people 24 hours a day, 365 days a year, and also reduces the high costs associated with hosting events and inviting celebrities to attend.

The processing flow is explained below.

Step 1:

The user opens the application. The user accesses the login screen and enters their ID and password.

Step 2:

The device receives the user's input and sends the ID and password information to the server. The communication is encrypted.

Step 3:

The server checks the received ID and password against a database and authenticates the user. If the ID and password do not match, an error message is sent to the terminal.

Step 4:

If the server successfully authenticates, it generates session information and sends it back to the device. The device uses the received session information to display the home screen.

Step 5:

The user presses the "Character Selection" button on the home screen and selects the character they want to interact with.

Step 6:

The device receives the user's selection and sends the character ID to the server. The server then retrieves the AI model data for the selected character.

Step 7:

The server sends the acquired AI model data to the device. The device then displays a conversation interface with the character based on the received data.

Step 8:

The user presses the "Start conversation" button to begin the conversation. The device's microphone is turned on to capture the user's voice.

Step 9:

The device records the user's voice and converts the voice data into text data in real time. This text data is then sent to the server.

Step 10:

The server analyzes the received text data and generates a response using an AI model. For example, if the user says "Hello, Character A," the response will be "Hello! How's your day?"

Step 11:

The response text generated by the server is converted into audio data and sent to the device.

Step 12:

The device plays back the received audio data and provides a response to the user. The user can then listen to this and ask questions or make comments.

Step 13:

If the user needs to pay a usage fee, the terminal will display a payment screen. The user will then enter their credit card information, etc.

Step 14:

The terminal sends the entered payment information to the server, which then processes the payment through the payment gateway.

Step 15:

The server sends information about the success or failure of the payment to the terminal. If successful, the usage rights are updated.

Step 16:

The terminal will notify the user of the payment result, allowing them to continue using the system.

(Example 1)

Next, we will explain Example 1. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

In conventional dialogue systems with characters or people, it was difficult to achieve real-time voice conversation, and users were unable to receive a satisfactory experience. Furthermore, there was a lack of means to guarantee the uniqueness and ownership of trained AI data, creating a risk of data tampering or unauthorized use. Furthermore, there was no established mechanism for collecting usage fees from users and distributing revenue fairly among the parties involved. New technology is needed to solve these problems.

The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

In this invention, the server includes: means for learning the appearance and personality of a specific character or person using artificial intelligence technology; means for storing the learned data as a non-fungible token (NFT); means for providing an interface for users to select a conversation partner and converse with them; means for acquiring the user's voice and converting it into text data using voice recognition technology; means for generating an appropriate response based on the text data using an AI model and converting the response into voice data; means for transmitting the converted voice data to the user's terminal to conduct voice conversations in real time; and means for collecting usage fees from users and distributing revenue among the parties involved. This enables real-time voice conversations, guarantees the uniqueness and ownership of learned data, and enables fair revenue distribution.

"Specific characters or people" refers to fictional or real people that users select when interacting, whose appearance, personality, behavior patterns, etc. are predefined.

"Artificial intelligence technology" refers to technology that uses machine learning and deep learning models to learn from specific data and generate appropriate responses even for unknown data.

A non-fungible token (NFT) is a unique digital asset created using blockchain technology with guaranteed ownership and authenticity.

"Interface" refers to the user interface provided to users to select and interact with characters or people, and is usually provided as an application.

"Speech recognition technology" refers to technology that converts a user's voice into text data, and includes services such as Google Cloud Speech-to-Text and Microsoft Azure Speech Recognition Services.

An "AI model" is a neural network model trained on collected data and used to generate responses for specific characters or people.

"Converting into voice data" means using speech synthesis technology to convert the text response generated by the AI model into voice data that users can hear.

"Real-time voice conversation" refers to a form in which users can instantly communicate back and forth with characters or people using voice.

"Collecting usage fees" means collecting fees from users for using a service, often using an online payment system.

"Distributing revenues among the parties involved" means distributing the collected usage fees in fair proportions to copyright holders of characters and people, service providers, and other rights holders.

Form for implementing the invention

This invention is a system that uses artificial intelligence technology to learn the appearance and personality of specific characters or people, stores that learning data as non-fungible tokens (NFTs), and enables real-time voice conversations with users 24 hours a day, 365 days a year. The system is configured as follows:

Learning using artificial intelligence technology

The server collects data on the appearance, personality, and past behavior of specific characters or people. For example, it uses scripts that retrieve data from YouTube, social media, etc. This data is trained using the TensorFlow or PyTorch machine learning frameworks. During the data preprocessing process, noise removal, tokenization, normalization, etc. are performed, and a neural network model is built. The trained model is then evaluated for accuracy and saved if it is highly accurate.

Stored as an NFT

The server converts the model data into ONNX format to store the trained AI data as an NFT. The NFT is then issued using a blockchain platform such as Ethereum or Binance Smart Chain, which incorporates the model's unique identifier and metadata in the process.

Providing an interface

The device provides the user with an interactive user interface. The user logs in using an application developed with React Native or Flutter. The login screen has username and password input fields, and the user authenticates. If authentication is successful, a character selection screen is displayed, where the user selects the character they want to interact with.

Real-time voice conversation

The user initiates a conversation with the selected character. The device uses the microphone to capture the user's voice and converts it into text using the Google Cloud Speech-to-Text API or Microsoft Azure Speech Recognition Service. The text is sent to a server, which uses an AI model (e.g., GTP-3 or BERT) to generate an appropriate response. This response is converted into audio data by a speech synthesis engine (e.g., Amazon Polly or Google Text-to-Speech) and sent to the device. The device plays back the audio data, enabling real-time conversation.

Collection of usage fees and revenue sharing

The usage fee paid by users when using the service is collected by the device acquiring payment information. Payment processing is carried out by the server using the Stripe or PayPal API. The collected usage fee is distributed fairly among the copyright holders of the characters and people and the service provider.

Specific examples

1. User login and character selection

The user opens the app and logs in by entering "user123" and "password123" on the login screen. The device sends the input data to the server and authentication is performed. If authentication is successful, the home screen is displayed.

The user presses the "Character Selection" button on the home screen and selects "Character A." The device sends the selected character information to the server, and the server retrieves the corresponding AI data and sends it to the device.

2. Initiating and conducting actual conversations

The user presses the "Start Conversation" button and speaks "Hello, Character A" into the device's microphone. The device converts the speech into text and sends it to the server.

The server analyzes the text data and generates an appropriate response based on the AI model. For example, the response generated might be, "Hello, I'm Character A. How's your day?"

The generated response is converted into audio data and sent to the device. The device plays the audio data, allowing the user to hear the response.

Example prompt

"Hello, I'm Character A. How's your day going?"

This system allows fans to safely and easily interact with their favorite characters and people 24 hours a day, 365 days a year, and also reduces the high costs associated with hosting events and inviting celebrities to attend.

The flow of the identification process in Example 1 will be explained using Figure 11.

Step 1: Data Collection

The server collects data on the appearance, personality, and past behavior of specific characters or people. Inputs include links to YouTube videos and URLs of social media posts. The data is obtained using scraping technology and APIs. The output is the obtained text, images, and audio data.

Step 2: Data Preprocessing

The server preprocesses the collected data. The raw data collected in step 1 is provided as input. Specifically, processes such as noise removal, text tokenization, and normalization are performed. For example, unnecessary special characters are removed from the text data and it is converted into a standard format. The output is the preprocessed data.

Step 3: Train the model

The server uses the preprocessed data to train an artificial intelligence model (e.g., TensorFlow or PyTorch). The preprocessed data from step 2 is provided as input. Specifically, the data is divided into batches and the model parameters are optimized using a GPU. The output is a trained, highly accurate model.

Step 4: Evaluate and save the model

The server evaluates and saves the trained model. The model trained in step 3 and the validation dataset are used as input. Specifically, the accuracy of the model is evaluated using the validation data, and if the accuracy meets the criteria, the model is saved. The output is the evaluation results and the saved model.

Step 5: Save as an NFT

The server saves the trained AI model as an NFT. The model data saved in step 4 is used as input. Specifically, the model is converted to ONNX format and an NFT is issued on a blockchain platform (e.g., Ethereum or Binance Smart Chain). The output is the model data saved as an NFT.

Step 6: User authentication and character selection

The device displays a login screen for the user and performs authentication. A username and password are provided as input. The authentication data is sent to the server and checked against a database. If authentication is successful, the character selection screen is displayed. The output is information about the selected character.

Step 7: Capture and convert audio input

The device uses a microphone to capture the user's voice. The user's voice data is provided as input. Specifically, the voice is converted into text data using the Google Cloud Speech-to-Text API. The output is text data.

Step 8: Generate a response

The server receives the text data and generates a response using an AI model. The text data converted in step 7 is provided as input. Specifically, the AI model (e.g., GPT-3 or BERT) analyzes the text data and generates an appropriate response. The output is the text data of the generated response.

Step 9: Convert to audio and send

The server converts the generated text response into audio data and sends it to the device. The text data generated in step 8 is provided as input. Specifically, it converts it into audio data using Amazon Polly or Google Text-to-Speech. The output is audio data.

Step 10: Play real-time audio conversations

The device receives the audio data and plays it on the speaker. The input is the audio data sent in step 9. Specifically, the device decodes the audio data and plays it on the speaker so that it can be heard by the user. The output is audio played in real time.

Step 11: Collection and distribution of usage fees

The terminal obtains payment information to collect usage fees and sends it to the server. Payment information is provided as input. The server processes the payment using Stripe or PayPal APIs and distributes revenue among the parties involved. Specifically, after confirming the payment process, the revenue is distributed via bank transfer or cryptocurrency. The output is the distributed revenue.

(Application Example 1)

Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the smart device 14 will be referred to as the "terminal."

Currently, providing high-quality customer service to customers visiting physical stores requires a lot of labor and resources. It is also difficult to provide detailed information about specific products in the store, making it difficult to increase customer satisfaction. Furthermore, while there is a demand for personalized guidance that allows customers to select products based on their preferences and interests, achieving this is complex. To solve these challenges, a system that can interact with customers in real time and provide personalized guidance is needed.

The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

In this invention, the server includes means for learning the characteristics of a specific virtual character using machine learning technology, means for saving the learned information as a non-fungible character (NFT), means for providing an operation screen on which a user can select a conversation partner and conduct a conversation, means for real-time voice communication, means for receiving service fees from users and distributing revenue among relevant parties, means for providing guidance using a display device that supports conversations between customers and virtual characters in physical stores, and means for users to obtain product information in the store and receive support through the character. This makes it possible to improve the customer experience in physical stores and provide personalized, high-quality customer service.

"Characteristics of a specific virtual person" refers to data including information such as the appearance, personality, and past words and actions of a character or celebrity.

"Machine learning technology" is a technology that allows computers to automatically learn from data and make predictions and classifications.

A "Non-Fungible Trading Card (NFT)" is a digital asset whose uniqueness and ownership are guaranteed using blockchain technology.

An "operation screen" is a display screen that provides an interface for users to select a conversation partner and conduct a conversation.

"Real-time voice communication" refers to technology that allows users to instantly communicate via voice with a character of their choice.

"Service fee" refers to the amount of money paid by the user in consideration for using the service.

"Related parties" refers to people or organizations that are subject to revenue sharing, such as character or system providers and developers.

"Display devices" refer to hardware for displaying information, such as displays installed in physical stores, smart glasses, and head-mounted displays (HMDs).

"Supporting virtual character conversations" refers to a specific virtual character communicating with the user via voice, providing guidance and information.

"Get product information" means viewing or obtaining detailed information about products in the store.

"Receiving support through a character" means that a virtual character guides or explains products or services to the user.

To implement this invention, the following system configuration is required. The system consists of a server, a terminal, and a user as its main components.

1. System Programming

The server builds a database that uses machine learning technology to learn the characteristics of specific virtual characters. The terminal provides an operation screen that allows the user to select a conversation partner and engage in real-time voice communication with that virtual character. The server also stores the learned information as a non-fungible character (NFT).

The terminal displays an operation screen that allows the user to select a virtual person to interact with, and transmits the selection information to the server. When the user communicates via voice in real time, the terminal converts the user's voice into text information and transmits that text information to the server.

The server feeds the received text information into a machine learning model to generate a conversation-based response. The generated response is converted into audio data by the server and sent to the device. The device plays the audio data and provides the virtual character's response to the user.

2. Processing Description

The server is implemented using programming languages such as Python. Machine learning techniques include the NLP (natural language processing) model GPT-3. Libraries such as Google Speech Recognition are used for voice recognition and pyttsx3 for voice synthesis.

The device includes hardware such as smart glasses, a head-mounted display (HMD), and a microphone. The device recognizes the user's voice and sends it to the server. The server converts the voice into text and generates an appropriate response using an NLP model. The server then converts the response into audio data and sends it back to the device.

As a specific example, a user enters a physical store and puts on a pair of smart glasses. When the user says, "What products do you recommend?", the device's microphone picks up the user's voice and sends it to the server. The server analyzes the text data and generates a response using GPT-3. The response might be something like, "Currently, we recommend this smartwatch. It's the latest model with a long battery life and comprehensive health management features."

The generated response is converted into audio data and sent to the device. The device then plays the audio data, allowing the user to hear the voice of the virtual character.

3. Specific examples and prompt sentence examples

As a practical example, consider a situation where a user wears smart glasses and asks for assistance in a store. For example, the user might say, "Hello, what products do you recommend?"

Example prompt:

Customer Question: "Hello, what products do you recommend?"

Prompt for generative AI model: "A customer asked you for a product recommendation. How would you respond?"

The flow of the specific processing in Application Example 1 will be explained using Figure 12.

Program processing steps

Step 1:

The user puts on the smart glasses and launches the application. The input is the smart glasses' initialization data, and the output is the display of a login screen. Specifically, the device displays the login screen to the user, and the user enters their login information.

Step 2:

The user enters login information and presses the login button. The input is login information, and the output is an authentication request to the server. The device sends the user's input information to the server, and the server performs authentication.

Step 3:

The server authenticates the user's login information and returns the result to the device. The input is the user's login information, and the output is the authentication result. Specifically, the server searches the database and confirms the login information. If successful, the home screen is displayed on the device.

Step 4:

The user selects a character to interact with from the home screen. The input is character selection information, and the output is sending the selection information to the server. The device sends the selected character information to the server, and the server obtains the corresponding learning data.

Step 5:

The server obtains the learning data for the selected character and sends it to the device. The input is the character selection information, and the output is the learning data. The server retrieves the relevant data from the learning database and sends it to the device.

Step 6:

The user presses the "Start conversation" button and speaks into the device's microphone. The input is the user's voice, and the output is text data. The device uses voice recognition to convert the user's voice into text and sends the text data to the server.

Step 7:

The server receives the user's text data and generates an appropriate response using a generative AI model. The input is the user's text data and the output is the generated response. The server generates the response using an NLP model (e.g., GPT-3).

Step 8:

The response generated by the server is converted into audio data and sent to the device. The input is the generated response and the output is audio data. The server uses a speech synthesis library (e.g., pyttsx3) to generate the audio data and send it to the device.

Step 9:

The device plays back the audio data received from the server and lets the user listen. The input is audio data, and the output is audio playback. The device plays back the audio data, allowing the user to hear the virtual character's response.

Step 10:

When the user wants to end the service, they perform an operation to end the application. The input is the end operation, and the output is to end the application. The device will end the application and stop displaying the smart glasses.

Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

This invention is a system that uses artificial intelligence technology to learn the appearance and personality of specific characters or people, stores that learning data as non-fungible tokens (NFTs), and enables real-time voice conversations with users 24 hours a day, 365 days a year. Furthermore, by combining this invention with an emotion engine, it is possible to recognize the user's emotions and provide adaptive responses based on those emotions.

System Configuration

1. Learning with AI technology

The server collects data on the appearance, personality, and past behavior of specific characters or people, and uses an artificial intelligence algorithm to learn this information. The learning results, including the character's specific behavior and response patterns, are saved as a highly accurate model.

2. Save as an NFT

The server stores the learned AI data as an NFT, using blockchain technology to guarantee its uniqueness and ownership.

3. Providing an interface

The device provides the user with an interface for selecting a conversation partner. Users can log in and select their favorite character.

4. Real-time voice conversation

The user performs an operation to start a conversation with the selected character. The device captures the user's voice, converts it into text data, and sends it to the server.

Based on the text data received by the server, an appropriate response is generated using an AI model. This response is then converted back into audio data and sent to the device.

The device plays the audio data and the conversation takes place in real time.

5. Emotional Engine Integration

The server is equipped with an emotion engine that uses the user's voice data to analyze emotions. The emotion engine analyzes the user's emotional state (e.g., joy, sadness, anger, etc.) from the voice data.

Based on the results of emotion analysis, the AI model adaptively changes the response content. For example, if the user speaks in a sad voice, a response such as "Cheer up!" will be generated.

6. Usage Fee Collection and Revenue Sharing

The server collects usage fees from users. Payments are processed through the terminal. Revenues are distributed among the parties involved, including the copyright holders of characters and people, and service providers.

Specific examples

1. The user logs in and selects a character.

The user opens the app and logs in by entering "user123" and "password123" on the login screen. The device sends the entered data to the server and authentication is performed. If authentication is successful, the home screen is displayed on the device.

The user presses the "Select Character" button on the home screen and selects, for example, "Character A." The device sends the selected character information to the server, and the server retrieves the corresponding AI data and sends it to the device.

2. Initiating and conducting actual conversations

The user presses the "Start Conversation" button and speaks "Hello, Character A" into the device's microphone. The device converts the speech into text and sends it to the server.

The server analyzes the text data and generates an appropriate response based on the AI model. For example, the response generated might be, "Hello! How's your day?"

At the same time, the server sends the voice data to the emotion engine, which analyzes the user's emotional state.

The emotion engine analyzes the user's emotional state and feeds the results back to the AI model. For example, if the user sounds sad, the response will be changed to an encouraging message such as "Cheer up!"

The generated response is converted into audio data and sent to the device, which then plays the audio data and provides the response to the user.

3. Payment of fees and revenue sharing

If the user needs to pay a usage fee, the terminal will display a payment screen. The user will enter their credit card information, etc.

The terminal sends the entered payment information to the server, which processes the payment through the payment gateway.

The server sends information about the success or failure of the payment to the terminal. If successful, the usage rights are updated.

The terminal will notify the user of the payment result, allowing them to continue using the system.

This system allows fans to safely and easily interact with their favorite characters and people 24 hours a day, 365 days a year, and by providing appropriate responses based on the user's emotions, it enables more personal and intimate communication.

The processing flow is explained below.

Step 1:

The user opens the application. The user accesses the login screen and enters their ID and password.

Step 2:

The device receives the user's input and sends the ID and password information to the server. The communication is encrypted.

Step 3:

The server checks the received ID and password against a database and authenticates the user. If the ID and password do not match, an error message is sent to the terminal.

Step 4:

If the server successfully authenticates, it generates session information and sends it back to the device. The device uses the received session information to display the home screen.

Step 5:

The user presses the "Character Selection" button on the home screen and selects the character they want to interact with.

Step 6:

The device receives the user's selection and sends the character ID to the server. The server then retrieves the AI model data for the selected character.

Step 7:

The server sends the acquired AI model data to the device. The device displays a conversation interface with the character based on the received data.

Step 8:

The user presses the "Start conversation" button to begin the conversation. The device's microphone is turned on to capture the user's voice.

Step 9:

The device records the user's voice and converts the voice data into text data in real time. This text data is then sent to the server.

Step 10:

The server analyzes the received text data and generates a response using an AI model. For example, if the user says "Hello, Character A," the response will be "Hello! How's your day?"

Step 11:

At the same time, the server sends the user's voice data to the emotion engine, which analyzes the user's emotional state. The emotion engine analyzes the voice data and identifies the user's emotional state (joy, sadness, anger, etc.).

Step 12:

The emotion engine sends the analysis results back to the server, which then adaptively changes the response based on the results. For example, if the user speaks in a sad voice, the response will be "Cheer up!"

Step 13:

The server converts the response text into audio data and sends it to the device.

Step 14:

The device plays back the received audio data and provides a response to the user. The user can then listen to this and ask questions or make comments.

Step 15:

If the user needs to pay a usage fee, the terminal will display a payment screen. The user will then enter their credit card information, etc.

Step 16:

The terminal sends the entered payment information to the server, which then processes the payment through the payment gateway.

Step 17:

The server sends information about the success or failure of the payment to the terminal. If successful, the usage rights are updated.

Step 18:

The terminal will notify the user of the payment result, allowing them to continue using the system.

(Example 2)

Next, we will explain Example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

With conventional voice conversation systems using artificial intelligence, it was difficult to efficiently learn the appearance and personality of characters or people and conduct real-time voice conversations with users. Furthermore, technology for providing adaptive responses based on the user's emotions was not sufficiently developed, making it impossible to achieve natural, personal communication with users.

The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

In this invention, the server includes means for learning the appearance and personality of a specific character or person using artificial intelligence technology, means for saving the learned data as a non-fungible token (NFT), and means for analyzing emotions from the user's voice and adaptively changing responses based on the analysis results. This makes it possible to learn the characteristics of characters and people with high accuracy, safely store that data, and provide flexible responses that correspond to the user's emotions.

"Specific characters or persons" refers to fictional or real people or characters that users can select as their conversation partners.

"Artificial intelligence technology" refers to technology that uses techniques such as machine learning and deep learning to extract patterns from data and make predictions and classifications.

"Means of learning" refers to the process of collecting data and generating an AI model based on that data.

A "non-fungible token (NFT)" is a token that uses blockchain technology to guarantee the uniqueness and ownership of digital data.

"Means of providing an interface" refers to the method of providing screens and menus that allow users to access and operate the system.

"Means for conducting real-time voice conversation" refers to technology that allows users and characters to have instant voice conversations.

"Means for analyzing emotions from a user's voice" refers to technology that analyzes a user's emotional state based on their voice data.

"Means for adaptively changing responses" refers to a method for changing the AI's response content based on the results of emotion analysis.

"Means for collecting usage fees and distributing revenue among parties" refers to a system for collecting service usage fees from users and distributing the revenue obtained among parties.

Form for implementing the invention

This invention is a system that uses artificial intelligence technology to learn the appearance and personality of specific characters or people, stores that learning data as non-fungible tokens (NFTs), and provides users with real-time voice conversations 24 hours a day, 365 days a year. Furthermore, by combining it with an emotion engine, it is possible to recognize the user's emotions and provide adaptive responses based on those emotions.

System Overview

The system consists of the following main components:

1. Server

2. Device

3. User

Server Role

1. Training data collection and AI model generation

The server collects data on the appearance, personality, and past behavior of specific characters and people. This data includes video, audio files, text data, etc.

The collected data is trained using an AI algorithm (e.g., TensorFlow or PyTorch). During the initial training stage, data preprocessing (noise removal, text normalization, etc.) is performed.

Save the completed model in the database.

2. Data storage as NFTs

The learned AI data is stored as a non-fungible token (NFT). This process uses blockchain technology (e.g., Ethereum).

The server registers the generated NFT on the blockchain and maintains owner information.

3. Sentiment Analysis and Response Adaptation

The server uses an emotion engine (for example, IBM Watson's emotion recognition API) to analyze the user's voice data. Based on the analysis results, the AI model adaptively changes the response content.

For example, if a user says, "I'm feeling a bit down today," emotion recognition can generate a response such as, "Cheer up!"

4. Usage fee collection and revenue sharing

The server collects the usage fee and manages the payment process. Payment is made using credit card information entered by the user on the terminal.

Revenues will be distributed among the parties involved (character copyright holders, service providers, etc.).

Device role

1. Providing an interface

The device provides the user with an interface to select a conversation partner and conduct a conversation. The user launches the app, enters their authentication information on the login screen, and logs in.

After logging in, users can press the "Select Character" button to choose their preferred character and begin a conversation with that character.

2. Audio data processing

The device captures the user's voice and converts it into text data using a speech recognition engine such as Google Speech-to-Text.

The converted text data is sent to the server, and after an appropriate response is generated, it is converted back into voice data and sent to the device.

The device plays the audio data, enabling real-time conversations.

User Roles

1. Log in and select a character

The user launches the app and enters authentication information such as "user123" and "password123" on the login screen.

Once authentication is successful, select a character from the home screen and begin a conversation with the character you select.

2. Actual conversation progression

The user presses the "Start Conversation" button and speaks into the device's microphone, saying "Hello, Character A."

The device converts this voice into text data and sends it to the server. The server uses an AI model to generate a response based on the text data and sends an adaptive reply.

For example, you can receive responses such as "Hello! How are you today?" or adaptive messages such as "Cheer up!"

Examples and prompts

"Hello, Character A. How was your day?"

"I've been feeling a bit down lately. Can you cheer me up?"

"What do you think about today's news?"

By using these prompts, users can enjoy more personal and natural conversations with the characters.

The flow of the identification process in Example 2 will be explained using Figure 13.

Program processing flow

Step 1:

Training data collection and AI model generation

The server collects data on the appearance, personality, and past behavior of specific characters or people. The data used for this collection includes text, audio files, images, and videos. Specifically, the data is retrieved from a database via web scraping or API and stored on the server.

Input: Data about the character's appearance, personality, and past words and actions.

Data processing: noise removal, text normalization, data cleaning.

Output: Clean dataset after preprocessing.

The server uses an AI algorithm (such as TensorFlow or PyTorch) to train a model using this data. After training, the model reflects the character's behavior and response patterns.

Input: Clean dataset after preprocessing.

Data computation: Applying machine learning algorithms and training models.

Output: Trained AI model.

Step 2:

Storing data as NFTs

The server stores the learned AI data as a non-fungible token (NFT).

Input: Trained AI model.

Data processing: Blockchain technology (e.g., Ethereum) is used to ensure the uniqueness and ownership of the model for NFT conversion.

Output: NFT registered on the blockchain.

The server registers the generated NFT on the blockchain and retains owner information. Specifically, the NFT is registered through the blockchain API.

Step 3:

User login and character selection

The user launches the app and enters their authentication information (e.g., "user123" and "password123") on the login screen.

The device sends the entered authentication information to the server and performs authentication.

Input: User authentication information.

Data calculation: Compare with the database and determine whether authentication is successful.

Output: Authentication success or failure status.

If authentication is successful, the home screen will be displayed on the device. The user presses the "Select Character" button to choose the character they want to interact with.

Input: Character information selected by the user.

Data processing: Search and obtain the AI model corresponding to the selected character.

Output: AI data for the selected character.

Step 4:

Real-time voice conversation processing

The user presses the "Start conversation" button and speaks into the device's microphone.

The device captures the user's voice and converts it into text data using Google Speech-to-Text.

Input: User's voice data.

Data processing: voice-to-text conversion.

Output: Text data.

The converted text data is sent to the server, which generates an appropriate response.

Input: User's text data.

Data calculations: Use AI models to generate appropriate responses.

Output: The generated text response.

The response is converted back into audio data and sent to the device.

Input: The generated text response.

Data processing: converting text to audio.

Output: Generated audio data.

The device plays the audio data and the conversation takes place in real time.

Step 5:

Sentiment analysis and response adaptation

The server analyzes the user's voice data using an emotion engine (e.g., IBM Watson's emotion recognition API).

Input: User's voice data.

Data calculation: Analyze the user's emotional state from voice data.

Output: User's emotional state data.

Based on the results of emotion analysis, the AI model adaptively changes the response content.

Input: User emotional state data and generated text response.

Data calculation: Change the response content to match the emotional state.

Output: The final adapted response text.

For example, if the user sounds sad, the response will be changed to something like "Cheer up!"

Step 6:

Collecting usage fees and processing payments

If the user needs to pay a usage fee, the terminal will display a payment screen. The user will enter their credit card information, etc.

Input: User's payment information.

The terminal sends payment information to the server, which processes the payment through a payment gateway (e.g., Stripe).

Data calculation: Executes payment processing based on payment information.

Output: Payment success or failure status.

The server notifies the terminal of the success or failure of the payment, and if successful, the usage rights are updated.

The terminal will notify the user of the payment result, allowing them to continue using the system.

(Application Example 2)

Next, we will explain Application Example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the smart device 14 will be referred to as the "terminal."

The objective of this invention is to provide a system that can reproduce the appearance and personality of a specific character or person in real time, while providing appropriate responses based on the user's emotions through dialogue with them. Furthermore, the aim is to realize a more personal and sympathetic shopping experience by having characters act as shopping assistants in virtual stores, explaining products and recommending items.

The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

In this invention, the server includes: means for learning the appearance and personality of a specific character or person using artificial intelligence technology; means for storing the learned data as a non-fungible token (NFT); means for providing an interface for users to select a conversation partner and converse with them; means for conducting real-time voice conversations; means for collecting usage fees from users and distributing revenue among the parties involved; means for analyzing the emotional state from the user's voice data using an emotion engine and generating an adaptive response based on the analysis results; means for converting the generated response into voice data and providing it to the user as voice; and means for the character to act as a shopping assistant in a virtual store, explaining products and recommending items. This enables real-time, emotion-based dialogue with a specific character and a personalized, sympathetic shopping assistant experience in a virtual store.

"Specific characters or people" are virtual beings whose appearances and personalities are reproduced using artificial intelligence technology.

"Artificial intelligence technology" is a technology that learns the appearance and personality of a specific character or person, and generates responses and actions based on that information.

A non-fungible token (NFT) is a means of using blockchain technology to uniquely identify a specific digital asset and prove ownership of it.

An "interface" is a device or software that provides input and display means for a user to select a conversation partner and initiate a conversation.

"Means for conducting real-time voice conversations" refers to technology that allows users to have instant voice conversations with specific characters or people.

An "emotion engine" is an algorithm and system that analyzes a user's emotional state from their voice data and generates a response based on the analysis results.

An "adaptive response" is a response that changes depending on the user's speech content and emotional state, and is generated individually and dynamically.

A "virtual store" is a virtual shopping environment that exists on the Internet and is a platform that allows users to conduct purchasing activities in a virtual space.

A "shopping assistant" is a virtual character or person who explains products and recommends items to users in a virtual store.

"Collection" means receiving usage fees from users.

"Means for distributing revenues among parties involved" refers to a mechanism for appropriately distributing revenues earned through system operation to parties involved, such as character copyright holders and service providers.

System Overview

This invention is a system that recreates specific characters or people in real time and engages in voice conversations with users. It primarily functions as a shopping assistant in virtual stores. This system consists of three elements: a server, a terminal, and a user.

Program structure and various technologies

Speech recognition engine

The server receives the user's voice input and converts the voice into text data using a speech recognition engine (Google Speech-to-Text API).

Sentiment analysis engine

The server performs emotion analysis on the acquired text data using IBM Watson Tone Analyzer, thereby determining the user's emotional state.

Dialogue management system

The server uses Rasa to generate responses based on the user's conversation content and sentiment analysis results.

Speech synthesis engine

The server converts the generated response into audio data using the Google Text-to-Speech API and sends it to the device.

Program processing

1. Voice input and text conversion

When a user speaks into their smartphone's microphone, the audio is picked up by the device and converted into text data using the Google Speech-to-Text API.

2. Emotion analysis

The transferred text data is subjected to emotion analysis using IBM Watson Tone Analyzer to determine the user's emotional state (e.g., joy, sadness, anger, etc.).

3. Response Generation

Rasa is used to generate appropriate responses based on analysis results and conversation content. These responses are adapted to the user's emotions.

4. Conversion to audio data and transmission

The generated response is converted into audio data using the Google Text-to-Speech API, and this audio data is sent to the device, enabling real-time conversation.

Specific examples

1. User login and character selection

The user launches the app and logs in by entering "user123" and "password123." After logging in, they select "Character A" on the character selection screen.

Example prompt:

User: Hello, I'm looking for clothes today.

---

Assistant: Hello, what kind of clothes are you looking for?

---

User: I'd like a casual shirt.

---

2. Voice input and analysis

When a user says, "Hello, I'm looking for clothes today," the speech is captured and converted into text data. The text data is sent to IBM Watson Tone Analyzer for sentiment analysis.

3. Response generation and delivery

Based on sentiment analysis and the conversation content, Rasa generates a response: "Hello, what kind of clothes are you looking for?" This response is converted into audio using the Google Text-to-Speech API and sent to the device.

4. Shopping Assistant Function

If a user says, "I want a casual shirt," the assistant will suggest, "How about this shirt? It's a trendy design and very popular these days," and explain the product. It is also possible to provide more detailed product descriptions and suggestions based on the results of user sentiment analysis.

This allows the invention to provide emotion-based real-time interactions with specific characters and a personalized, immersive shopping assistant experience in virtual stores.

The flow of the specific processing in Application Example 2 will be explained using Figure 14.

Step 1:

The user launches the app and logs in. The user enters their username and password, and the device sends that information to the server. The server performs authentication, and if successful, displays the character selection screen on the device.

Enter: User's username and password

Process: The server authenticates the user.

Output: Upon successful authentication, the character selection screen will be displayed.

Step 2:

The user selects a character. The user chooses a specific character on the character selection screen, and the device sends the selection information to the server. The server loads the selected character's AI model and sends the relevant data to the device.

Input: User's character selection information

Processing: The server loads the AI model and retrieves relevant data.

Output: Character information is displayed on the device.

Step 3:

The user begins the conversation. They speak into the smartphone's microphone, and the device picks up the audio. The speech is converted into text data using a speech recognition engine (Google Speech-to-Text API), and the text data is sent to the server.

Input: User's voice

Processing: The device captures the audio and converts it into text data.

Output: The converted text data is sent to the server.

Step 4:

The server performs emotion analysis. The server sends the acquired text data to an emotion analysis engine (IBM Watson Tone Analyzer), which analyzes the user's emotional state. The analysis results are added to the text data.

Input: Text data

Processing: The server analyzes the emotional state using an emotion analysis engine.

Output: Text data with emotional states added

Step 5:

The server generates a response. The server uses a dialogue management system (Rasa) to generate a response based on the sentiment analysis results and the content of the conversation. The response is in text format.

Input: Text data with sentiment analysis results added

Processing: The server generates a response using the dialogue management system.

Output: Generated response text

Step 6:

The server converts the response text into speech. The generated response text is converted into audio data using a speech synthesis engine (Google Text-to-Speech API). The converted audio data is then sent back to the device.

Input: Generated response text

Processing: The server uses a speech synthesis engine to convert the data into audio.

Output: Converted audio data

Step 7:

The device plays the audio data. The audio data sent to the device is played back through the speaker and becomes audible to the user. This allows for real-time audio conversation between the user and the character.

Input: Audio data

Processing: The device plays the audio data.

Output: User hears response

The specific processing unit 290 transmits the results of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the results of the specific processing. The microphone 38B acquires audio indicating the user input regarding the results of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

Data generation model 58 is what is known as generative AI (artificial intelligence). Examples of data generation model 58 include generative AI such as ChatGPT (Internet search <URL: https://openai.com/blog/chatgpt>) and Gemini (Internet search <URL: https://gemini.google.com/?hl=ja>). Data generation model 58 is obtained by performing deep learning on a neural network. A prompt containing an instruction is input to data generation model 58, and inference data such as voice data indicating speech, text data indicating text, and image data indicating an image is also input. Data generation model 58 performs inference on the input inference data in accordance with the instructions indicated by the prompt, and outputs the inference results in the form of data such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and/or summarization.

In the above embodiment, an example was given in which the specific processing was performed by the data processing device 12, but the technology disclosed herein is not limited to this, and the specific processing may also be performed by the smart device 14.

[Second embodiment]

Figure 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

As shown in FIG. 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

The data processing device 12 includes a computer 22, a database 24, and a communication I/F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and communication I/F 26 are also connected to the bus 34. The communication I/F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and/or a LAN (Local Area Network).

The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I/F 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the user 20's surroundings (e.g., an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

The communication I/F 44 is connected to the network 54. The communication I/Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I/Fs 44 and 26 is carried out in a secure manner.

Figure 4 shows an example of the main functions of the data processing device 12 and smart glasses 214. As shown in Figure 4, in the data processing device 12, specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32.

The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as the specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

Storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

Next, we will explain the identification process performed by the identification processing unit 290 of the data processing device 12. In the following explanation, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

This invention is a system that uses artificial intelligence technology to learn the appearance and personality of specific characters or people, stores that learning data as non-fungible tokens (NFTs), and enables real-time voice conversations with users 24 hours a day, 365 days a year. The system is configured as follows:

System Configuration

1. Learning with AI technology

The server collects data on the appearance, personality, and past behavior of specific characters or individuals, and uses artificial intelligence algorithms to learn this information.

The learning results include the character's specific behavioral and response patterns and are saved as a highly accurate model.

2. Save as an NFT

The server stores the learned AI data as an NFT, using blockchain technology to guarantee its uniqueness and ownership.

3. Providing an interface

The device provides the user with an interface for selecting a conversation partner. Users can log in and select their favorite character.

4. Real-time voice conversation

The user performs an operation to start a conversation with the selected character. The device captures the user's voice, converts it into text data, and sends it to the server.

Based on the text data received by the server, an appropriate response is generated using an AI model. This response is then converted back into audio data and sent to the device.

The device plays the audio data and the conversation takes place in real time.

5. Usage Fee Collection and Revenue Sharing

The server collects the usage fee from the user. Payment processing is carried out through the terminal.

Revenues will be shared among the parties involved, including copyright holders of characters and people, and service providers.

Specific examples

1. The user logs in and selects a character.

The user opens the app and logs in by entering "user123" and "password123" on the login screen. The device sends the entered data to the server and authentication is performed. If authentication is successful, the home screen is displayed on the device.

The user presses the "Select Character" button on the home screen and selects, for example, "Character A." The device sends the selected character information to the server, and the server retrieves the corresponding AI data and sends it to the device.

2. Initiating and conducting actual conversations

The user presses the "Start Conversation" button and speaks "Hello, Character A" into the device's microphone. The device converts the speech into text and sends it to the server.

The server analyzes the text data and generates an appropriate response based on the AI model. For example, the response generated might be, "Hello, I'm Character A. How's your day?"

The generated response is converted into audio data and sent to the device. The device plays the audio data, allowing the user to hear the response.

3. Payment of fees and revenue sharing

When a user pays a usage fee, the terminal captures the payment information and sends it to the server. The server processes the payment using a payment gateway and, after confirmation, distributes the revenue among the parties.

This system allows fans to safely and easily interact with their favorite characters and people 24 hours a day, 365 days a year, and also reduces the high costs associated with hosting events and inviting celebrities to attend.

The processing flow is explained below.

Step 1:

The user opens the application. The user accesses the login screen and enters their ID and password.

Step 2:

The device receives the user's input and sends the ID and password information to the server. The communication is encrypted.

Step 3:

The server checks the received ID and password against a database and authenticates the user. If the ID and password do not match, an error message is sent to the terminal.

Step 4:

If the server successfully authenticates, it generates session information and sends it back to the device. The device uses the received session information to display the home screen.

Step 5:

The user presses the "Character Selection" button on the home screen and selects the character they want to interact with.

Step 6:

The device receives the user's selection and sends the character ID to the server. The server then retrieves the AI model data for the selected character.

Step 7:

The server sends the acquired AI model data to the device. The device displays a conversation interface with the character based on the received data.

Step 8:

The user presses the "Start conversation" button to begin the conversation. The device's microphone is turned on to capture the user's voice.

Step 9:

The device records the user's voice and converts the voice data into text data in real time. This text data is then sent to the server.

Step 10:

The server analyzes the received text data and generates a response using an AI model. For example, if the user says "Hello, Character A," the response will be "Hello! How's your day?"

Step 11:

The response text generated by the server is converted into audio data and sent to the device.

Step 12:

The device plays back the received audio data and provides a response to the user. The user can then listen to this and ask questions or make comments.

Step 13:

If the user needs to pay a usage fee, the terminal will display a payment screen. The user will then enter their credit card information, etc.

Step 14:

The terminal sends the entered payment information to the server, which then processes the payment through the payment gateway.

Step 15:

The server sends information about the success or failure of the payment to the terminal. If successful, the usage rights are updated.

Step 16:

The terminal will notify the user of the payment result, allowing them to continue using the system.

(Example 1)

Next, we will explain Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

In conventional dialogue systems with characters or people, it was difficult to achieve real-time voice conversation, and users were unable to receive a satisfactory experience. Furthermore, there was a lack of means to guarantee the uniqueness and ownership of trained AI data, creating a risk of data tampering or unauthorized use. Furthermore, there was no established mechanism for collecting usage fees from users and distributing revenue fairly among the parties involved. New technology is needed to solve these problems.

The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

In this invention, the server includes: means for learning the appearance and personality of a specific character or person using artificial intelligence technology; means for storing the learned data as a non-fungible token (NFT); means for providing an interface for users to select a conversation partner and converse with them; means for acquiring the user's voice and converting it into text data using voice recognition technology; means for generating an appropriate response based on the text data using an AI model and converting the response into voice data; means for transmitting the converted voice data to the user's terminal to conduct voice conversations in real time; and means for collecting usage fees from users and distributing revenue among the parties involved. This enables real-time voice conversations, guarantees the uniqueness and ownership of learned data, and enables fair revenue distribution.

"Specific characters or people" refers to fictional or real people that users select when interacting, whose appearance, personality, behavior patterns, etc. are predefined.

"Artificial intelligence technology" refers to technology that uses machine learning and deep learning models to learn from specific data and generate appropriate responses even for unknown data.

A non-fungible token (NFT) is a unique digital asset created using blockchain technology with guaranteed ownership and authenticity.

"Interface" refers to the user interface provided to users to select and interact with characters or people, and is usually provided as an application.

"Speech recognition technology" refers to technology that converts a user's voice into text data, and includes services such as Google Cloud Speech-to-Text and Microsoft Azure Speech Recognition Services.

An "AI model" is a neural network model trained based on collected data and used to generate the responses of a specific character or person.

"Converting into voice data" means using speech synthesis technology to convert the text response generated by the AI model into voice data that users can hear.

"Real-time voice conversation" refers to a form in which users can instantly communicate back and forth with characters or people using voice.

"Collecting usage fees" means collecting fees from users for using a service, often using an online payment system.

"Distributing revenues among the parties involved" means distributing the collected usage fees in fair proportions to copyright holders of characters and people, service providers, and other rights holders.

Form for implementing the invention

This invention is a system that uses artificial intelligence technology to learn the appearance and personality of specific characters or people, stores that learning data as non-fungible tokens (NFTs), and enables real-time voice conversations with users 24 hours a day, 365 days a year. The system is configured as follows:

Learning using artificial intelligence technology

The server collects data on the appearance, personality, and past behavior of specific characters or people. For example, it uses scripts that retrieve data from YouTube, social media, etc. This data is trained using the TensorFlow or PyTorch machine learning frameworks. During the data preprocessing process, noise removal, tokenization, normalization, etc. are performed, and a neural network model is built. The trained model is then evaluated for accuracy and saved if it is highly accurate.

Stored as an NFT

The server converts the model data into ONNX format to store the trained AI data as an NFT. The NFT is then issued using a blockchain platform such as Ethereum or Binance Smart Chain, which incorporates the model's unique identifier and metadata in the process.

Providing an interface

The device provides the user with an interactive user interface. The user logs in using an application developed with React Native or Flutter. The login screen has username and password input fields, and the user authenticates. If authentication is successful, a character selection screen is displayed, where the user selects the character they want to interact with.

Real-time voice conversation

The user initiates a conversation with the selected character. The device uses the microphone to capture the user's voice and converts it into text using the Google Cloud Speech-to-Text API or Microsoft Azure Speech Recognition Service. The text is sent to a server, which uses an AI model (e.g., GTP-3 or BERT) to generate an appropriate response. This response is converted into audio data by a speech synthesis engine (e.g., Amazon Polly or Google Text-to-Speech) and sent to the device. The device plays back the audio data, enabling real-time conversation.

Collection of usage fees and revenue sharing

The usage fee paid by users when using the service is collected by the device acquiring payment information. Payment processing is carried out by the server using the Stripe or PayPal API. The collected usage fee is distributed fairly among the copyright holders of the characters and people and the service provider.

Specific examples

1. User login and character selection

The user opens the app and logs in by entering "user123" and "password123" on the login screen. The device sends the input data to the server and authentication is performed. If authentication is successful, the home screen is displayed.

The user presses the "Character Selection" button on the home screen and selects "Character A." The device sends the selected character information to the server, and the server retrieves the corresponding AI data and sends it to the device.

2. Initiating and conducting actual conversations

The user presses the "Start Conversation" button and speaks "Hello, Character A" into the device's microphone. The device converts the speech into text and sends it to the server.

The server analyzes the text data and generates an appropriate response based on the AI model. For example, the response generated might be, "Hello, I'm Character A. How's your day?"

The generated response is converted into audio data and sent to the device. The device plays the audio data, allowing the user to hear the response.

Example prompt

"Hello, I'm Character A. How's your day going?"

This system allows fans to safely and easily interact with their favorite characters and people 24 hours a day, 365 days a year, and also reduces the high costs associated with hosting events and inviting celebrities to attend.

The flow of the identification process in Example 1 will be explained using Figure 11.

Step 1: Data Collection

The server collects data on the appearance, personality, and past behavior of specific characters or people. Inputs include links to YouTube videos and URLs of social media posts. The data is obtained using scraping technology and APIs. The output is the obtained text, images, and audio data.

Step 2: Data Preprocessing

The server preprocesses the collected data. The raw data collected in step 1 is provided as input. Specifically, processes such as noise removal, text tokenization, and normalization are performed. For example, unnecessary special characters are removed from the text data and it is converted into a standard format. The output is the preprocessed data.

Step 3: Train the model

The server uses the preprocessed data to train an artificial intelligence model (e.g., TensorFlow or PyTorch). The preprocessed data from step 2 is provided as input. Specifically, the data is divided into batches and the model parameters are optimized using a GPU. The output is a trained, highly accurate model.

Step 4: Evaluate and save the model

The server evaluates and saves the trained model. The model trained in step 3 and the validation dataset are used as input. Specifically, the accuracy of the model is evaluated using the validation data, and if the accuracy meets the criteria, the model is saved. The output is the evaluation results and the saved model.

Step 5: Save as an NFT

The server saves the trained AI model as an NFT. The model data saved in step 4 is used as input. Specifically, the model is converted to ONNX format and an NFT is issued on a blockchain platform (e.g., Ethereum or Binance Smart Chain). The output is the model data saved as an NFT.

Step 6: User authentication and character selection

The device displays a login screen for the user and performs authentication. A username and password are provided as input. The authentication data is sent to the server and checked against a database. If authentication is successful, the character selection screen is displayed. The output is information about the selected character.

Step 7: Capture and convert audio input

The device uses a microphone to capture the user's voice. The user's voice data is provided as input. Specifically, the voice is converted into text data using the Google Cloud Speech-to-Text API. The output is text data.

Step 8: Generate a response

The server receives the text data and generates a response using an AI model. The text data converted in step 7 is provided as input. Specifically, the AI model (e.g., GPT-3 or BERT) analyzes the text data and generates an appropriate response. The output is the text data of the generated response.

Step 9: Convert to audio and send

The server converts the generated text response into audio data and sends it to the device. The text data generated in step 8 is provided as input. Specifically, it converts it into audio data using Amazon Polly or Google Text-to-Speech. The output is audio data.

Step 10: Play real-time audio conversations

The device receives the audio data and plays it on the speaker. The input is the audio data sent in step 9. Specifically, it decodes the audio data and plays it on the speaker so that the user can hear it. The output is audio played in real time.

Step 11: Collection and distribution of usage fees

The terminal obtains payment information to collect usage fees and sends it to the server. Payment information is provided as input. The server processes the payment using Stripe or PayPal APIs and distributes revenue among the parties involved. Specifically, after confirming the payment process, the revenue is distributed via bank transfer or cryptocurrency. The output is the distributed revenue.

(Application Example 1)

Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

Currently, providing high-quality customer service to customers visiting physical stores requires a lot of labor and resources. It is also difficult to provide detailed information about specific products in the store, making it difficult to increase customer satisfaction. Furthermore, while there is a demand for personalized guidance that allows customers to select products based on their preferences and interests, achieving this is complex. To solve these challenges, a system that can interact with customers in real time and provide personalized guidance is needed.

The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

In this invention, the server includes means for learning the characteristics of a specific virtual character using machine learning technology, means for saving the learned information as a non-fungible character (NFT), means for providing an operation screen on which a user can select a conversation partner and conduct a conversation, means for real-time voice communication, means for receiving service fees from users and distributing revenue among relevant parties, means for providing guidance using a display device that supports conversations between customers and virtual characters in physical stores, and means for users to obtain product information in the store and receive support through the character. This makes it possible to improve the customer experience in physical stores and provide personalized, high-quality customer service.

"Characteristics of a specific virtual person" refers to data including information such as the appearance, personality, and past words and actions of a character or celebrity.

"Machine learning technology" is a technology that allows computers to automatically learn from data and make predictions and classifications.

A "Non-Fungible Trading Card (NFT)" is a digital asset whose uniqueness and ownership are guaranteed using blockchain technology.

An "operation screen" is a display screen that provides an interface for users to select a conversation partner and conduct a conversation.

"Real-time voice communication" refers to technology that allows users to instantly communicate via voice with a character of their choice.

"Service fee" refers to the amount of money paid by the user in consideration for using the service.

"Related parties" refers to people or organizations that are subject to revenue sharing, such as character or system providers and developers.

"Display devices" refer to hardware for displaying information, such as displays installed in physical stores, smart glasses, and head-mounted displays (HMDs).

"Supporting virtual character conversations" refers to a specific virtual character communicating with the user via voice, providing guidance and information.

"Get product information" means viewing or obtaining detailed information about products in the store.

"Receiving support through a character" means that a virtual character guides or explains products or services to the user.

To implement this invention, the following system configuration is required. The system consists of a server, a terminal, and a user as its main components.

1. System Programming

The server builds a database that uses machine learning technology to learn the characteristics of specific virtual characters. The terminal provides an operation screen that allows the user to select a conversation partner and engage in real-time voice communication with that virtual character. The server also stores the learned information as a non-fungible character (NFT).

The terminal displays an operation screen that allows the user to select a virtual person to interact with, and transmits the selection information to the server. When the user communicates via voice in real time, the terminal converts the user's voice into text information and transmits that text information to the server.

The server feeds the received text information into a machine learning model to generate a response based on the conversation. The generated response is converted into audio data by the server and sent to the device. The device plays the audio data and provides the virtual character's response to the user.

2. Processing Description

The server is implemented using programming languages such as Python. Machine learning techniques include the NLP (natural language processing) model GPT-3. Libraries such as Google Speech Recognition are used for voice recognition and pyttsx3 for voice synthesis.

The device includes hardware such as smart glasses, a head-mounted display (HMD), and a microphone. The device recognizes the user's voice and sends it to the server. The server converts the voice into text and generates an appropriate response using an NLP model. The server then converts the response into audio data and sends it back to the device.

As a specific example, a user enters a physical store and puts on a pair of smart glasses. When the user says, "What products do you recommend?", the device's microphone picks up the user's voice and sends it to the server. The server analyzes the text data and generates a response using GPT-3. The response might be something like, "Currently, we recommend this smartwatch. It's the latest model with a long battery life and comprehensive health management features."

The generated response is converted into audio data and sent to the device. The device then plays the audio data, allowing the user to hear the voice of the virtual character.

3. Specific examples and prompt sentence examples

As a practical example, consider a situation where a user wears smart glasses and asks for assistance in a store. For example, the user might say, "Hello, what products do you recommend?"

Example prompt:

Customer Question: "Hello, what products do you recommend?"

Prompt for generative AI model: "A customer asked you for a product recommendation. How would you respond?"

The flow of the specific processing in Application Example 1 will be explained using Figure 12.

Program processing steps

Step 1:

The user puts on the smart glasses and launches the application. The input is the smart glasses' initialization data, and the output is the display of a login screen. Specifically, the device displays the login screen to the user, and the user enters their login information.

Step 2:

The user enters login information and presses the login button. The input is login information, and the output is an authentication request to the server. The device sends the user's input information to the server, and the server performs authentication.

Step 3:

The server authenticates the user's login information and returns the result to the device. The input is the user's login information, and the output is the authentication result. Specifically, the server searches the database and confirms the login information. If successful, the home screen is displayed on the device.

Step 4:

The user selects a character to interact with from the home screen. The input is character selection information, and the output is sending the selection information to the server. The device sends the selected character information to the server, and the server obtains the corresponding learning data.

Step 5:

The server obtains the learning data for the selected character and sends it to the device. The input is the character selection information, and the output is the learning data. The server retrieves the relevant data from the learning database and sends it to the device.

Step 6:

The user presses the "Start conversation" button and speaks into the device's microphone. The input is the user's voice, and the output is text data. The device uses voice recognition to convert the user's voice into text and sends the text data to the server.

Step 7:

The server receives the user's text data and generates an appropriate response using a generative AI model. The input is the user's text data and the output is the generated response. The server generates the response using an NLP model (e.g., GPT-3).

Step 8:

The response generated by the server is converted into audio data and sent to the device. The input is the generated response, and the output is audio data. The server uses a speech synthesis library (e.g., pyttsx3) to generate the audio data and send it to the device.

Step 9:

The device plays back the audio data received from the server and lets the user listen. The input is audio data, and the output is audio playback. The device plays back the audio data, allowing the user to hear the virtual character's response.

Step 10:

When the user wants to end the service, they perform an operation to end the application. The input is the end operation, and the output is to end the application. The device will end the application and stop displaying the smart glasses.

It is also possible to further combine an emotion engine that estimates the user's emotion. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

This invention is a system that uses artificial intelligence technology to learn the appearance and personality of specific characters or people, stores that learning data as non-fungible tokens (NFTs), and enables real-time voice conversations with users 24 hours a day, 365 days a year. Furthermore, by combining this invention with an emotion engine, it is possible to recognize the user's emotions and provide adaptive responses based on those emotions.

System Configuration

1. Learning with AI technology

The server collects data on the appearance, personality, and past behavior of specific characters or people, and uses an artificial intelligence algorithm to learn this information. The learning results, including the character's specific behavior and response patterns, are saved as a highly accurate model.

2. Save as an NFT

The server stores the learned AI data as an NFT, using blockchain technology to guarantee its uniqueness and ownership.

3. Providing an interface

The device provides the user with an interface for selecting a conversation partner. Users can log in and select their favorite character.

4. Real-time voice conversation

The user performs an operation to start a conversation with the selected character. The device captures the user's voice, converts it into text data, and sends it to the server.

Based on the text data received by the server, an appropriate response is generated using an AI model. This response is then converted back into audio data and sent to the device.

The device plays the audio data and the conversation takes place in real time.

5. Emotional Engine Integration

The server is equipped with an emotion engine that uses the user's voice data to analyze emotions. The emotion engine analyzes the user's emotional state (e.g., joy, sadness, anger, etc.) from the voice data.

Based on the results of emotion analysis, the AI model adaptively changes the response content. For example, if the user speaks in a sad voice, a response such as "Cheer up!" will be generated.

6. Usage Fee Collection and Revenue Sharing

The server collects usage fees from users. Payments are processed through the terminal. Revenues are distributed among the parties involved, including the copyright holders of characters and people, and service providers.

Specific examples

1. The user logs in and selects a character.

The user opens the app and logs in by entering "user123" and "password123" on the login screen. The device sends the entered data to the server and authentication is performed. If authentication is successful, the home screen is displayed on the device.

The user presses the "Select Character" button on the home screen and selects, for example, "Character A." The device sends the selected character information to the server, and the server retrieves the corresponding AI data and sends it to the device.

2. Initiating and conducting actual conversations

The user presses the "Start Conversation" button and speaks "Hello, Character A" into the device's microphone. The device converts the speech into text and sends it to the server.

The server analyzes the text data and generates an appropriate response based on an AI model. For example, it might respond with "Hello! How's your day?"

At the same time, the server sends the voice data to the emotion engine, which analyzes the user's emotional state.

The emotion engine analyzes the user's emotional state and feeds the results back to the AI model. For example, if the user sounds sad, the response will be changed to an encouraging message such as "Cheer up!"

The generated response is converted into audio data and sent to the device, which then plays the audio data and provides the response to the user.

3. Payment of fees and revenue sharing

If the user needs to pay a usage fee, the terminal will display a payment screen. The user will enter their credit card information, etc.

The terminal sends the entered payment information to the server, which processes the payment through the payment gateway.

The server sends information about the success or failure of the payment to the terminal. If successful, the usage rights are updated.

The terminal will notify the user of the payment result, allowing them to continue using the system.

This system allows fans to safely and easily interact with their favorite characters and people 24 hours a day, 365 days a year, and by providing appropriate responses based on the user's emotions, it enables more personal and intimate communication.

The processing flow is explained below.

Step 1:

The user opens the application. The user accesses the login screen and enters their ID and password.

Step 2:

The device receives the user's input and sends the ID and password information to the server. The communication is encrypted.

Step 3:

The server checks the received ID and password against a database and authenticates the user. If the ID and password do not match, an error message is sent to the terminal.

Step 4:

If the server successfully authenticates, it generates session information and sends it back to the device. The device uses the received session information to display the home screen.

Step 5:

The user presses the "Character Selection" button on the home screen and selects the character they want to interact with.

Step 6:

The device receives the user's selection and sends the character ID to the server. The server then retrieves the AI model data for the selected character.

Step 7:

The server sends the acquired AI model data to the device. The device displays a conversation interface with the character based on the received data.

Step 8:

The user presses the "Start conversation" button to begin the conversation. The device's microphone is turned on to capture the user's voice.

Step 9:

The device records the user's voice and converts the voice data into text data in real time. This text data is then sent to the server.

Step 10:

The server analyzes the received text data and generates a response using an AI model. For example, if the user says "Hello, Character A," the response will be "Hello! How's your day?"

Step 11:

At the same time, the server sends the user's voice data to the emotion engine, which analyzes the user's emotional state. The emotion engine analyzes the voice data and identifies the user's emotional state (joy, sadness, anger, etc.).

Step 12:

The emotion engine sends the analysis results back to the server, which then adaptively changes the response based on the results. For example, if the user speaks in a sad voice, the response will be "Cheer up!"

Step 13:

The server converts the response text into audio data and sends it to the device.

Step 14:

The device plays back the received audio data and provides a response to the user. The user can then listen to this and ask questions or make comments.

Step 15:

If the user needs to pay a usage fee, the terminal will display a payment screen. The user will then enter their credit card information, etc.

Step 16:

The terminal sends the entered payment information to the server, which then processes the payment through the payment gateway.

Step 17:

The server sends information about the success or failure of the payment to the terminal. If successful, the usage rights are updated.

Step 18:

The terminal will notify the user of the payment result, allowing them to continue using the system.

(Example 2)

Next, we will explain Example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

With conventional voice conversation systems using artificial intelligence, it was difficult to efficiently learn the appearance and personality of characters or people and conduct real-time voice conversations with users. Furthermore, technology for providing adaptive responses based on the user's emotions was not sufficiently developed, making it impossible to achieve natural, personal communication with users.

The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

In this invention, the server includes means for learning the appearance and personality of a specific character or person using artificial intelligence technology, means for saving the learned data as a non-fungible token (NFT), and means for analyzing emotions from the user's voice and adaptively changing responses based on the analysis results. This makes it possible to learn the characteristics of characters and people with high accuracy, safely store that data, and provide flexible responses that correspond to the user's emotions.

"Specific characters or persons" refers to fictional or real people or characters that users can select as their conversation partners.

"Artificial intelligence technology" refers to technology that uses techniques such as machine learning and deep learning to extract patterns from data and make predictions and classifications.

"Means of learning" refers to the process of collecting data and generating an AI model based on that data.

A "non-fungible token (NFT)" is a token that uses blockchain technology to guarantee the uniqueness and ownership of digital data.

"Means of providing an interface" refers to the method of providing screens and menus that allow users to access and operate the system.

"Means for conducting real-time voice conversation" refers to technology that allows users and characters to have instant voice conversations.

"Means for analyzing emotions from a user's voice" refers to technology that analyzes a user's emotional state based on their voice data.

"Means for adaptively changing responses" refers to a method for changing the AI's response content based on the results of emotion analysis.

"Means for collecting usage fees and distributing revenue among parties" refers to a system for collecting service usage fees from users and distributing the revenue obtained among parties.

Form for implementing the invention

This invention is a system that uses artificial intelligence technology to learn the appearance and personality of specific characters or people, stores that learning data as non-fungible tokens (NFTs), and provides users with real-time voice conversations 24 hours a day, 365 days a year. Furthermore, by combining it with an emotion engine, it is possible to recognize the user's emotions and provide adaptive responses based on those emotions.

System Overview

The system consists of the following main components:

1. Server

2. Device

3. User

Server Role

1. Training data collection and AI model generation

The server collects data on the appearance, personality, and past behavior of specific characters and people. This data includes video, audio files, text data, etc.

The collected data is trained using an AI algorithm (e.g., TensorFlow or PyTorch). During the initial training stage, data preprocessing (noise removal, text normalization, etc.) is performed.

Save the completed model in the database.

2. Data storage as NFTs

The learned AI data is stored as a non-fungible token (NFT). This process uses blockchain technology (e.g., Ethereum).

The server registers the generated NFT on the blockchain and maintains owner information.

3. Sentiment Analysis and Response Adaptation

The server uses an emotion engine (for example, IBM Watson's emotion recognition API) to analyze the user's voice data. Based on the analysis results, the AI model adaptively changes the response content.

For example, if a user says, "I'm feeling a bit down today," emotion recognition can generate a response such as, "Cheer up!"

4. Usage fee collection and revenue sharing

The server collects the usage fee and manages the payment process. Payment is made using credit card information entered by the user on the terminal.

Revenues will be distributed among the parties involved (character copyright holders, service providers, etc.).

Device role

1. Providing an interface

The device provides the user with an interface to select a conversation partner and conduct a conversation. The user launches the app, enters their authentication information on the login screen, and logs in.

After logging in, users can press the "Select Character" button to choose their preferred character and begin a conversation with that character.

2. Audio data processing

The device captures the user's voice and converts it into text data using a speech recognition engine such as Google Speech-to-Text.

The converted text data is sent to the server, and after an appropriate response is generated, it is converted back into voice data and sent to the device.

The device plays the audio data, enabling real-time conversations.

User Roles

1. Log in and select a character

The user launches the app and enters authentication information such as "user123" and "password123" on the login screen.

Once authentication is successful, select a character from the home screen and begin a conversation with the character you select.

2. Actual conversation progression

The user presses the "Start Conversation" button and speaks into the device's microphone, saying "Hello, Character A."

The device converts this voice into text data and sends it to the server. The server uses an AI model to generate a response based on the text data and sends an adaptive reply.

For example, you can receive responses such as "Hello! How are you today?" or adaptive messages such as "Cheer up!"

Examples and prompts

"Hello, Character A. How was your day?"

"I've been feeling a bit down lately. Can you cheer me up?"

"What do you think about today's news?"

By using these prompts, users can enjoy more personal and natural conversations with the characters.

The flow of the identification process in Example 2 will be explained using Figure 13.

Program processing flow

Step 1:

Training data collection and AI model generation

The server collects data on the appearance, personality, and past behavior of specific characters or people. The data used for this collection includes text, audio files, images, and videos. Specifically, the data is retrieved from a database via web scraping or API and stored on the server.

Input: Data about the character's appearance, personality, and past words and actions.

Data processing: noise removal, text normalization, data cleaning.

Output: Clean dataset after preprocessing.

The server uses an AI algorithm (such as TensorFlow or PyTorch) to train a model using this data. After training, the model reflects the character's behavior and response patterns.

Input: Clean dataset after preprocessing.

Data computation: Applying machine learning algorithms and training models.

Output: Trained AI model.

Step 2:

Storing data as NFTs

The server stores the learned AI data as a non-fungible token (NFT).

Input: Trained AI model.

Data processing: Blockchain technology (e.g., Ethereum) is used to ensure the uniqueness and ownership of the model for NFT conversion.

Output: NFT registered on the blockchain.

The server registers the generated NFT on the blockchain and retains owner information. Specifically, the NFT is registered through the blockchain API.

Step 3:

User login and character selection

The user launches the app and enters their authentication information (e.g., "user123" and "password123") on the login screen.

The device sends the entered authentication information to the server and performs authentication.

Input: User authentication information.

Data calculation: Compare with the database and determine whether authentication is successful.

Output: Authentication success or failure status.

If authentication is successful, the home screen will be displayed on the device. The user presses the "Select Character" button to choose the character they want to interact with.

Input: Character information selected by the user.

Data processing: Search and obtain the AI model corresponding to the selected character.

Output: AI data for the selected character.

Step 4:

Real-time voice conversation processing

The user presses the "Start conversation" button and speaks into the device's microphone.

The device captures the user's voice and converts it into text data using Google Speech-to-Text.

Input: User's voice data.

Data processing: voice-to-text conversion.

Output: Text data.

The converted text data is sent to the server, which generates an appropriate response.

Input: User's text data.

Data calculations: Use AI models to generate appropriate responses.

Output: The generated text response.

The response is converted back into audio data and sent to the device.

Input: The generated text response.

Data processing: converting text to audio.

Output: Generated audio data.

The device plays the audio data and the conversation takes place in real time.

Step 5:

Sentiment analysis and response adaptation

The server analyzes the user's voice data using an emotion engine (e.g., IBM Watson's emotion recognition API).

Input: User's voice data.

Data calculation: Analyze the user's emotional state from voice data.

Output: User's emotional state data.

Based on the results of emotion analysis, the AI model adaptively changes the response content.

Input: User emotional state data and generated text response.

Data calculation: Change the response content to match the emotional state.

Output: The final adapted response text.

For example, if the user sounds sad, the response will be changed to something like "Cheer up!"

Step 6:

Collecting usage fees and processing payments

If the user needs to pay a usage fee, the terminal will display a payment screen. The user will enter their credit card information, etc.

Input: User's payment information.

The terminal sends payment information to the server, which processes the payment through a payment gateway (e.g., Stripe).

Data calculation: Executes payment processing based on payment information.

Output: Payment success or failure status.

The server notifies the terminal of the success or failure of the payment, and if successful, the usage rights are updated.

The terminal will notify the user of the payment result, allowing them to continue using the system.

(Application Example 2)

Next, we will explain Application Example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

The objective of this invention is to provide a system that can reproduce the appearance and personality of a specific character or person in real time, while providing appropriate responses based on the user's emotions through dialogue with them. Furthermore, the aim is to realize a more personal and sympathetic shopping experience by having characters act as shopping assistants in virtual stores, explaining products and recommending items.

The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

In this invention, the server includes: means for learning the appearance and personality of a specific character or person using artificial intelligence technology; means for storing the learned data as a non-fungible token (NFT); means for providing an interface for users to select a conversation partner and converse with them; means for conducting real-time voice conversations; means for collecting usage fees from users and distributing revenue among the parties involved; means for analyzing the emotional state from the user's voice data using an emotion engine and generating an adaptive response based on the analysis results; means for converting the generated response into voice data and providing it to the user as voice; and means for the character to act as a shopping assistant in a virtual store, explaining products and recommending items. This enables real-time, emotion-based dialogue with a specific character and a personalized, sympathetic shopping assistant experience in a virtual store.

"Specific characters or people" are virtual beings whose appearances and personalities are reproduced using artificial intelligence technology.

"Artificial intelligence technology" is a technology that learns the appearance and personality of a specific character or person, and generates responses and actions based on that information.

A non-fungible token (NFT) is a means of using blockchain technology to uniquely identify a specific digital asset and prove ownership of it.

An "interface" is a device or software that provides input and display means for a user to select a conversation partner and initiate a conversation.

"Means for conducting real-time voice conversations" refers to technology that allows users to have instant voice conversations with specific characters or people.

An "emotion engine" is an algorithm and system that analyzes a user's emotional state from their voice data and generates a response based on the analysis results.

An "adaptive response" is a response that changes depending on the user's speech content and emotional state, and is generated individually and dynamically.

A "virtual store" is a virtual shopping environment that exists on the Internet and is a platform that allows users to conduct purchasing activities in a virtual space.

A "shopping assistant" is a virtual character or person who explains products and recommends items to users in a virtual store.

"Collection" means receiving usage fees from users.

"Means for distributing revenues among parties involved" refers to a mechanism for appropriately distributing revenues earned through system operation to parties involved, such as character copyright holders and service providers.

System Overview

This invention is a system that recreates specific characters or people in real time and engages in voice conversations with users. It primarily functions as a shopping assistant in virtual stores. This system consists of three elements: a server, a terminal, and a user.

Program structure and various technologies

Speech recognition engine

The server receives the user's voice input and converts the voice into text data using a speech recognition engine (Google Speech-to-Text API).

Sentiment analysis engine

The server performs emotion analysis on the acquired text data using IBM Watson Tone Analyzer, thereby determining the user's emotional state.

Dialogue management system

The server uses Rasa to generate responses based on the user's conversation content and sentiment analysis results.

Speech synthesis engine

The server converts the generated response into audio data using the Google Text-to-Speech API and sends it to the device.

Program processing

1. Voice input and text conversion

When a user speaks into their smartphone's microphone, the audio is picked up by the device and converted into text data using the Google Speech-to-Text API.

2. Emotion analysis

The transferred text data is subjected to emotion analysis using IBM Watson Tone Analyzer to determine the user's emotional state (e.g., joy, sadness, anger, etc.).

3. Response Generation

Rasa is used to generate appropriate responses based on analysis results and conversation content. These responses are adapted to the user's emotions.

4. Conversion to audio data and transmission

The generated response is converted into audio data using the Google Text-to-Speech API, and this audio data is sent to the device, enabling real-time conversation.

Specific examples

1. User login and character selection

The user launches the app and logs in by entering "user123" and "password123." After logging in, they select "Character A" on the character selection screen.

Example prompt:

User: Hello, I'm looking for clothes today.

---

Assistant: Hello, what kind of clothes are you looking for?

---

User: I'd like a casual shirt.

---

2. Voice input and analysis

When a user says, "Hello, I'm looking for clothes today," the speech is captured and converted into text data. The text data is sent to IBM Watson Tone Analyzer for sentiment analysis.

3. Response generation and delivery

Based on sentiment analysis and the conversation content, Rasa generates a response: "Hello, what kind of clothes are you looking for?" This response is converted into audio using the Google Text-to-Speech API and sent to the device.

4. Shopping Assistant Function

If a user says, "I want a casual shirt," the assistant will suggest, "How about this shirt? It's a trendy design and very popular these days," and explain the product. It is also possible to provide more detailed product descriptions and suggestions based on the results of user sentiment analysis.

This allows the invention to provide emotion-based real-time interactions with specific characters and a personalized, immersive shopping assistant experience in virtual stores.

The flow of the specific processing in Application Example 2 will be explained using Figure 14.

Step 1:

The user launches the app and logs in. The user enters their username and password, and the device sends that information to the server. The server performs authentication, and if successful, displays the character selection screen on the device.

Enter: User's username and password

Process: The server authenticates the user.

Output: Upon successful authentication, the character selection screen will be displayed.

Step 2:

The user selects a character. The user chooses a specific character on the character selection screen, and the device sends the selection information to the server. The server loads the selected character's AI model and sends the relevant data to the device.

Input: User's character selection information

Processing: The server loads the AI model and retrieves relevant data.

Output: Character information is displayed on the device.

Step 3:

The user begins the conversation. They speak into the smartphone's microphone, and the device picks up the audio. The speech is converted into text data using a speech recognition engine (Google Speech-to-Text API), and the text data is sent to the server.

Input: User's voice

Processing: The device captures the audio and converts it into text data.

Output: The converted text data is sent to the server.

Step 4:

The server performs emotion analysis. The server sends the acquired text data to an emotion analysis engine (IBM Watson Tone Analyzer), which analyzes the user's emotional state. The analysis results are added to the text data.

Input: Text data

Processing: The server uses an emotion analysis engine to analyze the emotional state.

Output: Text data with emotional states added

Step 5:

The server generates a response. The server uses a dialogue management system (Rasa) to generate a response based on the sentiment analysis results and the content of the conversation. The response is in text format.

Input: Text data with sentiment analysis results added

Processing: The server generates a response using the dialogue management system.

Output: Generated response text

Step 6:

The server converts the response text into speech. The generated response text is converted into audio data using a speech synthesis engine (Google Text-to-Speech API). The converted audio data is then sent back to the device.

Input: Generated response text

Processing: The server uses a speech synthesis engine to convert the data into audio.

Output: Converted audio data

Step 7:

The device plays the audio data. The audio data sent to the device is played back through the speaker and becomes audible to the user. This allows for real-time audio conversation between the user and the character.

Input: Audio data

Processing: The device plays the audio data.

Output: User hears response

The specific processing unit 290 transmits the results of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the results of the specific processing. The microphone 238 acquires audio indicating the user input regarding the results of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

Data generation model 58 is what is known as generative AI (artificial intelligence). Examples of data generation model 58 include generative AI such as ChatGPT (Internet search <URL: https://openai.com/blog/chatgpt>) and Gemini (Internet search <URL: https://gemini.google.com/?hl=ja>). Data generation model 58 is obtained by performing deep learning on a neural network. A prompt containing an instruction is input to data generation model 58, and inference data such as voice data indicating speech, text data indicating text, and image data indicating an image is also input. Data generation model 58 performs inference on the input inference data in accordance with the instructions indicated by the prompt, and outputs the inference results in the form of data such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and/or summarization.

In the above embodiment, an example was given in which the specific processing was performed by the data processing device 12, but the technology disclosed herein is not limited to this, and the specific processing may also be performed by the smart glasses 214.

[Third embodiment]

Figure 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

As shown in FIG. 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

The data processing device 12 includes a computer 22, a database 24, and a communication I/F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and communication I/F 26 are also connected to the bus 34. The communication I/F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and/or a LAN (Local Area Network).

The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I/F 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the user 20's surroundings (e.g., an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

The communication I/F 44 is connected to the network 54. The communication I/Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I/Fs 44 and 26 is carried out in a secure manner.

Figure 6 shows an example of the main functions of the data processing device 12 and headset terminal 314. As shown in Figure 6, in the data processing device 12, specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32.

The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as the specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

Storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

In the headset terminal 314, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

Next, we will explain the identification process performed by the identification processing unit 290 of the data processing device 12. In the following explanation, the data processing device 12 will be referred to as the "server" and the headset-type terminal 314 will be referred to as the "terminal."

This invention is a system that uses artificial intelligence technology to learn the appearance and personality of specific characters or people, stores that learning data as non-fungible tokens (NFTs), and enables real-time voice conversations with users 24 hours a day, 365 days a year. The system is configured as follows:

System Configuration

1. Learning with AI technology

The server collects data on the appearance, personality, and past behavior of specific characters or individuals, and uses artificial intelligence algorithms to learn this information.

The learning results include the character's specific behavioral and response patterns and are saved as a highly accurate model.

2. Save as an NFT

The server stores the learned AI data as an NFT, using blockchain technology to guarantee its uniqueness and ownership.

3. Providing an interface

The device provides the user with an interface for selecting a conversation partner. Users can log in and select their favorite character.

4. Real-time voice conversation

The user performs an operation to start a conversation with the selected character. The device captures the user's voice, converts it into text data, and sends it to the server.

Based on the text data received by the server, an appropriate response is generated using an AI model. This response is then converted back into audio data and sent to the device.

The device plays the audio data and the conversation takes place in real time.

5. Usage Fee Collection and Revenue Sharing

The server collects the usage fee from the user. Payment processing is carried out through the terminal.

Revenues will be shared among the parties involved, including copyright holders of characters and people, and service providers.

Specific examples

1. The user logs in and selects a character.

The user opens the app and logs in by entering "user123" and "password123" on the login screen. The device sends the entered data to the server and authentication is performed. If authentication is successful, the home screen is displayed on the device.

The user presses the "Select Character" button on the home screen and selects, for example, "Character A." The device sends the selected character information to the server, and the server retrieves the corresponding AI data and sends it to the device.

2. Initiating and conducting actual conversations

The user presses the "Start Conversation" button and speaks "Hello, Character A" into the device's microphone. The device converts the speech into text and sends it to the server.

The server analyzes the text data and generates an appropriate response based on the AI model. For example, the response generated might be, "Hello, I'm Character A. How's your day?"

The generated response is converted into audio data and sent to the device. The device plays the audio data, allowing the user to hear the response.

3. Payment of fees and revenue sharing

When a user pays a usage fee, the terminal captures the payment information and sends it to the server. The server processes the payment using a payment gateway and, after confirmation, distributes the revenue among the parties.

This system allows fans to safely and easily interact with their favorite characters and people 24 hours a day, 365 days a year, and also reduces the high costs associated with hosting events and inviting celebrities to attend.

The processing flow is explained below.

Step 1:

The user opens the application. The user accesses the login screen and enters their ID and password.

Step 2:

The device receives the user's input and sends the ID and password information to the server. The communication is encrypted.

Step 3:

The server checks the received ID and password against a database and authenticates the user. If the ID and password do not match, an error message is sent to the terminal.

Step 4:

If the server successfully authenticates, it generates session information and sends it back to the device. The device uses the received session information to display the home screen.

Step 5:

The user presses the "Character Selection" button on the home screen and selects the character they want to interact with.

Step 6:

The device receives the user's selection and sends the character ID to the server. The server then retrieves the AI model data for the selected character.

Step 7:

The server sends the acquired AI model data to the device. The device then displays a conversation interface with the character based on the received data.

Step 8:

The user presses the "Start conversation" button to begin the conversation. The device's microphone is turned on to capture the user's voice.

Step 9:

The device records the user's voice and converts the voice data into text data in real time. This text data is then sent to the server.

Step 10:

The server analyzes the received text data and generates a response using an AI model. For example, if the user says "Hello, Character A," the response will be "Hello! How's your day?"

Step 11:

The response text generated by the server is converted into audio data and sent to the device.

Step 12:

The device plays back the received audio data and provides a response to the user. The user can then listen to this and ask questions or make comments.

Step 13:

If the user needs to pay a usage fee, the terminal will display a payment screen. The user will then enter their credit card information, etc.

Step 14:

The terminal sends the entered payment information to the server, which then processes the payment through the payment gateway.

Step 15:

The server sends information about the success or failure of the payment to the terminal. If successful, the usage rights are updated.

Step 16:

The terminal will notify the user of the payment result, allowing them to continue using the system.

(Example 1)

Next, Example 1 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset-type terminal 314 will be referred to as the "terminal."

In conventional dialogue systems with characters or people, it was difficult to achieve real-time voice conversation, and users were unable to receive a satisfactory experience. Furthermore, there was a lack of means to guarantee the uniqueness and ownership of trained AI data, creating a risk of data tampering or unauthorized use. Furthermore, there was no established mechanism for collecting usage fees from users and distributing revenue fairly among the parties involved. New technology is needed to solve these problems.

The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

In this invention, the server includes: means for learning the appearance and personality of a specific character or person using artificial intelligence technology; means for storing the learned data as a non-fungible token (NFT); means for providing an interface for users to select a conversation partner and converse with them; means for acquiring the user's voice and converting it into text data using voice recognition technology; means for generating an appropriate response based on the text data using an AI model and converting the response into voice data; means for transmitting the converted voice data to the user's terminal to conduct voice conversations in real time; and means for collecting usage fees from users and distributing revenue among the parties involved. This enables real-time voice conversations, guarantees the uniqueness and ownership of learned data, and enables fair revenue distribution.

"Specific characters or people" refers to fictional or real people that users select when interacting, whose appearance, personality, behavior patterns, etc. are predefined.

"Artificial intelligence technology" refers to technology that uses machine learning and deep learning models to learn from specific data and generate appropriate responses even for unknown data.

A non-fungible token (NFT) is a unique digital asset created using blockchain technology with guaranteed ownership and authenticity.

"Interface" refers to the user interface provided to users to select and interact with characters or people, and is usually provided as an application.

"Speech recognition technology" refers to technology that converts a user's voice into text data, and includes services such as Google Cloud Speech-to-Text and Microsoft Azure Speech Recognition Services.

An "AI model" is a neural network model trained based on collected data and used to generate the responses of a specific character or person.

"Converting into voice data" means using speech synthesis technology to convert the text response generated by the AI model into voice data that users can hear.

"Real-time voice conversation" refers to a form in which users can instantly communicate back and forth with characters or people using voice.

"Collecting usage fees" means collecting fees from users for using a service, often using an online payment system.

"Distributing revenues among the parties involved" means distributing the collected usage fees in fair proportions to copyright holders of characters and people, service providers, and other rights holders.

Form for implementing the invention

This invention is a system that uses artificial intelligence technology to learn the appearance and personality of specific characters or people, stores that learning data as non-fungible tokens (NFTs), and enables real-time voice conversations with users 24 hours a day, 365 days a year. The system is configured as follows:

Learning using artificial intelligence technology

The server collects data on the appearance, personality, and past behavior of specific characters or people. For example, it uses scripts that retrieve data from YouTube, social media, etc. This data is trained using the TensorFlow or PyTorch machine learning frameworks. During the data preprocessing process, noise removal, tokenization, normalization, etc. are performed, and a neural network model is built. The trained model is then evaluated for accuracy and saved if it is highly accurate.

Stored as an NFT

The server converts the model data into ONNX format to store the trained AI data as an NFT. The NFT is then issued using a blockchain platform such as Ethereum or Binance Smart Chain, which incorporates the model's unique identifier and metadata in the process.

Providing an interface

The device provides the user with an interactive user interface. The user logs in using an application developed with React Native or Flutter. The login screen has username and password input fields, and the user authenticates. If authentication is successful, a character selection screen is displayed, where the user selects the character they want to interact with.

Real-time voice conversation

The user initiates a conversation with the selected character. The device uses the microphone to capture the user's voice and converts it into text using the Google Cloud Speech-to-Text API or Microsoft Azure Speech Recognition Service. The text is sent to a server, which uses an AI model (e.g., GTP-3 or BERT) to generate an appropriate response. This response is converted into audio data by a speech synthesis engine (e.g., Amazon Polly or Google Text-to-Speech) and sent to the device. The device plays back the audio data, enabling real-time conversation.

Collection of usage fees and revenue sharing

The usage fee paid by users when using the service is collected by the device acquiring payment information. Payment processing is carried out by the server using the Stripe or PayPal API. The collected usage fee is distributed fairly among the copyright holders of the characters and people and the service provider.

Specific examples

1. User login and character selection

The user opens the app and logs in by entering "user123" and "password123" on the login screen. The device sends the input data to the server and authentication is performed. If authentication is successful, the home screen is displayed.

The user presses the "Character Selection" button on the home screen and selects "Character A." The device sends the selected character information to the server, and the server retrieves the corresponding AI data and sends it to the device.

2. Initiating and conducting actual conversations

The user presses the "Start Conversation" button and speaks "Hello, Character A" into the device's microphone. The device converts the speech into text and sends it to the server.

The server analyzes the text data and generates an appropriate response based on the AI model. For example, the response generated might be, "Hello, I'm Character A. How's your day?"

The generated response is converted into audio data and sent to the device. The device plays the audio data, allowing the user to hear the response.

Example prompt

"Hello, I'm Character A. How's your day going?"

This system allows fans to safely and easily interact with their favorite characters and people 24 hours a day, 365 days a year, and also reduces the high costs associated with hosting events and inviting celebrities to attend.

The flow of the identification process in Example 1 will be explained using Figure 11.

Step 1: Data Collection

The server collects data on the appearance, personality, and past behavior of specific characters or people. Inputs include links to YouTube videos and URLs of social media posts. The data is obtained using scraping technology and APIs. The output is the obtained text, images, and audio data.

Step 2: Data Preprocessing

The server preprocesses the collected data. The raw data collected in step 1 is provided as input. Specifically, processes such as noise removal, text tokenization, and normalization are performed. For example, unnecessary special characters are removed from the text data and it is converted into a standard format. The output is the preprocessed data.

Step 3: Train the model

The server uses the preprocessed data to train an artificial intelligence model (e.g., TensorFlow or PyTorch). The preprocessed data from step 2 is provided as input. Specifically, the data is divided into batches and the model parameters are optimized using a GPU. The output is a trained, highly accurate model.

Step 4: Evaluate and save the model

The server evaluates and saves the trained model. The model trained in step 3 and the validation dataset are used as input. Specifically, the accuracy of the model is evaluated using the validation data, and if the accuracy meets the criteria, the model is saved. The output is the evaluation results and the saved model.

Step 5: Save as an NFT

The server saves the trained AI model as an NFT. The model data saved in step 4 is used as input. Specifically, the model is converted to ONNX format and an NFT is issued on a blockchain platform (e.g., Ethereum or Binance Smart Chain). The output is the model data saved as an NFT.

Step 6: User authentication and character selection

The device displays a login screen for the user and performs authentication. A username and password are provided as input. The authentication data is sent to the server and checked against a database. If authentication is successful, the character selection screen is displayed. The output is information about the selected character.

Step 7: Capture and convert audio input

The device uses a microphone to capture the user's voice. The user's voice data is provided as input. Specifically, the voice is converted into text data using the Google Cloud Speech-to-Text API. The output is text data.

Step 8: Generate a response

The server receives the text data and generates a response using an AI model. The text data converted in step 7 is provided as input. Specifically, the AI model (e.g., GPT-3 or BERT) analyzes the text data and generates an appropriate response. The output is the text data of the generated response.

Step 9: Convert to audio and send

The server converts the generated text response into audio data and sends it to the device. The text data generated in step 8 is provided as input. Specifically, it converts it into audio data using Amazon Polly or Google Text-to-Speech. The output is audio data.

Step 10: Play real-time audio conversations

The device receives the audio data and plays it on the speaker. The input is the audio data sent in step 9. Specifically, it decodes the audio data and plays it on the speaker so that the user can hear it. The output is audio played in real time.

Step 11: Collection and distribution of usage fees

The terminal obtains payment information to collect usage fees and sends it to the server. Payment information is provided as input. The server processes the payment using Stripe or PayPal APIs and distributes revenue among the parties involved. Specifically, after confirming the payment process, the revenue is distributed via bank transfer or cryptocurrency. The output is the distributed revenue.

(Application Example 1)

Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the headset-type terminal 314 will be referred to as the "terminal."

Currently, providing high-quality customer service to customers visiting physical stores requires a lot of labor and resources. It is also difficult to provide detailed information about specific products in the store, making it difficult to increase customer satisfaction. Furthermore, while there is a demand for personalized guidance that allows customers to select products based on their preferences and interests, achieving this is complex. To solve these challenges, a system that can interact with customers in real time and provide personalized guidance is needed.

The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

In this invention, the server includes means for learning the characteristics of a specific virtual character using machine learning technology, means for saving the learned information as a non-fungible character (NFT), means for providing an operation screen on which a user can select a conversation partner and conduct a conversation, means for real-time voice communication, means for receiving service fees from users and distributing revenue among relevant parties, means for providing guidance using a display device that supports conversations between customers and virtual characters in physical stores, and means for users to obtain product information in the store and receive support through the character. This makes it possible to improve the customer experience in physical stores and provide personalized, high-quality customer service.

"Characteristics of a specific virtual person" refers to data including information such as the appearance, personality, and past words and actions of a character or celebrity.

"Machine learning technology" is a technology that allows computers to automatically learn from data and make predictions and classifications.

A "Non-Fungible Trading Card (NFT)" is a digital asset whose uniqueness and ownership are guaranteed using blockchain technology.

An "operation screen" is a display screen that provides an interface for users to select a conversation partner and conduct a conversation.

"Real-time voice communication" refers to technology that allows users to instantly communicate via voice with a character of their choice.

"Service fee" refers to the amount of money paid by the user in consideration for using the service.

"Related parties" refers to people or organizations that are subject to revenue sharing, such as character or system providers and developers.

"Display devices" refer to hardware for displaying information, such as displays installed in physical stores, smart glasses, and head-mounted displays (HMDs).

"Supporting virtual character conversations" refers to a specific virtual character communicating with the user via voice, providing guidance and information.

"Get product information" means viewing or obtaining detailed information about products in the store.

"Receiving support through a character" means that a virtual character guides or explains products or services to the user.

To implement this invention, the following system configuration is required. The system consists of a server, a terminal, and a user as its main components.

1. System Programming

The server builds a database that uses machine learning technology to learn the characteristics of specific virtual characters. The terminal provides an operation screen that allows the user to select a conversation partner and engage in real-time voice communication with that virtual character. The server also stores the learned information as a non-fungible character (NFT).

The terminal displays an operation screen that allows the user to select a virtual person to interact with, and transmits the selection information to the server. When the user communicates via voice in real time, the terminal converts the user's voice into text information and transmits that text information to the server.

The server feeds the received text information into a machine learning model to generate a response based on the conversation. The generated response is converted into audio data by the server and sent to the device. The device plays the audio data and provides the virtual character's response to the user.

2. Processing Description

The server is implemented using programming languages such as Python. Machine learning techniques include the NLP (natural language processing) model GPT-3. Libraries such as Google Speech Recognition are used for voice recognition and pyttsx3 for voice synthesis.

The device includes hardware such as smart glasses, a head-mounted display (HMD), and a microphone. The device recognizes the user's voice and sends it to the server. The server converts the voice into text and generates an appropriate response using an NLP model. The server then converts the response into audio data and sends it back to the device.

As a specific example, a user enters a physical store and puts on a pair of smart glasses. When the user says, "What products do you recommend?", the device's microphone picks up the user's voice and sends it to the server. The server analyzes the text data and generates a response using GPT-3. The response might be something like, "Currently, we recommend this smartwatch. It's the latest model with a long battery life and comprehensive health management features."

The generated response is converted into audio data and sent to the device. The device then plays the audio data, allowing the user to hear the voice of the virtual character.

3. Specific examples and prompt sentence examples

As a practical example, consider a situation where a user wears smart glasses and asks for assistance in a store. For example, the user might say, "Hello, what products do you recommend?"

Example prompt:

Customer Question: "Hello, what products do you recommend?"

Prompt for generative AI model: "A customer asked you for a product recommendation. How would you respond?"

The flow of the specific processing in Application Example 1 will be explained using Figure 12.

Program processing steps

Step 1:

The user puts on the smart glasses and launches the application. The input is the smart glasses' initialization data, and the output is the display of a login screen. Specifically, the device displays the login screen to the user, and the user enters their login information.

Step 2:

The user enters login information and presses the login button. The input is login information, and the output is an authentication request to the server. The device sends the user's input information to the server, and the server performs authentication.

Step 3:

The server authenticates the user's login information and returns the result to the device. The input is the user's login information, and the output is the authentication result. Specifically, the server searches the database and confirms the login information. If successful, the home screen is displayed on the device.

Step 4:

The user selects a character to interact with from the home screen. The input is character selection information, and the output is sending the selection information to the server. The device sends the selected character information to the server, and the server obtains the corresponding learning data.

Step 5:

The server obtains the learning data for the selected character and sends it to the device. The input is the character selection information, and the output is the learning data. The server retrieves the relevant data from the learning database and sends it to the device.

Step 6:

The user presses the "Start conversation" button and speaks into the device's microphone. The input is the user's voice, and the output is text data. The device uses voice recognition to convert the user's voice into text and sends the text data to the server.

Step 7:

The server receives the user's text data and generates an appropriate response using a generative AI model. The input is the user's text data and the output is the generated response. The server generates the response using an NLP model (e.g., GPT-3).

Step 8:

The response generated by the server is converted into audio data and sent to the device. The input is the generated response, and the output is audio data. The server uses a speech synthesis library (e.g., pyttsx3) to generate the audio data and send it to the device.

Step 9:

The device plays back the audio data received from the server and lets the user listen. The input is audio data, and the output is audio playback. The device plays back the audio data, allowing the user to hear the virtual character's response.

Step 10:

When the user wants to end the service, they perform an operation to end the application. The input is the end operation, and the output is to end the application. The device will end the application and stop displaying the smart glasses.

It is also possible to further combine an emotion engine that estimates the user's emotion. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

This invention is a system that uses artificial intelligence technology to learn the appearance and personality of specific characters or people, stores that learning data as non-fungible tokens (NFTs), and enables real-time voice conversations with users 24 hours a day, 365 days a year. Furthermore, by combining this invention with an emotion engine, it is possible to recognize the user's emotions and provide adaptive responses based on those emotions.

System Configuration

1. Learning with AI technology

The server collects data on the appearance, personality, and past behavior of specific characters or people, and uses an artificial intelligence algorithm to learn this information. The learning results, including the character's specific behavior and response patterns, are saved as a highly accurate model.

2. Save as an NFT

The server stores the learned AI data as an NFT, using blockchain technology to guarantee its uniqueness and ownership.

3. Providing an interface

The device provides the user with an interface for selecting a conversation partner. Users can log in and select their favorite character.

4. Real-time voice conversation

The user performs an operation to start a conversation with the selected character. The device captures the user's voice, converts it into text data, and sends it to the server.

Based on the text data received by the server, an appropriate response is generated using an AI model. This response is then converted back into audio data and sent to the device.

The device plays the audio data and the conversation takes place in real time.

5. Emotional Engine Integration

The server is equipped with an emotion engine that uses the user's voice data to analyze emotions. The emotion engine analyzes the user's emotional state (e.g., joy, sadness, anger, etc.) from the voice data.

Based on the results of emotion analysis, the AI model adaptively changes the response content. For example, if the user speaks in a sad voice, a response such as "Cheer up!" will be generated.

6. Usage Fee Collection and Revenue Sharing

The server collects usage fees from users. Payments are processed through the terminal. Revenues are distributed among the parties involved, including the copyright holders of characters and people, and service providers.

Specific examples

1. The user logs in and selects a character.

The user opens the app and logs in by entering "user123" and "password123" on the login screen. The device sends the entered data to the server and authentication is performed. If authentication is successful, the home screen is displayed on the device.

The user presses the "Select Character" button on the home screen and selects, for example, "Character A." The device sends the selected character information to the server, and the server retrieves the corresponding AI data and sends it to the device.

2. Initiating and conducting actual conversations

The user presses the "Start Conversation" button and speaks "Hello, Character A" into the device's microphone. The device converts the speech into text and sends it to the server.

The server analyzes the text data and generates an appropriate response based on an AI model. For example, it might respond with "Hello! How's your day?"

At the same time, the server sends the voice data to the emotion engine, which analyzes the user's emotional state.

The emotion engine analyzes the user's emotional state and feeds the results back to the AI model. For example, if the user sounds sad, the response will be changed to an encouraging message such as "Cheer up!"

The generated response is converted into audio data and sent to the device, which then plays the audio data and provides the response to the user.

3. Payment of fees and revenue sharing

If the user needs to pay a usage fee, the terminal will display a payment screen. The user will enter their credit card information, etc.

The terminal sends the entered payment information to the server, which processes the payment through the payment gateway.

The server sends information about the success or failure of the payment to the terminal. If successful, the usage rights are updated.

The terminal will notify the user of the payment result, allowing them to continue using the system.

This system allows fans to safely and easily interact with their favorite characters and people 24 hours a day, 365 days a year, and by providing appropriate responses based on the user's emotions, it enables more personal and intimate communication.

The processing flow is explained below.

Step 1:

The user opens the application. The user accesses the login screen and enters their ID and password.

Step 2:

The device receives the user's input and sends the ID and password information to the server. The communication is encrypted.

Step 3:

The server checks the received ID and password against a database and authenticates the user. If the ID and password do not match, an error message is sent to the terminal.

Step 4:

If the server successfully authenticates, it generates session information and sends it back to the device. The device uses the received session information to display the home screen.

Step 5:

The user presses the "Character Selection" button on the home screen and selects the character they want to interact with.

Step 6:

The device receives the user's selection and sends the character ID to the server. The server then retrieves the AI model data for the selected character.

Step 7:

The server sends the acquired AI model data to the device. The device displays a conversation interface with the character based on the received data.

Step 8:

The user presses the "Start conversation" button to begin the conversation. The device's microphone is turned on to capture the user's voice.

Step 9:

The device records the user's voice and converts the voice data into text data in real time. This text data is then sent to the server.

Step 10:

The server analyzes the received text data and generates a response using an AI model. For example, if the user says "Hello, Character A," the response will be "Hello! How's your day?"

Step 11:

At the same time, the server sends the user's voice data to the emotion engine, which analyzes the user's emotional state. The emotion engine analyzes the voice data and identifies the user's emotional state (joy, sadness, anger, etc.).

Step 12:

The emotion engine sends the analysis results back to the server, which then adaptively changes the response based on the results. For example, if the user speaks in a sad voice, the response will be "Cheer up!"

Step 13:

The server converts the response text into audio data and sends it to the device.

Step 14:

The device plays back the received audio data and provides a response to the user. The user can then listen to this and ask questions or make comments.

Step 15:

If the user needs to pay a usage fee, the terminal will display a payment screen. The user will then enter their credit card information, etc.

Step 16:

The terminal sends the entered payment information to the server, which then processes the payment through the payment gateway.

Step 17:

The server sends information about the success or failure of the payment to the terminal. If successful, the usage rights are updated.

Step 18:

The terminal will notify the user of the payment result, allowing them to continue using the system.

(Example 2)

Next, Example 2 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset-type terminal 314 will be referred to as the "terminal."

With conventional voice conversation systems using artificial intelligence, it was difficult to efficiently learn the appearance and personality of characters or people and conduct real-time voice conversations with users. Furthermore, technology for providing adaptive responses based on the user's emotions was not sufficiently developed, making it impossible to achieve natural, personal communication with users.

The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

In this invention, the server includes means for learning the appearance and personality of a specific character or person using artificial intelligence technology, means for saving the learned data as a non-fungible token (NFT), and means for analyzing emotions from the user's voice and adaptively changing responses based on the analysis results. This makes it possible to learn the characteristics of characters and people with high accuracy, safely store that data, and provide flexible responses that correspond to the user's emotions.

"Specific characters or persons" refers to fictional or real people or characters that users can select as their conversation partners.

"Artificial intelligence technology" refers to technology that uses techniques such as machine learning and deep learning to extract patterns from data and make predictions and classifications.

"Means of learning" refers to the process of collecting data and generating an AI model based on that data.

A "non-fungible token (NFT)" is a token that uses blockchain technology to guarantee the uniqueness and ownership of digital data.

"Means of providing an interface" refers to the method of providing screens and menus that allow users to access and operate the system.

"Means for conducting real-time voice conversation" refers to technology that allows users and characters to have instant voice conversations.

"Means for analyzing emotions from a user's voice" refers to technology that analyzes a user's emotional state based on their voice data.

"Means for adaptively changing responses" refers to a method for changing the AI's response content based on the results of emotion analysis.

"Means for collecting usage fees and distributing revenue among parties" refers to a system for collecting service usage fees from users and distributing the revenue obtained among parties.

Form for implementing the invention

This invention is a system that uses artificial intelligence technology to learn the appearance and personality of specific characters or people, stores that learning data as non-fungible tokens (NFTs), and provides users with real-time voice conversations 24 hours a day, 365 days a year. Furthermore, by combining it with an emotion engine, it is possible to recognize the user's emotions and provide adaptive responses based on those emotions.

System Overview

The system consists of the following main components:

1. Server

2. Device

3. User

Server Role

1. Training data collection and AI model generation

The server collects data on the appearance, personality, and past behavior of specific characters and people. This data includes video, audio files, text data, etc.

The collected data is trained using an AI algorithm (e.g., TensorFlow or PyTorch). During the initial training stage, data preprocessing (noise removal, text normalization, etc.) is performed.

Save the completed model in the database.

2. Data storage as NFTs

The learned AI data is stored as a non-fungible token (NFT). This process uses blockchain technology (e.g., Ethereum).

The server registers the generated NFT on the blockchain and maintains owner information.

3. Sentiment Analysis and Response Adaptation

The server uses an emotion engine (for example, IBM Watson's emotion recognition API) to analyze the user's voice data. Based on the analysis results, the AI model adaptively changes the response content.

For example, if a user says, "I'm feeling a bit down today," emotion recognition can generate a response such as, "Cheer up!"

4. Usage fee collection and revenue sharing

The server collects the usage fee and manages the payment process. Payment is made using credit card information entered by the user on the terminal.

Revenues will be distributed among the parties involved (character copyright holders, service providers, etc.).

Device role

1. Providing an interface

The device provides the user with an interface to select a conversation partner and conduct a conversation. The user launches the app, enters their authentication information on the login screen, and logs in.

After logging in, users can press the "Select Character" button to choose their preferred character and begin a conversation with that character.

2. Audio data processing

The device captures the user's voice and converts it into text data using a speech recognition engine such as Google Speech-to-Text.

The converted text data is sent to the server, and after an appropriate response is generated, it is converted back into voice data and sent to the device.

The device plays the audio data, enabling real-time conversations.

User Roles

1. Log in and select a character

The user launches the app and enters authentication information such as "user123" and "password123" on the login screen.

Once authentication is successful, select a character from the home screen and begin a conversation with the character you select.

2. Actual conversation progression

The user presses the "Start Conversation" button and speaks into the device's microphone, saying "Hello, Character A."

The device converts this voice into text data and sends it to the server. The server uses an AI model to generate a response based on the text data and sends an adaptive reply.

For example, you can receive responses such as "Hello! How are you today?" or adaptive messages such as "Cheer up!"

Examples and prompts

"Hello, Character A. How was your day?"

"I've been feeling a bit down lately. Can you cheer me up?"

"What do you think about today's news?"

By using these prompts, users can enjoy more personal and natural conversations with the characters.

The flow of the identification process in Example 2 will be explained using Figure 13.

Program processing flow

Step 1:

Training data collection and AI model generation

The server collects data on the appearance, personality, and past behavior of specific characters or people. The data used for this collection includes text, audio files, images, and videos. Specifically, the data is retrieved from a database via web scraping or API and stored on the server.

Input: Data about the character's appearance, personality, and past words and actions.

Data processing: noise removal, text normalization, data cleaning.

Output: Clean dataset after preprocessing.

The server uses an AI algorithm (such as TensorFlow or PyTorch) to train a model using this data. The trained model then reflects the character's behavior and response patterns.

Input: Clean dataset after preprocessing.

Data computation: Applying machine learning algorithms and training models.

Output: Trained AI model.

Step 2:

Storing data as NFTs

The server stores the learned AI data as a non-fungible token (NFT).

Input: Trained AI model.

Data processing: Blockchain technology (e.g., Ethereum) is used to ensure the uniqueness and ownership of the model for NFT conversion.

Output: NFT registered on the blockchain.

The server registers the generated NFT on the blockchain and retains owner information. Specifically, the NFT is registered through the blockchain API.

Step 3:

User login and character selection

The user launches the app and enters their authentication information (e.g., "user123" and "password123") on the login screen.

The device sends the entered authentication information to the server and performs authentication.

Input: User authentication information.

Data calculation: Compare with the database and determine whether authentication is successful.

Output: Authentication success or failure status.

If authentication is successful, the home screen will be displayed on the device. The user presses the "Select Character" button to choose the character they want to interact with.

Input: Character information selected by the user.

Data processing: Search and obtain the AI model corresponding to the selected character.

Output: AI data for the selected character.

Step 4:

Real-time voice conversation processing

The user presses the "Start conversation" button and speaks into the device's microphone.

The device captures the user's voice and converts it into text data using Google Speech-to-Text.

Input: User's voice data.

Data processing: voice-to-text conversion.

Output: Text data.

The converted text data is sent to the server, which generates an appropriate response.

Input: User's text data.

Data calculations: Use AI models to generate appropriate responses.

Output: The generated text response.

The response is converted back into audio data and sent to the device.

Input: The generated text response.

Data processing: converting text to audio.

Output: Generated audio data.

The device plays the audio data and the conversation takes place in real time.

Step 5:

Sentiment analysis and response adaptation

The server analyzes the user's voice data using an emotion engine (e.g., IBM Watson's emotion recognition API).

Input: User's voice data.

Data calculation: Analyze the user's emotional state from voice data.

Output: User's emotional state data.

Based on the results of emotion analysis, the AI model adaptively changes the response content.

Input: User emotional state data and generated text response.

Data calculation: Change the response content to match the emotional state.

Output: The final adapted response text.

For example, if the user sounds sad, the response will be changed to something like "Cheer up!"

Step 6:

Collecting usage fees and processing payments

If the user needs to pay a usage fee, the terminal will display a payment screen. The user will enter their credit card information, etc.

Input: User's payment information.

The terminal sends payment information to the server, which processes the payment through a payment gateway (e.g., Stripe).

Data calculation: Executes payment processing based on payment information.

Output: Payment success or failure status.

The server notifies the terminal of the success or failure of the payment, and if successful, the usage rights are updated.

The terminal will notify the user of the payment result, allowing them to continue using the system.

(Application Example 2)

Next, we will explain Application Example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the headset-type terminal 314 will be referred to as the "terminal."

The objective of this invention is to provide a system that can reproduce the appearance and personality of a specific character or person in real time, while providing appropriate responses based on the user's emotions through dialogue with them. Furthermore, the aim is to realize a more personal and sympathetic shopping experience by having characters act as shopping assistants in virtual stores, explaining products and recommending items.

The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

In this invention, the server includes: means for learning the appearance and personality of a specific character or person using artificial intelligence technology; means for storing the learned data as a non-fungible token (NFT); means for providing an interface for users to select a conversation partner and converse with them; means for conducting real-time voice conversations; means for collecting usage fees from users and distributing revenue among the parties involved; means for analyzing the emotional state from the user's voice data using an emotion engine and generating an adaptive response based on the analysis results; means for converting the generated response into voice data and providing it to the user as voice; and means for the character to act as a shopping assistant in a virtual store, explaining products and recommending items. This enables real-time, emotion-based dialogue with a specific character and a personalized, sympathetic shopping assistant experience in a virtual store.

"Specific characters or people" are virtual beings whose appearances and personalities are reproduced using artificial intelligence technology.

"Artificial intelligence technology" is a technology that learns the appearance and personality of a specific character or person, and generates responses and actions based on that information.

A non-fungible token (NFT) is a means of using blockchain technology to uniquely identify a specific digital asset and prove ownership of it.

An "interface" is a device or software that provides input and display means for a user to select a conversation partner and initiate a conversation.

"Means for conducting real-time voice conversations" refers to technology that allows users to have instant voice conversations with specific characters or people.

An "emotion engine" is an algorithm and system that analyzes a user's emotional state from their voice data and generates a response based on the analysis results.

An "adaptive response" is a response that changes depending on the user's speech content and emotional state, and is generated individually and dynamically.

A "virtual store" is a virtual shopping environment that exists on the Internet and is a platform that allows users to conduct purchasing activities in a virtual space.

A "shopping assistant" is a virtual character or person who explains products and recommends items to users in a virtual store.

"Collection" means receiving usage fees from users.

"Means for distributing revenues among parties involved" refers to a mechanism for appropriately distributing revenues earned through system operation to parties involved, such as character copyright holders and service providers.

System Overview

This invention is a system that recreates specific characters or people in real time and engages in voice conversations with users. It primarily functions as a shopping assistant in virtual stores. This system consists of three elements: a server, a terminal, and a user.

Program structure and various technologies

Speech recognition engine

The server receives the user's voice input and converts the voice into text data using a speech recognition engine (Google Speech-to-Text API).

Sentiment analysis engine

The server performs emotion analysis on the acquired text data using IBM Watson Tone Analyzer, thereby determining the user's emotional state.

Dialogue management system

The server uses Rasa to generate responses based on the user's conversation content and sentiment analysis results.

Speech synthesis engine

The server converts the generated response into audio data using the Google Text-to-Speech API and sends it to the device.

Program processing

1. Voice input and text conversion

When a user speaks into their smartphone's microphone, the audio is picked up by the device and converted into text data using the Google Speech-to-Text API.

2. Emotion analysis

The transferred text data is subjected to emotion analysis using IBM Watson Tone Analyzer to determine the user's emotional state (e.g., joy, sadness, anger, etc.).

3. Response Generation

Rasa is used to generate appropriate responses based on analysis results and conversation content. These responses are adapted to the user's emotions.

4. Conversion to audio data and transmission

The generated response is converted into audio data using the Google Text-to-Speech API, and this audio data is sent to the device, enabling real-time conversation.

Specific examples

1. User login and character selection

The user launches the app and logs in by entering "user123" and "password123." After logging in, they select "Character A" on the character selection screen.

Example prompt:

User: Hello, I'm looking for clothes today.

---

Assistant: Hello, what kind of clothes are you looking for?

---

User: I'd like a casual shirt.

---

2. Voice input and analysis

When a user says, "Hello, I'm looking for clothes today," the speech is captured and converted into text data. The text data is sent to IBM Watson Tone Analyzer for sentiment analysis.

3. Response generation and delivery

Based on sentiment analysis and the conversation content, Rasa generates a response: "Hello, what kind of clothes are you looking for?" This response is converted into audio using the Google Text-to-Speech API and sent to the device.

4. Shopping Assistant Function

If a user says, "I want a casual shirt," the assistant will suggest, "How about this shirt? It's a trendy design and very popular these days," and explain the product. It is also possible to provide more detailed product descriptions and suggestions based on the results of user sentiment analysis.

This allows the invention to provide emotion-based real-time interactions with specific characters and a personalized, immersive shopping assistant experience in virtual stores.

The flow of the specific processing in Application Example 2 will be explained using Figure 14.

Step 1:

The user launches the app and logs in. The user enters their username and password, and the device sends that information to the server. The server performs authentication, and if successful, displays the character selection screen on the device.

Enter: User's username and password

Process: The server authenticates the user.

Output: Upon successful authentication, the character selection screen will be displayed.

Step 2:

The user selects a character. The user chooses a specific character on the character selection screen, and the device sends the selection information to the server. The server loads the selected character's AI model and sends the relevant data to the device.

Input: User's character selection information

Processing: The server loads the AI model and retrieves relevant data.

Output: Character information is displayed on the device.

Step 3:

The user begins the conversation. They speak into the smartphone's microphone, and the device picks up the audio. The speech is converted into text data using a speech recognition engine (Google Speech-to-Text API), and the text data is sent to the server.

Input: User's voice

Processing: The device captures the audio and converts it into text data.

Output: The converted text data is sent to the server.

Step 4:

The server performs emotion analysis. The server sends the acquired text data to an emotion analysis engine (IBM Watson Tone Analyzer), which analyzes the user's emotional state. The analysis results are added to the text data.

Input: Text data

Processing: The server uses an emotion analysis engine to analyze the emotional state.

Output: Text data with emotional states added

Step 5:

The server generates a response. The server uses a dialogue management system (Rasa) to generate a response based on the sentiment analysis results and the content of the conversation. The response is in text format.

Input: Text data with sentiment analysis results added

Processing: The server generates a response using the dialogue management system.

Output: Generated response text

Step 6:

The server converts the response text into speech. The generated response text is converted into audio data using a speech synthesis engine (Google Text-to-Speech API). The converted audio data is then sent back to the device.

Input: Generated response text

Processing: The server uses a speech synthesis engine to convert the data into voice data.

Output: Converted audio data

Step 7:

The device plays the audio data. The audio data sent to the device is played back through the speaker and becomes audible to the user. This allows for real-time audio conversation between the user and the character.

Input: Audio data

Process: The device plays the audio data.

Output: User hears response

The specific processing unit 290 transmits the results of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the results of the specific processing. The microphone 238 acquires audio indicating the user input regarding the results of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

Data generation model 58 is what is known as generative AI (artificial intelligence). Examples of data generation model 58 include generative AI such as ChatGPT (Internet search <URL: https://openai.com/blog/chatgpt>) and Gemini (Internet search <URL: https://gemini.google.com/?hl=ja>). Data generation model 58 is obtained by performing deep learning on a neural network. A prompt containing an instruction is input to data generation model 58, and inference data such as voice data indicating speech, text data indicating text, and image data indicating an image is also input. Data generation model 58 performs inference on the input inference data in accordance with the instructions indicated by the prompt, and outputs the inference results in the form of data such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and/or summarization.

In the above embodiment, an example was given in which the specific processing was performed by the data processing device 12, but the technology disclosed herein is not limited to this, and the specific processing may also be performed by the headset-type terminal 314.

[Fourth embodiment]

Figure 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

As shown in FIG. 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

The data processing device 12 includes a computer 22, a database 24, and a communication I/F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and communication I/F 26 are also connected to the bus 34. The communication I/F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and/or a LAN (Local Area Network).

The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I/F 44, and a control target 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and control target 443 are also connected to the bus 52.

The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the user 20's surroundings (e.g., an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

The communication I/F 44 is connected to the network 54. The communication I/Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I/Fs 44 and 26 is carried out in a secure manner.

The control object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, in the data processing device 12, specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32.

The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as the specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

Storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

In the robot 414, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

Next, we will explain the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 will be referred to as the "terminal."

This invention is a system that uses artificial intelligence technology to learn the appearance and personality of specific characters or people, stores that learning data as non-fungible tokens (NFTs), and enables real-time voice conversations with users 24 hours a day, 365 days a year. The system is configured as follows:

System Configuration

1. Learning with AI technology

The server collects data on the appearance, personality, and past behavior of specific characters or individuals, and uses artificial intelligence algorithms to learn this information.

The learning results include the character's specific behavioral and response patterns and are saved as a highly accurate model.

2. Save as an NFT

The server stores the learned AI data as an NFT, using blockchain technology to guarantee its uniqueness and ownership.

3. Providing an interface

The device provides the user with an interface for selecting a conversation partner. Users can log in and select their favorite character.

4. Real-time voice conversation

The user performs an operation to start a conversation with the selected character. The device captures the user's voice, converts it into text data, and sends it to the server.

Based on the text data received by the server, an appropriate response is generated using an AI model. This response is then converted back into audio data and sent to the device.

The device plays the audio data and the conversation takes place in real time.

5. Usage Fee Collection and Revenue Sharing

The server collects usage fees from users. Payment processing is carried out through the terminal.

Revenues will be shared among the parties involved, including copyright holders of characters and people, and service providers.

Specific examples

1. The user logs in and selects a character.

The user opens the app and logs in by entering "user123" and "password123" on the login screen. The device sends the entered data to the server and authentication is performed. If authentication is successful, the home screen is displayed on the device.

The user presses the "Select Character" button on the home screen and selects, for example, "Character A." The device sends the selected character information to the server, and the server retrieves the corresponding AI data and sends it to the device.

2. Initiating and conducting actual conversations

The user presses the "Start Conversation" button and speaks "Hello, Character A" into the device's microphone. The device converts the speech into text and sends it to the server.

The server analyzes the text data and generates an appropriate response based on the AI model. For example, the response generated might be, "Hello, I'm Character A. How's your day?"

The generated response is converted into audio data and sent to the device. The device plays the audio data, allowing the user to hear the response.

3. Payment of fees and revenue sharing

When a user pays a usage fee, the terminal captures the payment information and sends it to the server. The server processes the payment using a payment gateway and, after confirmation, distributes the revenue among the parties.

This system allows fans to safely and easily interact with their favorite characters and people 24 hours a day, 365 days a year, and also reduces the high costs associated with hosting events and inviting celebrities to attend.

The processing flow is explained below.

Step 1:

The user opens the application. The user accesses the login screen and enters their ID and password.

Step 2:

The device receives the user's input and sends the ID and password information to the server. The communication is encrypted.

Step 3:

The server checks the received ID and password against a database and authenticates the user. If the ID and password do not match, an error message is sent to the terminal.

Step 4:

If the server successfully authenticates, it generates session information and sends it back to the device. The device uses the received session information to display the home screen.

Step 5:

The user presses the "Character Selection" button on the home screen and selects the character they want to interact with.

Step 6:

The device receives the user's selection and sends the character ID to the server. The server then retrieves the AI model data for the selected character.

Step 7:

The server sends the acquired AI model data to the device. The device then displays a conversation interface with the character based on the received data.

Step 8:

The user presses the "Start conversation" button to begin the conversation. The device's microphone is turned on to capture the user's voice.

Step 9:

The device records the user's voice and converts the voice data into text data in real time. This text data is then sent to the server.

Step 10:

The server analyzes the received text data and generates a response using an AI model. For example, if the user says "Hello, Character A," the response will be "Hello! How's your day?"

Step 11:

The response text generated by the server is converted into audio data and sent to the device.

Step 12:

The device plays back the received audio data and provides a response to the user. The user can then listen to this and ask questions or make comments.

Step 13:

If the user needs to pay a usage fee, the terminal will display a payment screen. The user will then enter their credit card information, etc.

Step 14:

The terminal sends the entered payment information to the server, which then processes the payment through the payment gateway.

Step 15:

The server sends information about the success or failure of the payment to the terminal. If successful, the usage rights are updated.

Step 16:

The terminal will notify the user of the payment result, allowing them to continue using the system.

(Example 1)

Next, Example 1 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 will be referred to as the "terminal."

In conventional dialogue systems with characters or people, it was difficult to achieve real-time voice conversation, and users were unable to receive a satisfactory experience. Furthermore, there was a lack of means to guarantee the uniqueness and ownership of trained AI data, creating a risk of data tampering or unauthorized use. Furthermore, there was no established mechanism for collecting usage fees from users and distributing revenue fairly among the parties involved. New technology is needed to solve these problems.

The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

In this invention, the server includes: means for learning the appearance and personality of a specific character or person using artificial intelligence technology; means for storing the learned data as a non-fungible token (NFT); means for providing an interface for users to select a conversation partner and converse with them; means for acquiring the user's voice and converting it into text data using voice recognition technology; means for generating an appropriate response based on the text data using an AI model and converting the response into voice data; means for transmitting the converted voice data to the user's terminal to conduct voice conversations in real time; and means for collecting usage fees from users and distributing revenue among the parties involved. This enables real-time voice conversations, guarantees the uniqueness and ownership of learned data, and enables fair revenue distribution.

"Specific characters or people" refers to fictional or real people that users select when interacting, whose appearance, personality, behavior patterns, etc. are predefined.

"Artificial intelligence technology" refers to technology that uses machine learning and deep learning models to learn from specific data and generate appropriate responses even for unknown data.

A non-fungible token (NFT) is a unique digital asset created using blockchain technology with guaranteed ownership and authenticity.

"Interface" refers to the user interface provided to users to select and interact with characters or people, and is usually provided as an application.

"Speech recognition technology" refers to technology that converts a user's voice into text data, and includes services such as Google Cloud Speech-to-Text and Microsoft Azure Speech Recognition Services.

An "AI model" is a neural network model trained based on collected data and used to generate the responses of a specific character or person.

"Converting into voice data" means using speech synthesis technology to convert the text response generated by the AI model into voice data that users can hear.

"Real-time voice conversation" refers to a form in which users can instantly communicate back and forth with characters or people using voice.

"Collecting usage fees" means collecting fees from users for using a service, often using an online payment system.

"Distributing revenues among the parties involved" means distributing the collected usage fees in fair proportions to copyright holders of characters and people, service providers, and other rights holders.

Form for implementing the invention

This invention is a system that uses artificial intelligence technology to learn the appearance and personality of specific characters or people, stores that learning data as non-fungible tokens (NFTs), and enables real-time voice conversations with users 24 hours a day, 365 days a year. The system is configured as follows:

Learning using artificial intelligence technology

The server collects data on the appearance, personality, and past behavior of specific characters or people. For example, it uses scripts that retrieve data from YouTube, social media, etc. This data is trained using the TensorFlow or PyTorch machine learning frameworks. During the data preprocessing process, noise removal, tokenization, normalization, etc. are performed, and a neural network model is built. The trained model is then evaluated for accuracy and saved if it is highly accurate.

Stored as an NFT

The server converts the model data into ONNX format to store the trained AI data as an NFT. The NFT is then issued using a blockchain platform such as Ethereum or Binance Smart Chain, which incorporates the model's unique identifier and metadata in the process.

Providing an interface

The device provides the user with an interactive user interface. The user logs in using an application developed with React Native or Flutter. The login screen has username and password input fields, and the user authenticates. If authentication is successful, a character selection screen is displayed, where the user selects the character they want to interact with.

Real-time voice conversation

The user initiates a conversation with the selected character. The device uses the microphone to capture the user's voice and converts it into text using the Google Cloud Speech-to-Text API or Microsoft Azure Speech Recognition Service. The text is sent to a server, which uses an AI model (e.g., GTP-3 or BERT) to generate an appropriate response. This response is converted into audio data by a speech synthesis engine (e.g., Amazon Polly or Google Text-to-Speech) and sent to the device. The device plays back the audio data, enabling real-time conversation.

Collection of usage fees and revenue sharing

The usage fee paid by users when using the service is collected by the device acquiring payment information. Payment processing is carried out by the server using the Stripe or PayPal API. The collected usage fee is distributed fairly among the copyright holders of the characters and people and the service provider.

Specific examples

1. User login and character selection

The user opens the app and logs in by entering "user123" and "password123" on the login screen. The device sends the input data to the server and authentication is performed. If authentication is successful, the home screen is displayed.

The user presses the "Character Selection" button on the home screen and selects "Character A." The device sends the selected character information to the server, and the server retrieves the corresponding AI data and sends it to the device.

2. Initiating and conducting actual conversations

The user presses the "Start Conversation" button and speaks "Hello, Character A" into the device's microphone. The device converts the speech into text and sends it to the server.

The server analyzes the text data and generates an appropriate response based on the AI model. For example, the response generated might be, "Hello, I'm Character A. How's your day?"

The generated response is converted into audio data and sent to the device. The device plays the audio data, allowing the user to hear the response.

Example prompt

"Hello, I'm Character A. How's your day going?"

This system allows fans to safely and easily interact with their favorite characters and people 24 hours a day, 365 days a year, and also reduces the high costs associated with hosting events and inviting celebrities to attend.

The flow of the identification process in Example 1 will be explained using Figure 11.

Step 1: Data Collection

The server collects data on the appearance, personality, and past behavior of specific characters or people. Inputs include links to YouTube videos and URLs of social media posts. The data is obtained using scraping technology and APIs. The output is the obtained text, images, and audio data.

Step 2: Data Preprocessing

The server preprocesses the collected data. The raw data collected in step 1 is provided as input. Specifically, processes such as noise removal, text tokenization, and normalization are performed. For example, unnecessary special characters are removed from the text data and it is converted into a standard format. The output is the preprocessed data.

Step 3: Train the model

The server uses the preprocessed data to train an artificial intelligence model (e.g., TensorFlow or PyTorch). The preprocessed data from step 2 is provided as input. Specifically, the data is divided into batches and the model parameters are optimized using a GPU. The output is a trained, highly accurate model.

Step 4: Evaluate and save the model

The server evaluates and saves the trained model. The model trained in step 3 and the validation dataset are used as input. Specifically, the accuracy of the model is evaluated using the validation data, and if the accuracy meets the criteria, the model is saved. The output is the evaluation results and the saved model.

Step 5: Save as an NFT

The server saves the trained AI model as an NFT. The model data saved in step 4 is used as input. Specifically, the model is converted to ONNX format and an NFT is issued on a blockchain platform (e.g., Ethereum or Binance Smart Chain). The output is the model data saved as an NFT.

Step 6: User authentication and character selection

The device displays a login screen for the user and performs authentication. A username and password are provided as input. The authentication data is sent to the server and checked against a database. If authentication is successful, the character selection screen is displayed. The output is information about the selected character.

Step 7: Capture and convert audio input

The device uses a microphone to capture the user's voice. The user's voice data is provided as input. Specifically, the voice is converted into text data using the Google Cloud Speech-to-Text API. The output is text data.

Step 8: Generate a response

The server receives the text data and generates a response using an AI model. The text data converted in step 7 is provided as input. Specifically, the AI model (e.g., GPT-3 or BERT) analyzes the text data and generates an appropriate response. The output is the text data of the generated response.

Step 9: Convert to audio and send

The server converts the generated text response into audio data and sends it to the device. The text data generated in step 8 is provided as input. Specifically, it converts it into audio data using Amazon Polly or Google Text-to-Speech. The output is audio data.

Step 10: Play real-time audio conversations

The device receives the audio data and plays it on the speaker. The input is the audio data sent in step 9. Specifically, it decodes the audio data and plays it on the speaker so that the user can hear it. The output is audio played in real time.

Step 11: Collection and distribution of usage fees

The terminal obtains payment information to collect usage fees and sends it to the server. Payment information is provided as input. The server processes the payment using Stripe or PayPal APIs and distributes revenue among the parties involved. Specifically, after confirming the payment process, the revenue is distributed via bank transfer or cryptocurrency. The output is the distributed revenue.

(Application Example 1)

Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 will be referred to as the "terminal."

Currently, providing high-quality customer service to customers visiting physical stores requires a lot of labor and resources. It is also difficult to provide detailed information about specific products in the store, making it difficult to increase customer satisfaction. Furthermore, while there is a demand for personalized guidance that allows customers to select products based on their preferences and interests, achieving this is complex. To solve these challenges, a system that can interact with customers in real time and provide personalized guidance is needed.

The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

In this invention, the server includes means for learning the characteristics of a specific virtual character using machine learning technology, means for saving the learned information as a non-fungible character (NFT), means for providing an operation screen on which a user can select a conversation partner and conduct a conversation, means for real-time voice communication, means for receiving service fees from users and distributing revenue among relevant parties, means for providing guidance using a display device that supports conversations between customers and virtual characters in physical stores, and means for users to obtain product information in the store and receive support through the character. This makes it possible to improve the customer experience in physical stores and provide personalized, high-quality customer service.

"Characteristics of a specific virtual person" refers to data including information such as the appearance, personality, and past words and actions of a character or celebrity.

"Machine learning technology" is a technology that allows computers to automatically learn from data and make predictions and classifications.

A "Non-Fungible Trading Card (NFT)" is a digital asset whose uniqueness and ownership are guaranteed using blockchain technology.

An "operation screen" is a display screen that provides an interface for users to select a conversation partner and conduct a conversation.

"Real-time voice communication" refers to technology that allows users to instantly communicate via voice with a character of their choice.

"Service fee" refers to the amount of money paid by the user in consideration for using the service.

"Related parties" refers to people or organizations that are subject to revenue sharing, such as character or system providers and developers.

"Display devices" refer to hardware for displaying information, such as displays installed in physical stores, smart glasses, and head-mounted displays (HMDs).

"Supporting virtual character conversations" refers to a specific virtual character communicating with the user via voice, providing guidance and information.

"Get product information" means viewing or obtaining detailed information about products in the store.

"Receiving support through a character" means that a virtual character guides or explains products or services to the user.

To implement this invention, the following system configuration is required. The system consists of a server, a terminal, and a user as its main components.

1. System Programming

The server builds a database that uses machine learning technology to learn the characteristics of specific virtual characters. The terminal provides an operation screen that allows the user to select a conversation partner and engage in real-time voice communication with that virtual character. The server also stores the learned information as a non-fungible character (NFT).

The terminal displays an operation screen that allows the user to select a virtual person to interact with, and transmits the selection information to the server. When the user communicates via voice in real time, the terminal converts the user's voice into text information and transmits that text information to the server.

The server feeds the received text information into a machine learning model to generate a response based on the conversation. The generated response is converted into audio data by the server and sent to the device. The device plays the audio data and provides the virtual character's response to the user.

2. Processing Description

The server is implemented using programming languages such as Python. Machine learning techniques include the NLP (natural language processing) model GPT-3. Libraries such as Google Speech Recognition are used for voice recognition and pyttsx3 for voice synthesis.

The device includes hardware such as smart glasses, a head-mounted display (HMD), and a microphone. The device recognizes the user's voice and sends it to the server. The server converts the voice into text and generates an appropriate response using an NLP model. The server then converts the response into audio data and sends it back to the device.

As a specific example, a user enters a physical store and puts on a pair of smart glasses. When the user says, "What products do you recommend?", the device's microphone picks up the user's voice and sends it to the server. The server analyzes the text data and generates a response using GPT-3. The response might be something like, "Currently, we recommend this smartwatch. It's the latest model with a long battery life and comprehensive health management features."

The generated response is converted into audio data and sent to the device. The device then plays the audio data, allowing the user to hear the voice of the virtual character.

3. Specific examples and prompt sentence examples

As a practical example, consider a situation where a user wears smart glasses and asks for assistance in a store. For example, the user might say, "Hello, what products do you recommend?"

Example prompt:

Customer Question: "Hello, what products do you recommend?"

Prompt for generative AI model: "A customer asked you for a product recommendation. How would you respond?"

The flow of the specific processing in Application Example 1 will be explained using Figure 12.

Program processing steps

Step 1:

The user puts on the smart glasses and launches the application. The input is the smart glasses' initialization data, and the output is the display of a login screen. Specifically, the device displays the login screen to the user, and the user enters their login information.

Step 2:

The user enters login information and presses the login button. The input is login information, and the output is an authentication request to the server. The device sends the user's input information to the server, and the server performs authentication.

Step 3:

The server authenticates the user's login information and returns the result to the device. The input is the user's login information, and the output is the authentication result. Specifically, the server searches the database and confirms the login information. If successful, the home screen is displayed on the device.

Step 4:

The user selects a character to interact with from the home screen. The input is character selection information, and the output is sending the selection information to the server. The device sends the selected character information to the server, and the server obtains the corresponding learning data.

Step 5:

The server obtains the learning data for the selected character and sends it to the device. The input is the character selection information, and the output is the learning data. The server retrieves the relevant data from the learning database and sends it to the device.

Step 6:

The user presses the "Start conversation" button and speaks into the device's microphone. The input is the user's voice, and the output is text data. The device uses voice recognition to convert the user's voice into text and sends the text data to the server.

Step 7:

The server receives the user's text data and generates an appropriate response using a generative AI model. The input is the user's text data and the output is the generated response. The server generates the response using an NLP model (e.g., GPT-3).

Step 8:

The response generated by the server is converted into audio data and sent to the device. The input is the generated response, and the output is audio data. The server uses a speech synthesis library (e.g., pyttsx3) to generate the audio data and send it to the device.

Step 9:

The device plays back the audio data received from the server and lets the user listen. The input is audio data, and the output is audio playback. The device plays back the audio data, allowing the user to hear the virtual character's response.

Step 10:

When the user wants to end the service, they perform an operation to end the application. The input is the end operation, and the output is to end the application. The device will end the application and stop displaying the smart glasses.

It is also possible to further combine an emotion engine that estimates the user's emotion. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

This invention is a system that uses artificial intelligence technology to learn the appearance and personality of specific characters or people, stores that learning data as non-fungible tokens (NFTs), and enables real-time voice conversations with users 24 hours a day, 365 days a year. Furthermore, by combining this invention with an emotion engine, it is possible to recognize the user's emotions and provide adaptive responses based on those emotions.

System Configuration

1. Learning with AI technology

The server collects data on the appearance, personality, and past behavior of specific characters or people, and uses an artificial intelligence algorithm to learn this information. The learning results, including the character's specific behavior and response patterns, are saved as a highly accurate model.

2. Save as an NFT

The server stores the learned AI data as an NFT, using blockchain technology to guarantee its uniqueness and ownership.

3. Providing an interface

The device provides the user with an interface for selecting a conversation partner. Users can log in and select their favorite character.

4. Real-time voice conversation

The user performs an operation to start a conversation with the selected character. The device captures the user's voice, converts it into text data, and sends it to the server.

Based on the text data received by the server, an appropriate response is generated using an AI model. This response is then converted back into audio data and sent to the device.

The device plays the audio data and the conversation takes place in real time.

5. Emotional Engine Integration

The server is equipped with an emotion engine that uses the user's voice data to analyze emotions. The emotion engine analyzes the user's emotional state (e.g., joy, sadness, anger, etc.) from the voice data.

Based on the results of emotion analysis, the AI model adaptively changes the response content. For example, if the user speaks in a sad voice, a response such as "Cheer up!" will be generated.

6. Usage Fee Collection and Revenue Sharing

The server collects usage fees from users. Payments are processed through the terminal. Revenues are distributed among the parties involved, including the copyright holders of characters and people, and service providers.

Specific examples

1. The user logs in and selects a character.

The user opens the app and logs in by entering "user123" and "password123" on the login screen. The device sends the entered data to the server and authentication is performed. If authentication is successful, the home screen is displayed on the device.

The user presses the "Select Character" button on the home screen and selects, for example, "Character A." The device sends the selected character information to the server, and the server retrieves the corresponding AI data and sends it to the device.

2. Initiating and conducting actual conversations

The user presses the "Start Conversation" button and speaks "Hello, Character A" into the device's microphone. The device converts the speech into text and sends it to the server.

The server analyzes the text data and generates an appropriate response based on an AI model. For example, it might respond with "Hello! How's your day?"

At the same time, the server sends the voice data to the emotion engine, which analyzes the user's emotional state.

The emotion engine analyzes the user's emotional state and feeds the results back to the AI model. For example, if the user sounds sad, the response will be changed to an encouraging message such as "Cheer up!"

The generated response is converted into audio data and sent to the device, which then plays the audio data and provides the response to the user.

3. Payment of fees and revenue sharing

If the user needs to pay a usage fee, the terminal will display a payment screen. The user will enter their credit card information, etc.

The terminal sends the entered payment information to the server, which processes the payment through the payment gateway.

The server sends information about the success or failure of the payment to the terminal. If successful, the usage rights are updated.

The terminal will notify the user of the payment result, allowing them to continue using the system.

This system allows fans to safely and easily interact with their favorite characters and people 24 hours a day, 365 days a year, and by providing appropriate responses based on the user's emotions, it enables more personal and intimate communication.

The processing flow is explained below.

Step 1:

The user opens the application. The user accesses the login screen and enters their ID and password.

Step 2:

The device receives the user's input and sends the ID and password information to the server. The communication is encrypted.

Step 3:

The server checks the received ID and password against a database and authenticates the user. If the ID and password do not match, an error message is sent to the terminal.

Step 4:

If the server successfully authenticates, it generates session information and sends it back to the device. The device uses the received session information to display the home screen.

Step 5:

The user presses the "Character Selection" button on the home screen and selects the character they want to interact with.

Step 6:

The device receives the user's selection and sends the character ID to the server. The server then retrieves the AI model data for the selected character.

Step 7:

The server sends the acquired AI model data to the device. The device displays a conversation interface with the character based on the received data.

Step 8:

The user presses the "Start conversation" button to begin the conversation. The device's microphone is turned on to capture the user's voice.

Step 9:

The device records the user's voice and converts the voice data into text data in real time. This text data is then sent to the server.

Step 10:

The server analyzes the received text data and generates a response using an AI model. For example, if the user says "Hello, Character A," the response will be "Hello! How's your day?"

Step 11:

At the same time, the server sends the user's voice data to the emotion engine, which analyzes the user's emotional state. The emotion engine analyzes the voice data and identifies the user's emotional state (joy, sadness, anger, etc.).

Step 12:

The emotion engine sends the analysis results back to the server, which then adaptively changes the response based on the results. For example, if the user speaks in a sad voice, the response will be "Cheer up!"

Step 13:

The server converts the response text into audio data and sends it to the device.

Step 14:

The device plays back the received audio data and provides a response to the user. The user can then listen to this and ask questions or make comments.

Step 15:

If the user needs to pay a usage fee, the terminal will display a payment screen. The user will then enter their credit card information, etc.

Step 16:

The terminal sends the entered payment information to the server, which then processes the payment through the payment gateway.

Step 17:

The server sends information about the success or failure of the payment to the terminal. If successful, the usage rights are updated.

Step 18:

The terminal will notify the user of the payment result, allowing them to continue using the system.

(Example 2)

Next, Example 2 will be described. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

With conventional voice conversation systems using artificial intelligence, it was difficult to efficiently learn the appearance and personality of characters or people and conduct real-time voice conversations with users. Furthermore, technology for providing adaptive responses based on the user's emotions was not sufficiently developed, making it impossible to achieve natural, personal communication with users.

The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

In this invention, the server includes means for learning the appearance and personality of a specific character or person using artificial intelligence technology, means for saving the learned data as a non-fungible token (NFT), and means for analyzing emotions from the user's voice and adaptively changing responses based on the analysis results. This makes it possible to learn the characteristics of characters and people with high accuracy, safely store that data, and provide flexible responses that correspond to the user's emotions.

"Specific characters or persons" refers to fictional or real people or characters that users can select as their conversation partners.

"Artificial intelligence technology" refers to technology that uses techniques such as machine learning and deep learning to extract patterns from data and make predictions and classifications.

"Means of learning" refers to the process of collecting data and generating an AI model based on that data.

A "non-fungible token (NFT)" is a token that uses blockchain technology to guarantee the uniqueness and ownership of digital data.

"Means of providing an interface" refers to the method of providing screens and menus that allow users to access and operate the system.

"Means for conducting real-time voice conversation" refers to technology that allows users and characters to have instant voice conversations.

"Means for analyzing emotions from a user's voice" refers to technology that analyzes a user's emotional state based on their voice data.

"Means for adaptively changing responses" refers to a method for changing the AI's response content based on the results of emotion analysis.

"Means for collecting usage fees and distributing revenue among parties" refers to a system for collecting service usage fees from users and distributing the revenue obtained among parties.

Form for implementing the invention

This invention is a system that uses artificial intelligence technology to learn the appearance and personality of specific characters or people, stores that learning data as non-fungible tokens (NFTs), and provides users with real-time voice conversations 24 hours a day, 365 days a year. Furthermore, by combining it with an emotion engine, it is possible to recognize the user's emotions and provide adaptive responses based on those emotions.

System Overview

The system consists of the following main components:

1. Server

2. Device

3. User

Server Role

1. Training data collection and AI model generation

The server collects data on the appearance, personality, and past behavior of specific characters and people. This data includes video, audio files, text data, etc.

The collected data is trained using an AI algorithm (e.g., TensorFlow or PyTorch). During the initial training stage, data preprocessing (noise removal, text normalization, etc.) is performed.

Save the completed model in the database.

2. Data storage as NFTs

The learned AI data is stored as a non-fungible token (NFT). This process uses blockchain technology (e.g., Ethereum).

The server registers the generated NFT on the blockchain and maintains owner information.

3. Sentiment Analysis and Response Adaptation

The server uses an emotion engine (for example, IBM Watson's emotion recognition API) to analyze the user's voice data. Based on the analysis results, the AI model adaptively changes the response content.

For example, if a user says, "I'm feeling a bit down today," emotion recognition can generate a response such as, "Cheer up!"

4. Usage fee collection and revenue sharing

The server collects the usage fee and manages the payment process. Payment is made using credit card information entered by the user on the terminal.

Revenues will be distributed among the parties involved (character copyright holders, service providers, etc.).

Device role

1. Providing an interface

The device provides the user with an interface to select a conversation partner and conduct a conversation. The user launches the app, enters their authentication information on the login screen, and logs in.

After logging in, users can press the "Select Character" button to choose their preferred character and begin a conversation with that character.

2. Audio data processing

The device captures the user's voice and converts it into text data using a speech recognition engine such as Google Speech-to-Text.

The converted text data is sent to the server, and after an appropriate response is generated, it is converted back into voice data and sent to the device.

The device plays the audio data, enabling real-time conversations.

User Roles

1. Log in and select a character

The user launches the app and enters authentication information such as "user123" and "password123" on the login screen.

Once authentication is successful, select a character from the home screen and begin a conversation with the character you select.

2. Actual conversation progression

The user presses the "Start Conversation" button and speaks into the device's microphone, saying "Hello, Character A."

The device converts this voice into text data and sends it to the server. The server uses an AI model to generate a response based on the text data and sends an adaptive reply.

For example, you can receive responses such as "Hello! How are you today?" or adaptive messages such as "Cheer up!"

Examples and prompts

"Hello, Character A. How was your day?"

"I've been feeling a bit down lately. Can you cheer me up?"

"What do you think about today's news?"

By using these prompts, users can enjoy more personal and natural conversations with the characters.

The flow of the identification process in Example 2 will be explained using Figure 13.

Program processing flow

Step 1:

Training data collection and AI model generation

The server collects data on the appearance, personality, and past behavior of specific characters or people. The data used for this collection includes text, audio files, images, and videos. Specifically, the data is retrieved from a database via web scraping or API and stored on the server.

Input: Data about the character's appearance, personality, and past words and actions.

Data processing: noise removal, text normalization, data cleaning.

Output: Clean dataset after preprocessing.

The server uses an AI algorithm (such as TensorFlow or PyTorch) to train a model using this data. The trained model then reflects the character's behavior and response patterns.

Input: Clean dataset after preprocessing.

Data computation: Applying machine learning algorithms and training models.

Output: Trained AI model.

Step 2:

Storing data as NFTs

The server stores the learned AI data as a non-fungible token (NFT).

Input: Trained AI model.

Data processing: Blockchain technology (e.g., Ethereum) is used to ensure the uniqueness and ownership of the model for NFT conversion.

Output: NFT registered on the blockchain.

The server registers the generated NFT on the blockchain and retains owner information. Specifically, the NFT is registered through the blockchain API.

Step 3:

User login and character selection

The user launches the app and enters their authentication information (e.g., "user123" and "password123") on the login screen.

The device sends the entered authentication information to the server and performs authentication.

Input: User authentication information.

Data calculation: Compare with the database and determine whether authentication is successful.

Output: Authentication success or failure status.

If authentication is successful, the home screen will be displayed on the device. The user presses the "Select Character" button to choose the character they want to interact with.

Input: Character information selected by the user.

Data processing: Search and obtain the AI model corresponding to the selected character.

Output: AI data for the selected character.

Step 4:

Real-time voice conversation processing

The user presses the "Start conversation" button and speaks into the device's microphone.

The device captures the user's voice and converts it into text data using Google Speech-to-Text.

Input: User's voice data.

Data processing: voice-to-text conversion.

Output: Text data.

The converted text data is sent to the server, which generates an appropriate response.

Input: User's text data.

Data calculations: Use AI models to generate appropriate responses.

Output: The generated text response.

The response is converted back into audio data and sent to the device.

Input: The generated text response.

Data processing: converting text to audio.

Output: Generated audio data.

The device plays the audio data and the conversation takes place in real time.

Step 5:

Sentiment analysis and response adaptation

The server analyzes the user's voice data using an emotion engine (e.g., IBM Watson's emotion recognition API).

Input: User's voice data.

Data calculation: Analyze the user's emotional state from voice data.

Output: User's emotional state data.

Based on the results of emotion analysis, the AI model adaptively changes the response content.

Input: User emotional state data and generated text response.

Data calculation: Change the response content to match the emotional state.

Output: The final adapted response text.

For example, if the user sounds sad, the response will be changed to something like "Cheer up!"

Step 6:

Collecting usage fees and processing payments

If the user needs to pay a usage fee, the terminal will display a payment screen. The user will enter their credit card information, etc.

Input: User's payment information.

The terminal sends payment information to the server, which processes the payment through a payment gateway (e.g., Stripe).

Data calculation: Executes payment processing based on payment information.

Output: Payment success or failure status.

The server notifies the terminal of the success or failure of the payment, and if successful, the usage rights are updated.

The terminal will notify the user of the payment result, allowing them to continue using the system.

(Application Example 2)

Next, we will explain Application Example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 will be referred to as the "terminal."

The objective of this invention is to provide a system that can reproduce the appearance and personality of a specific character or person in real time, while providing appropriate responses based on the user's emotions through dialogue with them. Furthermore, the aim is to realize a more personal and sympathetic shopping experience by having characters act as shopping assistants in virtual stores, explaining products and recommending items.

The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

In this invention, the server includes: means for learning the appearance and personality of a specific character or person using artificial intelligence technology; means for storing the learned data as a non-fungible token (NFT); means for providing an interface for users to select a conversation partner and converse with them; means for conducting real-time voice conversations; means for collecting usage fees from users and distributing revenue among the parties involved; means for analyzing the emotional state from the user's voice data using an emotion engine and generating an adaptive response based on the analysis results; means for converting the generated response into voice data and providing it to the user as voice; and means for the character to act as a shopping assistant in a virtual store, explaining products and recommending items. This enables real-time, emotion-based dialogue with a specific character and a personalized, sympathetic shopping assistant experience in a virtual store.

"Specific characters or people" are virtual beings whose appearances and personalities are reproduced using artificial intelligence technology.

"Artificial intelligence technology" is a technology that learns the appearance and personality of a specific character or person, and generates responses and actions based on that information.

A non-fungible token (NFT) is a means of using blockchain technology to uniquely identify a specific digital asset and prove ownership of it.

An "interface" is a device or software that provides input and display means for a user to select a conversation partner and initiate a conversation.

"Means for conducting real-time voice conversations" refers to technology that allows users to have instant voice conversations with specific characters or people.

An "emotion engine" is an algorithm and system that analyzes a user's emotional state from their voice data and generates a response based on the analysis results.

An "adaptive response" is a response that changes depending on the user's speech content and emotional state, and is generated individually and dynamically.

A "virtual store" is a virtual shopping environment that exists on the Internet and is a platform that allows users to conduct purchasing activities in a virtual space.

A "shopping assistant" is a virtual character or person who explains products and recommends items to users in a virtual store.

"Collection" means receiving usage fees from users.

"Means for distributing revenues among parties involved" refers to a mechanism for appropriately distributing revenues earned through system operation to parties involved, such as character copyright holders and service providers.

System Overview

This invention is a system that recreates specific characters or people in real time and engages in voice conversations with users. It primarily functions as a shopping assistant in virtual stores. This system consists of three elements: a server, a terminal, and a user.

Program structure and various technologies

Speech recognition engine

The server receives the user's voice input and converts the voice into text data using a speech recognition engine (Google Speech-to-Text API).

Sentiment analysis engine

The server performs emotion analysis on the acquired text data using IBM Watson Tone Analyzer, thereby determining the user's emotional state.

Dialogue management system

The server uses Rasa to generate responses based on the user's conversation content and sentiment analysis results.

Speech synthesis engine

The server converts the generated response into audio data using the Google Text-to-Speech API and sends it to the device.

Program processing

1. Voice input and text conversion

When a user speaks into their smartphone's microphone, the audio is picked up by the device and converted into text data using the Google Speech-to-Text API.

2. Emotion analysis

The transferred text data is subjected to emotion analysis using IBM Watson Tone Analyzer to determine the user's emotional state (e.g., joy, sadness, anger, etc.).

3. Response Generation

Rasa is used to generate appropriate responses based on analysis results and conversation content. These responses are adapted to the user's emotions.

4. Conversion to audio data and transmission

The generated response is converted into audio data using the Google Text-to-Speech API, and this audio data is sent to the device, enabling real-time conversation.

Specific examples

1. User login and character selection

The user launches the app and logs in by entering "user123" and "password123." After logging in, they select "Character A" on the character selection screen.

Example prompt:

User: Hello, I'm looking for clothes today.

---

Assistant: Hello, what kind of clothes are you looking for?

---

User: I'd like a casual shirt.

---

2. Voice input and analysis

When a user says, "Hello, I'm looking for clothes today," the speech is captured and converted into text data. The text data is sent to IBM Watson Tone Analyzer for sentiment analysis.

3. Response generation and delivery

Based on sentiment analysis and the conversation content, Rasa generates a response: "Hello, what kind of clothes are you looking for?" This response is converted into audio using the Google Text-to-Speech API and sent to the device.

4. Shopping Assistant Function

If a user says, "I want a casual shirt," the assistant will suggest, "How about this shirt? It's a trendy design and very popular these days," and explain the product. It is also possible to provide more detailed product descriptions and suggestions based on the results of user sentiment analysis.

This allows the invention to provide emotion-based real-time interactions with specific characters and a personalized, immersive shopping assistant experience in virtual stores.

The flow of the specific processing in Application Example 2 will be explained using Figure 14.

Step 1:

The user launches the app and logs in. The user enters their username and password, and the device sends that information to the server. The server performs authentication, and if successful, displays the character selection screen on the device.

Enter: User's username and password

Process: The server authenticates the user.

Output: Upon successful authentication, the character selection screen will be displayed.

Step 2:

The user selects a character. The user chooses a specific character on the character selection screen, and the device sends the selection information to the server. The server loads the selected character's AI model and sends the relevant data to the device.

Input: User's character selection information

Processing: The server loads the AI model and retrieves relevant data.

Output: Character information is displayed on the device.

Step 3:

The user begins the conversation. They speak into the smartphone's microphone, and the device picks up the audio. The speech is converted into text data using a speech recognition engine (Google Speech-to-Text API), and the text data is sent to the server.

Input: User's voice

Processing: The device captures the audio and converts it into text data.

Output: The converted text data is sent to the server.

Step 4:

The server performs emotion analysis. The server sends the acquired text data to an emotion analysis engine (IBM Watson Tone Analyzer), which analyzes the user's emotional state. The analysis results are added to the text data.

Input: Text data

Processing: The server analyzes the emotional state using an emotion analysis engine.

Output: Text data with emotional states added

Step 5:

The server generates a response. The server uses a dialogue management system (Rasa) to generate a response based on the sentiment analysis results and the content of the conversation. The response is in text format.

Input: Text data with sentiment analysis results added

Processing: The server generates a response using the dialogue management system.

Output: Generated response text

Step 6:

The server converts the response text into speech. The generated response text is converted into audio data using a speech synthesis engine (Google Text-to-Speech API). The converted audio data is then sent back to the device.

Input: Generated response text

Processing: The server uses a speech synthesis engine to convert the data into audio.

Output: Converted audio data

Step 7:

The device plays the audio data. The audio data sent to the device is played back through the speaker and becomes audible to the user. This allows for real-time audio conversation between the user and the character.

Input: Audio data

Processing: The device plays the audio data.

Output: The user hears the response.

The specific processing unit 290 transmits the results of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the results of the specific processing. The microphone 238 acquires audio indicating the user input regarding the results of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

Data generation model 58 is what is known as generative AI (artificial intelligence). Examples of data generation model 58 include generative AI such as ChatGPT (Internet search <URL: https://openai.com/blog/chatgpt>) and Gemini (Internet search <URL: https://gemini.google.com/?hl=ja>). Data generation model 58 is obtained by performing deep learning on a neural network. A prompt containing an instruction is input to data generation model 58, and inference data such as voice data indicating speech, text data indicating text, and image data indicating an image is also input. Data generation model 58 performs inference on the input inference data in accordance with the instructions indicated by the prompt, and outputs the inference results in the form of data such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and/or summarization.

In the above embodiment, an example was given in which the specific processing was performed by the data processing device 12, but the technology disclosed herein is not limited to this, and the specific processing may also be performed by the robot 414.

The emotion identification model 59, which serves as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see Figure 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

Figure 9 shows an emotion map 400 on which multiple emotions are mapped. In emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both emotions and mental states. Emotions that are generally generated from reactions that occur in the brain are arranged on the left side of the concentric circles. Emotions that are generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions that are generally generated from reactions that occur in the brain and are induced by situational judgment are arranged above and below the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the top side of the concentric circles, and the emotion of "discomfort" is arranged on the bottom side. In this way, emotion map 400 maps multiple emotions based on the structure by which emotions are generated, with emotions that tend to occur simultaneously being mapped close together.

These emotions are distributed in the 3 o'clock direction on emotion map 400, and usually fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness takes precedence over internal sensations, resulting in a calm impression.

The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further out you go on emotion map 400, the more visible (expressed in behavior) the emotion becomes.

Here, human emotions are based on various balances such as posture and blood sugar levels, and when these balances deviate from the ideal, it indicates discomfort, and when they approach the ideal, it indicates pleasure. Emotions can also be created for robots, cars, motorcycles, etc., based on various balances such as posture and remaining battery life, so that when these balances deviate from the ideal, it indicates discomfort, and when they approach the ideal, it indicates pleasure. Emotion maps may be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on speech emotion recognition and emotional brain physiological signal analysis systems, Tokushima University, doctoral dissertation: https://ci.nii.ac.jp/naid/500000375379). The left half of the emotion map is lined with emotions belonging to an area called "reaction," where sensation is dominant. The right half of the emotion map is lined with emotions belonging to an area called "situation," where situational awareness is dominant.

The emotion map defines two emotions that encourage learning. One is the negative emotion around the middle of "repentance" or "reflection" on the situation side. In other words, this is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the positive emotion around "desire" on the response side. In other words, this is when the robot experiences positive feelings such as "I want more" or "I want to know more."

The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are close to each other have similar values, as in the emotion map 900 shown in Figure 10. Figure 10 shows an example in which multiple emotions, such as "relieved," "calm," and "reassuring," have similar emotion values.

The system according to the present disclosure has been described above primarily in terms of the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may also be implemented as a general information processing system. The present disclosure may also be implemented, for example, as a software program that runs on a personal computer or an application that runs on a smartphone, etc. The method according to the present disclosure may also be provided to users in the form of SaaS (Software as a Service).

In the above embodiment, an example was given in which a specific process was performed by a single computer 22, but the technology of the present disclosure is not limited to this, and distributed processing of the specific process may be performed by multiple computers, including the computer 22. For example, the data generation model 58 may be provided in a device external to the data processing device 12, and data may be generated in that external device in response to input data.

In the above embodiment, an example was described in which the specific processing program 56 is stored in the storage 32, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

It is not necessary to store the entire specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entire specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

The following types of processors can be used as hardware resources for executing specific processes. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource for executing specific processes by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which are processors with a circuit configuration designed specifically for executing specific processes. All processors have built-in or connected memory, and all use the memory to execute specific processes.

The hardware resource that executes the specific processing may be composed of one of these various processors, or may be composed of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). The hardware resource that executes the specific processing may also be a single processor.

As an example of a configuration using a single processor, first, there is a configuration in which one processor is configured using a combination of one or more CPUs and software, and this processor functions as a hardware resource that executes specific processing. Second, there is a configuration in which a processor is used to realize the functions of an entire system, including multiple hardware resources that execute specific processing, on a single IC chip, as typified by SoC (System-on-a-chip). In this way, specific processing is realized using one or more of the various processors listed above as hardware resources.

More specifically, the hardware structure of these various processors can be an electrical circuit that combines circuit elements such as semiconductor devices. Furthermore, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps can be added, or the processing order can be rearranged, all within the scope of the spirit of the invention.

The above-described written content and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above explanation of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or substitutions may be made to the above-described written content and illustrations, as long as they do not deviate from the spirit of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described written content and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

All publications, patent applications, and technical standards mentioned in this specification are incorporated by reference herein to the same extent as if each individual publication, patent application, and technical standard was specifically and individually indicated to be incorporated by reference.

The following is further disclosed regarding the above embodiments.

(Claim 1)

A means of learning the appearance and personality of specific characters or people using artificial intelligence technology,

A means of storing learned data as non-fungible tokens (NFTs),

A means for providing an interface for users to select a conversation partner and converse with them;

A means of conducting real-time voice conversations,

A means of collecting usage fees from users and distributing the revenue among the parties involved,

A system including

(Claim 2)

The system of claim 1 further includes means for using the learned artificial intelligence data to generate a response based on the user's conversation content and converting the response into voice data.

(Claim 3)

The system of claim 1 further includes means for converting user-input speech into text data and supplying the text data to the artificial intelligence.

"Example 1"

(Claim 1)

A means of learning the appearance and personality of specific characters or people using artificial intelligence technology,

A means of storing learned data as non-fungible tokens (NFTs),

A means for providing an interface for users to select a conversation partner and converse with them;

A means of capturing the user's voice and converting it into text data using voice recognition technology;

A means to generate appropriate responses using an AI model based on text data and convert those responses into voice data;

A means for transmitting the converted voice data to the user's terminal and conducting a real-time voice conversation;

A means of collecting usage fees from users and distributing the revenue among the parties involved,

A system including

(Claim 2)

The system of claim 1 further includes means for using the learned artificial intelligence data to generate a response based on the user's conversation content and converting the response into voice data.

(Claim 3)

The system of claim 1 further includes means for converting user-input speech into text data and supplying the text data to the artificial intelligence.

"Application Example 1"

(Claim 1)

A means of learning the characteristics of a specific virtual person using machine learning technology,

A means of storing learned information as non-fungible symbols (NFTs),

A means for providing an operation screen that allows users to select a conversation partner and conduct a conversation;

A means of real-time voice communication,

Means for collecting service fees from users and distributing revenue among relevant parties;

A means for providing guidance using a display device that supports conversations between customers and virtual characters in a physical store;

A way for users to obtain product information in-store and receive support through characters,

A system including

(Claim 2)

The system of claim 1 further includes means for using the learned machine learning data to create a response based on a user's voice input and converting the response into voice data.

(Claim 3)

The system of claim 1 further includes means for converting a user's voice input into text information and feeding the text information into machine learning.

"Example 2: Combining Emotion Engines"

(Claim 1)

A means of learning the appearance and personality of specific characters or people using artificial intelligence technology,

A means of storing learned data as non-fungible tokens (NFTs),

A means for providing an interface for users to select a conversation partner and converse with them;

A means of conducting real-time voice conversations,

A means for analyzing emotions from the user's voice and adaptively changing responses based on the analysis results;

A means of collecting usage fees from users and distributing the revenue among the parties involved,

A system including

(Claim 2)

The system of claim 1 further includes means for using the learned artificial intelligence data to generate a response based on the user's conversation content and converting the response into voice data.

(Claim 3)

The system of claim 1 further includes means for converting speech input by a user into text data and supplying the text data to the artificial intelligence.

"Application Example 2: Combining Emotion Engines"

(Claim 1)

A means of learning the appearance and personality of specific characters or people using artificial intelligence technology,

A means of storing learned data as non-fungible tokens (NFTs),

A means for providing an interface for users to select a conversation partner and converse with them;

A means of conducting real-time voice conversations,

A means of collecting usage fees from users and distributing the revenue among the parties involved,

A means for analyzing the user's emotional state from their voice data using an emotion engine and generating an adaptive response based on the analysis results;

A means for converting the generated response into audio data and providing it to the user as audio;

The characters will act as shopping assistants in virtual stores, explaining products and recommending items,

A system including

(Claim 2)

The system of claim 1 further includes means for using the trained artificial intelligence data to generate a response based on the user's conversation content and sentiment analysis results, and converting the response into voice data.

(Claim 3)

The system described in claim 1 further includes means for converting the user's input voice into text data and supplying the text data and emotion analysis results to the artificial intelligence.

10, 210, 310, 410 Data processing system 12 Data processing device 14 Smart device 214 Smart glasses 314 Headset type terminal 414 Robot

Claims (3)

A means for learning the appearance and personality of a specific character or person using artificial intelligence technology;
A means of storing the learned data as a non-fungible token (NFT);
means for providing an interface for a user to select a conversation partner and converse with;
a means for conducting real-time voice conversations;
A means of collecting fees from users and distributing the revenue among the parties involved;
A system including: The system of claim 1 further includes means for using the learned artificial intelligence data to generate a response based on the user's conversation content and converting the response into voice data. The system of claim 1 further includes means for converting user-input speech into text data and supplying the text data to the artificial intelligence.