CN113850898B - Scene rendering method and device, storage medium and electronic equipment - Google Patents
Scene rendering method and device, storage medium and electronic equipment Download PDFInfo
- Publication number
- CN113850898B CN113850898B CN202111212378.4A CN202111212378A CN113850898B CN 113850898 B CN113850898 B CN 113850898B CN 202111212378 A CN202111212378 A CN 202111212378A CN 113850898 B CN113850898 B CN 113850898B
- Authority
- CN
- China
- Prior art keywords
- scene
- rendering
- semantic
- text
- picture sequence
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—Three-dimensional [3D] image rendering
- G06T15/005—General purpose rendering architectures
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/33—Querying
- G06F16/332—Query formulation
- G06F16/3329—Natural language query formulation
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/33—Querying
- G06F16/3331—Query processing
- G06F16/334—Query execution
- G06F16/3343—Query execution using phonetics
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/33—Querying
- G06F16/3331—Query processing
- G06F16/334—Query execution
- G06F16/3344—Query execution using natural language analysis
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/30—Semantic analysis
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- General Engineering & Computer Science (AREA)
- Artificial Intelligence (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Mathematical Physics (AREA)
- Health & Medical Sciences (AREA)
- Audiology, Speech & Language Pathology (AREA)
- General Health & Medical Sciences (AREA)
- Human Computer Interaction (AREA)
- Acoustics & Sound (AREA)
- Computer Graphics (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
Abstract
The invention provides a scene rendering method and device, a storage medium and electronic equipment, wherein the method comprises the following steps: receiving a rendering instruction sent by a user through a client, and acquiring a question text and a response text corresponding to the rendering instruction; carrying out semantic scene analysis on the question text and the response text, and determining a semantic scene corresponding to the rendering instruction; and acquiring a scene picture sequence corresponding to the semantic scene, generating a scene rendering data stream based on the scene picture sequence, and sending the scene rendering data stream to the client so that the client applies the scene rendering data stream rendering response data. By analyzing the problem text and the response text, determining the semantic scene, and generating a scene rendering data stream based on a scene picture sequence corresponding to the semantic scene, the client renders the response scene based on the scene rendering data stream.
Description
Technical Field
The present invention relates to the field of man-machine interaction technologies, and in particular, to a scene rendering method and apparatus, a storage medium, and an electronic device.
Background
With the development of computer technology, more and more scenes are available to digital people, and when the digital people are used for providing services for clients, immersive experience can be provided for the clients by rendering scenes related to the clients and the digital people in the interaction process, so that better application experience and services are provided for the clients.
The current 2D-form digital person is rendered by using a neural network, in order to render scenes for clients and achieve an immersive effect, the neural network needs to be trained for each scene to render the corresponding scene, the number and variety of the scenes are numerous, the training of the neural network takes a lot of time and cost, and a lot of cost is required to be input for rendering the scenes by using the current mode.
Disclosure of Invention
In view of this, the present invention provides a method and apparatus for rendering scenes, a storage medium, and an electronic device, in which the present invention does not need to train a corresponding neural network for each scene, and renders scenes using a scene rendering data stream generated based on a scene picture sequence, thereby reducing the cost and time input when rendering scenes, and reducing the cost required for rendering scenes.
In order to achieve the above object, the embodiment of the present invention provides the following technical solutions:
The first aspect of the invention discloses a scene rendering method, which comprises the following steps:
Receiving a rendering instruction sent by a user through a client, and acquiring a question text and a response text corresponding to the rendering instruction;
Carrying out semantic scene analysis on the question text and the response text, and determining a semantic scene corresponding to the rendering instruction;
Acquiring a scene picture sequence corresponding to the semantic scene, and generating a scene rendering data stream based on the scene picture sequence;
and sending the scene rendering data stream to the client so that the client applies the scene rendering data stream to render the response scene.
The method, optionally, the obtaining the question text and the answer text corresponding to the rendering instruction includes:
analyzing the rendering instruction, obtaining problem data in the rendering instruction, and determining a problem text based on the problem data;
and processing the question text to obtain response audio data and response text corresponding to the question text.
The method, optionally, the determining the question text based on the question data includes:
Determining a data format of the problem data;
and processing the question data according to the data format to obtain a question text.
The above method, optionally, performs semantic scene analysis on the question text and the response text, and determines a semantic scene corresponding to the rendering instruction, including:
extracting semantic scene keywords from the question text and the response text, and determining whether scene individuation requirements exist or not based on the semantic scene keywords;
If the scene individuation requirement is determined to exist, an individuation scene corresponding to the semantic scene keyword is determined, and the individuation scene is determined to be the semantic scene corresponding to the rendering instruction;
And if the scene individuation requirement does not exist, taking a preset default scene as a semantic scene corresponding to the rendering instruction.
The method, optionally, generating a scene rendering data stream based on the scene picture sequence includes:
determining a sequence of digital person rendered pictures based on the answer text;
and applying a preset silent video generation server to perform data synthesis processing on the scene picture sequence, the response audio data and the digital person rendering picture sequence to obtain a scene rendering data stream.
The method, optionally, the obtaining a scene picture sequence corresponding to the semantic scene includes:
when the personalized scene is determined to be a semantic scene corresponding to the rendering instruction, determining whether a personalized picture sequence corresponding to the personalized scene exists in a scene background picture storage server;
If the situation that a personalized picture sequence corresponding to the personalized scene exists in the scene background picture storage server is determined, taking a preset personalized picture sequence corresponding to the personalized scene as a scene picture sequence corresponding to the semantic scene;
if the fact that the preset picture sequence corresponding to the personalized scene does not exist in the scene background picture storage server is determined, the preset default picture sequence is used as a scene picture sequence corresponding to the semantic scene;
When a preset default scene is used as a semantic scene corresponding to the rendering instruction, the default picture sequence is used as a scene picture sequence corresponding to the semantic scene.
The method, optionally, sends the scene rendering data stream to the client, including:
and transmitting the scene rendering data stream to the client by using a preset streaming media server.
A second aspect of the present invention discloses a scene rendering device, comprising:
The system comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for receiving a rendering instruction sent by a user through a client and acquiring a question text and a response text corresponding to the rendering instruction;
The semantic scene analysis unit is used for carrying out semantic scene analysis on the question text and the response text and determining a semantic scene corresponding to the rendering instruction;
the generating unit is used for acquiring a scene picture sequence corresponding to the semantic scene and generating a scene rendering data stream based on the scene picture sequence;
And the rendering unit is used for sending the scene rendering data stream to the client so that the client can apply the scene rendering data stream to render the response scene.
The above device, optionally, the acquiring unit includes:
the analysis subunit is used for analyzing the rendering instruction, acquiring problem data in the rendering instruction and determining a problem text based on the problem data;
And the processing subunit is used for processing the question text to obtain response audio data and response text corresponding to the question text.
The above apparatus, optionally, the parsing subunit includes:
A determining module, configured to determine a data format of the problem data;
and the processing module is used for processing the question data according to the data format to obtain a question text.
The above device, optionally, the semantic scene analysis unit includes:
The extraction subunit is used for extracting semantic scene keywords from the question text and the response text and determining whether scene individuation requirements exist or not based on the semantic scene keywords;
The first determining subunit is used for determining personalized scenes corresponding to the semantic scene keywords if the scene personalized requirements are determined to exist, and determining the personalized scenes as semantic scenes corresponding to the rendering instructions;
And the second determining subunit is used for taking a preset default scene as a semantic scene corresponding to the rendering instruction if the scene individuation requirement does not exist.
The above apparatus, optionally, the generating unit includes:
a third determining subunit configured to determine a sequence of digital person rendered pictures based on the answer text;
And the synthesis subunit is used for applying a preset silent video generation server to perform data synthesis processing on the scene picture sequence, the response audio data and the digital person rendering picture sequence to obtain a scene rendering data stream.
The above apparatus, optionally, the generating unit includes:
a fourth determining subunit, configured to determine, when the personalized scene is determined to be a semantic scene corresponding to the rendering instruction, whether a personalized picture sequence corresponding to the personalized scene exists in a scene background picture storage server;
A fifth determining subunit, configured to, if it is determined that a personalized picture sequence corresponding to the personalized scene exists in the scene background picture storage server, use a preset personalized picture sequence corresponding to the personalized scene as a scene picture sequence corresponding to the semantic scene;
A sixth determining subunit, configured to, if it is determined that the preset picture sequence corresponding to the personalized scene does not exist in the scene background picture storage server, use the preset default picture sequence as the scene picture sequence corresponding to the semantic scene;
A seventh determining subunit, configured to, when a preset default scene is used as a semantic scene corresponding to the rendering instruction, use the default picture sequence as a scene picture sequence corresponding to the semantic scene.
The above apparatus, optionally, the rendering unit includes:
and the sending subunit is used for sending the scene rendering data stream to the client by applying a preset streaming media server.
A third aspect of the present invention discloses a storage medium comprising stored instructions, wherein the instructions, when executed, control a device in which the storage medium is located to perform a scene rendering method as described above.
A fourth aspect of the invention discloses an electronic device comprising a memory, and one or more instructions, wherein the one or more instructions are stored in the memory and configured to perform a scene rendering method as described above by one or more processors.
Compared with the prior art, the invention has the following advantages:
The invention provides a scene rendering method and device, a storage medium and electronic equipment, wherein the method comprises the following steps: acquiring a rendering instruction sent by a user through a client, and acquiring a question text and a response text corresponding to the rendering instruction; carrying out semantic scene analysis on the question text and the response text, and determining a semantic scene corresponding to the voice; the method comprises the steps of obtaining a scene picture sequence corresponding to a semantic scene, generating a scene rendering data stream based on the scene picture sequence, and sending the scene rendering data stream to a client, so that the client renders a response scene of a digital person when replying to a user based on the scene rendering data stream. According to the method, after the semantic scene is determined according to the problem text and the response text of the voice carrying the problem information, the rendering data stream for rendering the response scene is generated based on the scene picture sequence corresponding to the semantic scene, and the process does not need training of a neural network corresponding to the scene, so that a convenient scene rendering mode is provided, and the cost input in scene rendering is reduced.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required to be used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only embodiments of the present invention, and that other drawings can be obtained according to the provided drawings without inventive effort for a person skilled in the art.
Fig. 1 is an environmental application schematic diagram of a scene rendering method according to an embodiment of the present invention;
FIG. 2 is a method flowchart of a scene rendering method according to an embodiment of the present invention;
FIG. 3 is a flowchart of a method for rendering a scene according to an embodiment of the present invention;
FIG. 4 is a flowchart of another method of a scene rendering method according to an embodiment of the present invention;
Fig. 5 is a schematic diagram of a system architecture of a scene rendering method according to an embodiment of the present invention;
Fig. 6 is a schematic structural diagram of a scene rendering device according to an embodiment of the present invention;
fig. 7 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
In the present disclosure, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
In order to better understand the scene rendering method and device, the storage medium and the electronic device provided by the embodiment of the application, the following describes an application environment for the embodiment of the application.
Referring to fig. 1, fig. 1 shows a schematic view of an application environment suitable for an embodiment of the present application. The scene rendering method provided by the embodiment of the application can be applied to the scene rendering system 100 shown in fig. 1. The scene rendering system 100 includes a terminal device 101 and a server 102, and the server 102 is communicatively connected to the terminal device 101. The server 102 may be a conventional server or a cloud server, which is not specifically limited herein.
The terminal device 101 may be various electronic devices having a display screen, having a data processing module, having a photographing camera, having audio input/output, etc., and supporting data input, including but not limited to a smart phone, a tablet computer, a laptop portable computer, a desktop computer, a self-service terminal, a wearable electronic device, etc. Specifically, the data input may be inputting voice based on a voice module provided on the electronic device, inputting characters by a character input module, and the like.
The terminal device 101 may be provided with a client application program, and a user may communicate with the server 102 based on the client application program (such as APP, weChat applet, etc.), specifically, the user may register a user account with the server 102 based on the client application program, and communicate with the server 102 based on the user account, for example, the user logs in to the user account with the client application program based on the user account, inputs text information or voice information through the client application program based on the user account, and after receiving information input by the user, the client application program may send the information to the server 102, so that the server 102 may receive the information, process and store the information, and the server 102 may also receive the information and return a corresponding output information to the terminal device 101 according to the information.
In some embodiments, the client application may be configured to provide customer service to the user, communicate with the user through customer service, interact with the user based on the digital person, and render a corresponding scene for the user during the interaction of the digital person with the user. Specifically, the client application program may receive information input by a user, send the information input by the user to the scene rendering system for processing, receive data of a rendering scene fed back by the scene rendering system, and render a scene involved in the interaction process with the digital person for the client based on the data of the rendering scene. The digital person is a software program based on visual graphics, and the software program can present a robot form simulating biological behaviors or ideas to a user after being executed. The digital person may be a simulated digital person of a simulated human type, for example, a simulated digital person of a simulated human type established according to the form of the user or other persons, or a simulated digital person of a cartoon effect type, for example, a simulated digital person of an animal form or a cartoon character form.
In some embodiments, the terminal device 101 sends interactive information of the client to the server 102, and the server 102 processes the interactive information to generate rendering scene data of the digital person responding to the client; the terminal device 101 receives the rendering scene data sent by the server 102, and renders a scene of the digital person answering the client on a display screen or other image output devices connected with the rendering scene data, wherein the scene can be a page background of the digital person answering the client. As another implementation manner, when a scene of a digital person answering a client is rendered, audio corresponding to an emulated digital person image can be played through a speaker of the intelligent terminal 101 or other audio output devices connected with the speaker, characters or graphics corresponding to the replying information can be displayed on a display screen of the terminal device 101, and polymorphic interaction with the user in multiple aspects of images, voices, characters and the like can be realized.
In some embodiments, the device for processing the data to be identified may also be disposed on the terminal device 101, so that the terminal device 101 may implement interaction with the user without relying on the server 102 to establish communication, where the scene rendering system 100 may only include the terminal device 101.
The above application environments are merely examples for facilitating understanding, and it is to be understood that embodiments of the present application are not limited to the above application environments.
The method and apparatus for rendering a scene, the storage medium, and the electronic device provided by the embodiments of the present invention are described in detail below by means of specific embodiments.
Referring to fig. 2, an embodiment of the present invention provides a scene rendering method, which can be applied to the scene rendering system 100 described above. Specifically, the scene rendering system may include the following steps S201 to S204.
The invention can be applied to a scene rendering system supporting rendering of response scenes of digital persons, and referring to fig. 2, a method flowchart of a scene rendering method provided for an embodiment of the invention is specifically described as follows:
s201, receiving a rendering instruction sent by a user through a client, and acquiring a question text and a response text corresponding to the rendering instruction.
The user sends a rendering instruction to the scene rendering system through the client, and the rendering instruction contains data for the user to interact with the digital person, so that a question text and a response text are obtained after the rendering instruction is processed.
The user inputs a rendering instruction to the client, the client sends the rendering instruction to the scene rendering system, and the scene rendering system processes the rendering execution, so that a question text and a response text are obtained.
Further, there are various ways in which the user inputs the rendering instruction to the client, specifically, inputting by voice, inputting by text, or inputting after selecting an option on the operation interface of the client.
Referring to fig. 3, a specific description of acquiring a question text and a response text corresponding to a rendering instruction according to an embodiment of the present invention is as follows:
s301, analyzing the rendering instruction, obtaining problem data in the rendering instruction, and determining a problem text based on the problem data.
The rendering instruction is a carrier of problem data, the problem data can be obtained after the rendering instruction is analyzed, and the problem data can be understood as data of interaction between a user and a digital person, such as data of a question asked by the user to the digital person or expression packages sent by the user to the digital person; further, the data format of the problem data in the embodiment of the present invention supports multiple types, specifically, various data formats such as text, voice, and picture. Supporting multiple data formats can increase the application scenarios supported by the invention, so that the universality and applicability of the invention are stronger.
Specifically, when the problem data is in a text format, the problem data can be text information related to the input of the user on an operation page of the client, or can be specific text information generated directly after the user selects a specific option on the operation page of the client, for example, after the user selects a weather inquiry option on the operation page of the client, the text information corresponding to the weather inquiry option is generated directly.
For another example, when the data format of the question data is a voice format, the question data is converted into the corresponding question text by using a voice-to-text technology, and in particular, the voice-to-text technology may be a voice recognition program, a voice recognition algorithm, or a voice recognition model. The specific process of processing the question data to obtain the question text includes, but is not limited to, the above illustration, and when the data format of the question data is other formats, only the operation corresponding to the data format needs to be executed, and the corresponding question text is determined from the question data.
S302, processing the question text to obtain response audio data and response text corresponding to the question text.
Processing the problem text by using a preset response module, so that the response module outputs response audio data and response text corresponding to the problem text; specifically, the response audio uses a preset response robot to process the problem text, so that the response robot outputs the response text; after the response text output by the response robot is obtained, the response text is processed by using a text-to-speech technology to obtain response audio data corresponding to the response text, wherein the text-to-speech technology can be realized through a special text-to-speech algorithm or model.
The response robot is a model constructed by using a deep neural network, the response robot can obtain corresponding response data after processing the problem text, and the response module processes the response data into response audio data.
Furthermore, the response robot can be put into use after the training is successful, and the subsequent response robot can also learn autonomously, so that the output data is more accurate.
S202, carrying out semantic scene analysis on the question text and the response text, and determining a semantic scene corresponding to the rendering instruction.
Referring to fig. 4, a flowchart of a method for determining a semantic scene corresponding to a rendering instruction according to an embodiment of the present invention is specifically described below:
s401, extracting semantic scene keywords from the question text and the response text.
Extracting semantic scene keywords from the question text and the response text by using a preset keyword extraction algorithm; by way of example, the semantic scene keywords may be words such as weather, cloudiness, olympics, etc.
S402, determining whether scene individuation requirements exist or not based on semantic scene keywords; if it is determined that there is a scene personalization need, S403 is executed; if it is determined that there is no scene personalization need, S404 is performed.
It should be noted that, the scene personalization requirement is that a personalized scene needs to be rendered, that is, a default scene set in the application system is not used.
If the scene individuation requirement exists, the situation that the digital person needs to render an individuation scene when responding can be determined, wherein the individuation scene is related to the semantics in the response text and the question text; if the scene personalization requirement does not exist, it can be determined that the digital person does not need to render the personalized scene when responding, in other words, the semantics in the response text and the question text are characterized as not needing to render the personalized scene.
S403, determining personalized scenes corresponding to the semantic scene keywords, and determining the personalized scenes as semantic scenes corresponding to the rendering instructions.
When the scene personalization requirement is determined to exist, a personalization scene can be determined according to the semantic scene keywords, for example, when the semantic scene keywords are weather and sunny days, the personalization scene can be determined to be a scene with rendering weather and sunny days.
S404, taking a preset default scene as a semantic scene corresponding to the rendering instruction.
Preferably, the default scenario is a scenario used by default in the system.
In the method provided by the embodiment of the invention, when the semantic scene is determined, the semantic scene keyword can be extracted from the question text and the response text, and the semantic scene is determined based on the semantic scene keyword, so that the scene related to the question text and the response text can be accurately determined, and the appropriate scene can be displayed for the client when the digital person replies to the client, and the immersive service can be provided for the client.
S203, a scene picture sequence corresponding to the semantic scene is obtained, and a scene rendering data stream is generated based on the scene picture sequence.
The specific process of acquiring a scene picture sequence corresponding to a semantic scene is as follows:
when the personalized scene is determined to be a semantic scene corresponding to the rendering instruction, determining whether a personalized picture sequence corresponding to the personalized scene exists in a scene background picture storage server;
If the situation that a personalized picture sequence corresponding to the personalized scene exists in the scene background picture storage server is determined, taking a preset personalized picture sequence corresponding to the personalized scene as a scene picture sequence corresponding to the semantic scene;
if the fact that the preset picture sequence corresponding to the personalized scene does not exist in the scene background picture storage server is determined, the preset default picture sequence is used as a scene picture sequence corresponding to the semantic scene;
When a preset default scene is used as a semantic scene corresponding to the rendering instruction, the default picture sequence is used as a scene picture sequence corresponding to the semantic scene.
It should be noted that, the scene background picture storage server stores personalized picture sequences of a plurality of personalized scenes and default picture sequences of default scenes, and further, the personalized picture sequences include at least one background picture; preferably, the default picture sequence may be a null sequence or a sequence containing a plurality of background pictures.
Determining a sequence of digital person rendered pictures based on the response text when generating the scene rendering data stream; and applying a preset silent video generation server to perform data synthesis processing on the scene picture sequence, the response audio data and the digital person rendering picture sequence to obtain a scene rendering data stream. Specifically, the digital person rendering picture sequence is a picture for rendering the digital person, and the picture sequence includes but is not limited to a picture for rendering the state, action and mouth shape of the digital person.
S204, sending the scene rendering data stream to the client so that the client applies the scene rendering data stream to render the response scene.
In the method provided by the embodiment of the invention, the application streaming media server sends the scene rendering data stream to the client so that the client application scene rendering data stream renders the response scene, wherein the response scene rendered by the client is a scene when a digital person interacts with a user, specifically, the scene is a page background displayed to the client when the digital person interacts with the user, and preferably, the response scene can be a dynamic scene or a static scene. Optionally, the client renders the digital person while rendering the response scene, and plays the response voice at the same time, so that the immersive scene can be provided for the user when the digital person interacts with the user, and the use experience of the user is improved.
In the method provided by the embodiment of the invention, a rendering instruction sent by a user through a client is received, and a question text and a response text corresponding to the rendering instruction are obtained; carrying out semantic scene analysis on the question text and the response text, and determining a semantic scene corresponding to the rendering instruction; and acquiring a scene picture sequence corresponding to the semantic scene, generating a scene rendering data stream based on the scene picture sequence, and sending the scene rendering data stream to the client so that the client applies the scene rendering data stream rendering response data. By analyzing the question text and the response text, determining the semantic scene, and generating a scene rendering data stream based on a scene picture sequence corresponding to the semantic scene, the client renders the response scene based on the scene rendering data stream, and a simple scene rendering mode is provided.
Referring to fig. 5, a schematic diagram of a system architecture supporting the above method according to an embodiment of the present invention is specifically described below:
The schematic diagram of the system architecture comprises a client 501, a gateway 502, a central control module 503, a response module 504, a picture reasoning server 505, a scene semantic analysis server 506, a scene background picture storage server 507, a silent video generation server 508 and a streaming media server 509; the workflow of the system architecture diagram is illustrated in a specific example scenario, as well as the operation of the various devices in the system architecture diagram.
The client sends a rendering instruction to the gateway, and the gateway sends the rendering instruction to the central control module; the response module receives the rendering instruction sent by the central control module, analyzes the rendering instruction, acquires problem data in the rendering instruction, determines a problem text corresponding to the problem data, processes the problem text to obtain response text and response audio data, and sends the problem text, the response text and the response audio data to the central control module; the central control module sends the response text to the picture reasoning server, so that the picture reasoning server processes the response text to obtain a digital person rendering picture sequence, and digital person rendering data is fed back to the central control module; the scene semantic analysis server receives the problem text and the response text sent by the central control module, determines a semantic scene based on the problem text and the response text, and sends the determined semantic scene information to the scene background picture storage server, so that the scene background picture server determines a scene picture sequence based on the determined semantic scene information, sends the scene picture sequence to the scene semantic analysis server, and then sends the scene picture sequence to the central control module; the central control module sends the response audio data, the digital person rendering picture sequence and the scene picture sequence to the silent video generation server; the silent video generation server synthesizes the response audio data, the digital person rendering picture sequence and the scene picture sequence into a scene rendering data stream; the silent video generation server sends the scene rendering data stream to the streaming media server, and the streaming media server sends the scene rendering data stream to the client; the client renders the response scene using the scene rendering data stream.
For other matters in the workflow described by the system architecture diagram shown in fig. 5, reference may be made to the corresponding matters described in the foregoing embodiments of the present invention, which are not described herein.
The system architecture diagram shown in fig. 5 is only one example, and the system architecture diagram to which the present invention is applicable is not limited to the above-described system architecture diagram.
Corresponding to fig. 1, the present invention also provides a scene rendering device, which can support the implementation of the method shown in fig. 1 in real life, and the device can be disposed in a scene rendering system, and the scene rendering system can be composed of an intelligent terminal or a distributed computing environment.
Referring to fig. 6, a schematic structural diagram of a scene rendering device according to an embodiment of the present invention is described below:
an obtaining unit 601, configured to receive a rendering instruction sent by a user through a client, and obtain a question text and a response text corresponding to the rendering instruction;
The semantic scene analysis unit 602 is configured to perform semantic scene analysis on the question text and the response text, and determine a semantic scene corresponding to the rendering instruction;
A generating unit 603, configured to obtain a scene picture sequence corresponding to the semantic scene, and generate a scene rendering data stream based on the scene picture sequence;
And a rendering unit 604, configured to send the scene rendering data stream to the client, so that the client applies the scene rendering data stream to render a response scene.
In the device provided by the embodiment of the invention, a rendering instruction sent by a user through a client is received, and a question text and a response text corresponding to the rendering instruction are obtained; carrying out semantic scene analysis on the question text and the response text, and determining a semantic scene corresponding to the rendering instruction; and acquiring a scene picture sequence corresponding to the semantic scene, generating a scene rendering data stream based on the scene picture sequence, and sending the scene rendering data stream to the client so that the client applies the scene rendering data stream rendering response data. By analyzing the question text and the response text, determining the semantic scene, and generating a scene rendering data stream based on a scene picture sequence corresponding to the semantic scene, the client renders the response scene based on the scene rendering data stream, and a simple scene rendering mode is provided.
In the apparatus provided by the embodiment of the present invention, the obtaining unit 601 may be configured to:
the analysis subunit is used for analyzing the rendering instruction, acquiring problem data in the rendering instruction and determining a problem text based on the problem data;
And the processing subunit is used for processing the question text to obtain response audio data and response text corresponding to the question text.
In the apparatus provided by the embodiment of the present invention, the parsing subunit may be configured to:
A determining module, configured to determine a data format of the problem data;
and the processing module is used for processing the question data according to the data format to obtain a question text.
In the apparatus provided by the embodiment of the present invention, the semantic scene analysis unit 602 may be configured to:
The extraction subunit is used for extracting semantic scene keywords from the question text and the response text and determining whether scene individuation requirements exist or not based on the semantic scene keywords;
The first determining subunit is used for determining personalized scenes corresponding to the semantic scene keywords if the scene personalized requirements are determined to exist, and determining the personalized scenes as semantic scenes corresponding to the rendering instructions;
And the second determining subunit is used for taking a preset default scene as a semantic scene corresponding to the rendering instruction if the scene individuation requirement does not exist.
In the apparatus provided by the embodiment of the present invention, the generating unit 603 may be configured to:
a third determining subunit configured to determine a sequence of digital person rendered pictures based on the answer text;
And the synthesis subunit is used for applying a preset silent video generation server to perform data synthesis processing on the scene picture sequence, the response audio data and the digital person rendering picture sequence to obtain a scene rendering data stream.
In the apparatus provided by the embodiment of the present invention, the generating unit 603 may be configured to:
a fourth determining subunit, configured to determine, when the personalized scene is determined to be a semantic scene corresponding to the rendering instruction, whether a personalized picture sequence corresponding to the personalized scene exists in a scene background picture storage server;
A fifth determining subunit, configured to, if it is determined that a personalized picture sequence corresponding to the personalized scene exists in the scene background picture storage server, use a preset personalized picture sequence corresponding to the personalized scene as a scene picture sequence corresponding to the semantic scene;
A sixth determining subunit, configured to, if it is determined that the preset picture sequence corresponding to the personalized scene does not exist in the scene background picture storage server, use the preset default picture sequence as the scene picture sequence corresponding to the semantic scene;
A seventh determining subunit, configured to, when a preset default scene is used as a semantic scene corresponding to the rendering instruction, use the default picture sequence as a scene picture sequence corresponding to the semantic scene.
In the apparatus provided by the embodiment of the present invention, the rendering unit 604 may be configured to:
and the sending subunit is used for sending the scene rendering data stream to the client by applying a preset streaming media server.
The embodiment of the invention also provides a storage medium, which comprises stored instructions, wherein the equipment where the storage medium is located is controlled to execute the scene rendering method when the instructions run.
The embodiment of the invention also provides an electronic device, whose structural schematic diagram is shown in fig. 7, specifically including a memory 701 and one or more instructions 702, where the one or more instructions 702 are stored in the memory 701, and configured to be executed by the one or more processors 703 to perform the above-mentioned scene rendering method by executing the one or more instructions 702.
The specific implementation process and derivative manner of the above embodiments are all within the protection scope of the present invention.
In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for a system or system embodiment, since it is substantially similar to a method embodiment, the description is relatively simple, with reference to the description of the method embodiment being made in part. The systems and system embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative elements and steps are described above generally in terms of functionality in order to clearly illustrate the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (8)
1. A method of scene rendering, comprising:
Receiving a rendering instruction sent by a user through a client, analyzing the rendering instruction, acquiring problem data in the rendering instruction, and determining a problem text based on the problem data; processing the question text to obtain response audio data and response text corresponding to the question text;
Carrying out semantic scene analysis on the question text and the response text, and determining a semantic scene corresponding to the rendering instruction;
acquiring a scene picture sequence corresponding to the semantic scene, and determining a digital person rendering picture sequence based on the response text; applying a preset silent video generation server to perform data synthesis processing on the scene picture sequence, the response audio data and the digital person rendering picture sequence to obtain a scene rendering data stream; the scene picture sequence is a page background picture displayed to a user by a digital person when the digital person interacts with the user; the digital person rendering picture sequence is used for rendering pictures of the digital person, and at least comprises pictures of the state, the action and the mouth shape of the digital person;
and sending the scene rendering data stream to the client so that the client applies the scene rendering data stream to render the response scene.
2. The method of claim 1, wherein the determining question text based on the question data comprises:
Determining a data format of the problem data;
and processing the question data according to the data format to obtain a question text.
3. The method of claim 1, wherein the performing semantic scene analysis on the question text and the answer text to determine a semantic scene corresponding to the rendering instruction comprises:
extracting semantic scene keywords from the question text and the response text, and determining whether scene individuation requirements exist or not based on the semantic scene keywords;
If the scene individuation requirement is determined to exist, an individuation scene corresponding to the semantic scene keyword is determined, and the individuation scene is determined to be the semantic scene corresponding to the rendering instruction;
And if the scene individuation requirement does not exist, taking a preset default scene as a semantic scene corresponding to the rendering instruction.
4. A method according to claim 3, wherein said obtaining a sequence of scene pictures corresponding to said semantic scene comprises:
when the personalized scene is determined to be a semantic scene corresponding to the rendering instruction, determining whether a personalized picture sequence corresponding to the personalized scene exists in a scene background picture storage server;
If the situation that a personalized picture sequence corresponding to the personalized scene exists in the scene background picture storage server is determined, taking a preset personalized picture sequence corresponding to the personalized scene as a scene picture sequence corresponding to the semantic scene;
if the fact that the preset picture sequence corresponding to the personalized scene does not exist in the scene background picture storage server is determined, the preset default picture sequence is used as a scene picture sequence corresponding to the semantic scene;
When a preset default scene is used as a semantic scene corresponding to the rendering instruction, the default picture sequence is used as a scene picture sequence corresponding to the semantic scene.
5. The method of claim 1, wherein the sending the scene rendering data stream to the client comprises:
and transmitting the scene rendering data stream to the client by using a preset streaming media server.
6. A scene rendering device, comprising:
An acquisition unit including: a parsing subunit and a processing subunit;
The analysis subunit is used for receiving a rendering instruction sent by a user through a client, analyzing the rendering instruction, acquiring problem data in the rendering instruction, and determining a problem text based on the problem data;
the processing subunit is used for processing the question text to obtain response audio data and response text corresponding to the question text;
The semantic scene analysis unit is used for carrying out semantic scene analysis on the question text and the response text and determining a semantic scene corresponding to the rendering instruction;
a generating unit comprising: a third determining subunit and a synthesizing subunit;
The third determining subunit is configured to obtain a scene picture sequence corresponding to the semantic scene, and determine a digital person rendering picture sequence based on the response text;
The synthesis subunit is used for applying a preset silent video generation server to perform data synthesis processing on the scene picture sequence, the response audio data and the digital person rendering picture sequence to obtain a scene rendering data stream;
And the rendering unit is used for sending the scene rendering data stream to the client so that the client can apply the scene rendering data stream to render the response scene.
7. A storage medium comprising stored instructions, wherein the instructions, when executed, control a device in which the storage medium is located to perform a scene rendering method according to any one of claims 1 to 5.
8. An electronic device comprising a memory and one or more instructions, wherein the one or more instructions are stored in the memory and configured to be executed by the one or more processors to perform the scene rendering method of any of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111212378.4A CN113850898B (en) | 2021-10-18 | 2021-10-18 | Scene rendering method and device, storage medium and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111212378.4A CN113850898B (en) | 2021-10-18 | 2021-10-18 | Scene rendering method and device, storage medium and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113850898A CN113850898A (en) | 2021-12-28 |
| CN113850898B true CN113850898B (en) | 2024-08-06 |
Family
ID=78978761
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111212378.4A Active CN113850898B (en) | 2021-10-18 | 2021-10-18 | Scene rendering method and device, storage medium and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113850898B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114449310A (en) * | 2022-02-15 | 2022-05-06 | 平安科技（深圳）有限公司 | Video editing method and device, computer equipment and storage medium |
| CN114860358B (en) * | 2022-03-31 | 2024-06-21 | 北京达佳互联信息技术有限公司 | Object processing method and device, electronic equipment and storage medium |
| CN117078835B (en) * | 2023-07-18 | 2024-11-19 | 广州数字集合科技有限公司 | Vehicle preview generation method, device, equipment and medium thereof |
| CN120751201A (en) * | 2025-09-03 | 2025-10-03 | 中移(苏州)软件技术有限公司 | Digital human video generation method, device, equipment, medium and product |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106095403A (en) * | 2016-05-30 | 2016-11-09 | 努比亚技术有限公司 | The exhibiting device of chat message and method |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105631925B (en) * | 2015-12-29 | 2019-02-26 | 北京航天测控技术有限公司 | 3D scene generation method and device based on OSG 3D rendering engine preprocessing |
| US10574597B2 (en) * | 2017-09-18 | 2020-02-25 | Microsoft Technology Licensing, Llc | Conversational log replay with voice and debugging information |
| CN112529992B (en) * | 2019-08-30 | 2022-08-19 | 阿里巴巴集团控股有限公司 | Dialogue processing method, device, equipment and storage medium of virtual image |
-
2021
- 2021-10-18 CN CN202111212378.4A patent/CN113850898B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106095403A (en) * | 2016-05-30 | 2016-11-09 | 努比亚技术有限公司 | The exhibiting device of chat message and method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113850898A (en) | 2021-12-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113850898B (en) | Scene rendering method and device, storage medium and electronic equipment | |
| US11151765B2 (en) | Method and apparatus for generating information | |
| US11158102B2 (en) | Method and apparatus for processing information | |
| US11670015B2 (en) | Method and apparatus for generating video | |
| CN113886551A (en) | Digital human rendering method and device, storage medium and electronic equipment | |
| CN116431316A (en) | Task processing method, system, platform and automatic question-answering method | |
| KR20210098623A (en) | Method and system for mimicking tone and style of real person | |
| CN110992958B (en) | Content recording method, content recording apparatus, electronic device, and storage medium | |
| US9110888B2 (en) | Service server apparatus, service providing method, and service providing program for providing a service other than a telephone call during the telephone call on a telephone | |
| US12367892B2 (en) | Method for providing speech video and computing device for executing the method | |
| CN107733722A (en) | Method and apparatus for configuring voice service | |
| CN113918856A (en) | Digital human application system, application method, computer device, and storage medium | |
| CN117632109A (en) | Virtual digital assistant construction method, device, electronic equipment and storage medium | |
| CN113157241A (en) | Interaction equipment, interaction device and interaction system | |
| CN107783650A (en) | A kind of man-machine interaction method and device based on virtual robot | |
| US20200365147A1 (en) | Configuration for simulating an interactive voice response system for language interpretation | |
| US9277051B2 (en) | Service server apparatus, service providing method, and service providing program | |
| CN110377842A (en) | Voice remark display methods, system, medium and electronic equipment | |
| CN118945389B (en) | Live broadcast task processing system, method, device, storage medium and program product | |
| US20240386887A1 (en) | Method and system for dynamic ivr prompt generation via prior contextual language analysis | |
| US11830120B2 (en) | Speech image providing method and computing device for performing the same | |
| CN113129925A (en) | Mouth action driving model training method and assembly based on VC model | |
| JP7829266B1 (en) | Information processing systems, information processing methods, and programs | |
| CN119693513A (en) | Interaction method, device, electronic device and storage medium | |
| CN120909693A (en) | Data processing method and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |