KR102093030B1 - Smart projector and method for controlling thereof - Google Patents

Abstract

The present invention relates to a method for controlling media content using a smart device. In the media content control method according to an aspect of the present invention, a sound signal received from the outside includes a wake-up word. A media content control method performed by a smart device having an operating state including a wake-up word detection state and a listening state for recognizing a voice command included in the sound signal, When entering the listening state while playing media content, adjusting the volume of the media content and text data corresponding to the volume adjusted audio data of the media content together with video data of the media content And displaying.

Description

Smart device and its control method {SMART PROJECTOR AND METHOD FOR CONTROLLING THEREOF}

The present invention relates to a smart device and a control method thereof, and more particularly, to a smart device and a control method for controlling the output of feedback in a listening mode.

As the recognition accuracy of the speech recognition technology has improved, the 'artificial intelligence voice assistant (voice assistant) function' is gradually being mounted on various smart devices. In this trend, smart speakers equipped with the voice assistant function are becoming a key device in the smart home field in connection with the spread of the Internet of Things (IoT).

In the smart speaker field, since Amazon released the first smart speaker Echo in 2014, not only IT dinosaurs such as Google, Apple, and Facebook, but also domestic companies such as Daum Kakao and Naver have outsmarted the smart speaker with its own software. It is showing fierce competition with the launch.

Since the smart speaker basically receives commands from the user via voice or performs a conversation with the user, the smart speaker has an advantage that the user can freely use in an indoor space such as a home or office, but the audio-type information has User convenience may be degraded in some situations due to limitations in information amount, difficulty in processing visual information, and non-persistence of information output.

One object of the present invention relates to a smart device and a control method for receiving a user voice without noise during media content playback.

One object of the present invention relates to a smart device and a control method for performing a listening mode without disturbing the media content being played.

The problem to be solved by the present invention is not limited to the above-described problems, and the problems not mentioned can be clearly understood by those having ordinary knowledge in the technical field to which the present invention pertains from this specification and the accompanying drawings. will be.

According to an aspect of the present invention, a wake-up word detection state and a voice command included in the sound signal are determined to determine whether the received sound signal includes a wake-up word. A media content control method performed by a smart device having an operating state including a listening state for recognizing, playing media content related to a first voice command input by a user, wherein the voice of the smart device Playing the media content by outputting audio data of the media content through an output module and displaying video data of the media content through an image output module of the smart device; Entering the wakeup word detection state while maintaining the playback of the media content after starting the playback of the media content; Determining whether the wakeup word is included in a first sound signal received during playback of the media content by the smart device in the wakeup word detection state; If the wakeup word is included in the first sound signal, entering the listening state while maintaining playback of the media content; And when entering the listening state, performing a first operation related to the media content, wherein performing the first operation comprises: reducing or removing the volume of the audio data to reduce the audio content. A control method of a smart device may be provided, comprising adjusting the volume of data and displaying text data corresponding to the volume-adjusted audio data of the media content together with video data of the media content.

According to another aspect of the present invention, a wake-up word detection state to determine whether a sound signal received from the outside includes a wake-up word and a voice included in the sound signal A media content control method performed by a smart device having an operating state including a listening state for recognizing a command, the media content related to a first voice command input by a user is reproduced, Playing the media content by outputting audio data of the media content through the audio output module and displaying video data of the media content through the video output module of the smart device; Entering the wakeup word detection state while maintaining the playback of the media content after starting the playback of the media content; Determining whether the wakeup word is included in a first sound signal received during playback of the media content by the smart device in the wakeup word detection state; If the wakeup word is included in the first sound signal, entering the listening state while maintaining playback of the media content; And when entering the listening state, performing any one of a pause of playback of the media content or a volume control of audio data of the media content according to the type of the media content; A control method can be provided.

According to another aspect of the present invention, a wake-up word detection state for determining whether a sound signal received from the outside contains a wake-up word and a wake-up word detection state included in the sound signal 1. A smart device having an operating state including a listening state for recognizing a voice command, comprising: a voice input module receiving a sound signal; A voice output module for outputting voice; An image output module for displaying an image; And playing media content related to the first voice command input by the user, outputting audio data of the media content through the audio output module of the smart device, and video of the media content through the video output module of the smart device. Playing the media content by displaying data, and starting to play the media content, enters the wake-up word detection state while maintaining playback of the media content, and the smart device in the wake-up word detection state is the During playback of the media content, it is determined whether the wakeup word is included in the first sound signal received through the voice input module, and when the wakeup word is included in the first sound signal, playback of the media content is performed. Retain while holding A controller that performs a first operation related to the media content when entering a ning state and entering the listening state, wherein the controller includes the audio data by reducing or removing the volume of the audio data. A smart device performing the first operation may be provided by adjusting the volume of and displaying text data corresponding to the volume-adjusted audio data of the media content together with video data of the media content.

The solving means of the subject matter of the present invention is not limited to the above-mentioned solving means, and the solving means not mentioned will be clearly understood by those skilled in the art from the present specification and the accompanying drawings. Will be able to.

According to the present invention, when a smart device enters a listening mode while playing a media content, the media content is stopped or the voice volume of the media content is eliminated or reduced to prevent the voice of the media content from being mixed into the user's voice as noise. have.

According to the present invention, when the smart device enters the listening mode during the playback of the media content, the user can enjoy the media content even in the listening mode by displaying the subtitles instead of removing or reducing the voice volume of the playing media content.

The effects of the present invention are not limited to the above-described effects, and effects not mentioned will be clearly understood by those skilled in the art from the present specification and the accompanying drawings.

1 is a block diagram of a smart device according to an embodiment of the present invention.
2 to 6 are diagrams for some implementations of a smart device according to an embodiment of the present invention.
7 and 8 are diagrams for an example of operation of an operation mode of a smart device according to an embodiment of the present invention.
9 is a diagram for communication between a smart device and a voice assistant server according to an embodiment of the present invention.
10 is a diagram for operation of a standby mode during media content playback according to an embodiment of the present invention.
11 is a diagram for operation of a listening mode during media content playback according to an embodiment of the present invention.
12 is a flowchart of a first example of a method for controlling a smart device according to an embodiment of the present invention.
13 is a flowchart of a second example of a method for controlling a smart device according to an embodiment of the present invention.
14 and 15 are diagrams for displaying text data according to an embodiment of the present invention.
16 is a flowchart of a third example of a method for controlling a smart device according to an embodiment of the present invention.
17 is a flowchart of a fourth example of a method for controlling a smart device according to an embodiment of the present invention.
18 is a flowchart of a fifth example of a method for controlling a smart device according to an embodiment of the present invention.

Since the embodiments described in this specification are intended to clearly explain the spirit of the present invention to those skilled in the art to which the present invention pertains, the present invention is not limited by the embodiments described in the present specification. The scope of should be construed as including modifications or variations that do not depart from the spirit of the present invention.

The terminology used in the present specification selects a general terminology that is currently widely used in consideration of the functions in the present invention, but this varies depending on the intention, customs, or the emergence of new technologies, etc. of those skilled in the art to which the present invention pertains. You can. However, if a specific term is defined and used in an arbitrary way, the meaning of the term will be described separately. Therefore, the terms used in this specification should be interpreted based on the actual meaning of the terms and the contents of the entire specification, not simply the names of the terms.

The drawings attached to the present specification are intended to easily describe the present invention, and the shapes illustrated in the drawings may be exaggerated and displayed as necessary to aid understanding of the present invention, and the present invention is not limited by the drawings.

In the present specification, when it is determined that detailed descriptions of known configurations or functions related to the present invention may obscure the subject matter of the present invention, detailed descriptions thereof will be omitted as necessary.

According to an aspect of the present invention, a wake-up word detection state and a voice command included in the sound signal are determined to determine whether the received sound signal includes a wake-up word. A media content control method performed by a smart device having an operating state including a listening state for recognizing, playing media content related to a first voice command input by a user, wherein the voice of the smart device Playing the media content by outputting audio data of the media content through an output module and displaying video data of the media content through an image output module of the smart device; Entering the wakeup word detection state while maintaining the playback of the media content after starting the playback of the media content; Determining whether the wakeup word is included in a first sound signal received during playback of the media content by the smart device in the wakeup word detection state; If the wakeup word is included in the first sound signal, entering the listening state while maintaining playback of the media content; And when entering the listening state, performing a first operation related to the media content, wherein performing the first operation comprises: reducing or removing the volume of the audio data to reduce the audio content. A control method of a smart device may be provided, comprising adjusting the volume of data and displaying text data corresponding to the volume-adjusted audio data of the media content together with video data of the media content.

In addition, when entering the listening state and then returning to the wake-up word detection state, performing a second operation related to the media content; and further comprising performing the second operation, the media The method may include adjusting the volume of the audio data of the content before performing the first operation and ending the display of the text data.

In addition, after entering the listening state, outputting feedback related to a second voice command included in the received second sound signal; And after outputting the feedback, performing a second operation related to the media content. The performing of the second operation may further include determining the volume of the audio data of the media content in the first operation. It may include the step of adjusting before performing and the step of ending the display of the text data.

Also, after entering the listening state, outputting a talk-back related to a second voice command included in the received second sound signal; And after the output of the talk-back ends, performing a second operation related to the media content. The performing of the second operation may further include determining the volume of the audio data of the media content. It may include adjusting before performing the first operation and ending displaying the text data.

In addition, the text data displayed first among the text data may include text data corresponding to audio data at a time when the volume is adjusted and text data corresponding to audio data before a predetermined time from a time when the volume is adjusted. have.

In addition, the text data that is displayed last among the text data may include text data corresponding to audio data at the time when the volume returns, and text data corresponding to audio data after a predetermined time from the time when the volume returns. You can.

According to another aspect of the present invention, a wake-up word detection state to determine whether a sound signal received from the outside includes a wake-up word and a voice included in the sound signal A media content control method performed by a smart device having an operating state including a listening state for recognizing a command, the media content related to a first voice command input by a user is reproduced, Playing the media content by outputting audio data of the media content through the audio output module and displaying video data of the media content through the video output module of the smart device; Entering the wakeup word detection state while maintaining the playback of the media content after starting the playback of the media content; Determining whether the wakeup word is included in a first sound signal received during playback of the media content by the smart device in the wakeup word detection state; If the wakeup word is included in the first sound signal, entering the listening state while maintaining playback of the media content; And when entering the listening state, performing any one of a pause of playback of the media content or a volume control of audio data of the media content according to the type of the media content; A control method can be provided.

In addition, in the step of performing any one operation, when the media content is live streaming, the volume of the audio data can be adjusted by reducing or removing the volume of the audio data.

Also, when adjusting the volume of the audio data, displaying text data corresponding to the audio data of which the volume of the media content is adjusted together with video data of the media content may further include.

According to another aspect of the present invention, a wake-up word detection state for determining whether a sound signal received from the outside contains a wake-up word and a wake-up word detection state included in the sound signal 1. A smart device having an operating state including a listening state for recognizing a voice command, comprising: a voice input module receiving a sound signal; A voice output module for outputting voice; An image output module for displaying an image; And playing media content related to the first voice command input by the user, outputting audio data of the media content through the audio output module of the smart device, and video of the media content through the video output module of the smart device. Playing the media content by displaying data, and starting to play the media content, enters the wake-up word detection state while maintaining playback of the media content, and the smart device in the wake-up word detection state is the During playback of the media content, it is determined whether the wakeup word is included in the first sound signal received through the voice input module, and when the wakeup word is included in the first sound signal, playback of the media content is performed. Retain while holding A controller that performs a first operation related to the media content when entering a ning state and entering the listening state, wherein the controller includes the audio data by reducing or removing the volume of the audio data. A smart device performing the first operation may be provided by adjusting the volume of and displaying text data corresponding to the volume-adjusted audio data of the media content together with video data of the media content.

In addition, when the controller enters the listening state and returns to the wake-up word detection state, the controller performs a second operation related to the media content, wherein the controller controls the volume of the audio data of the media content. The second operation may be performed by adjusting before performing the first operation and ending the display of the text data.

In addition, the controller, after entering the listening state, outputs feedback related to a second voice command included in the received second sound signal, outputs the feedback, and then performs a second operation related to the media content, The controller may perform the second operation by adjusting the volume of the audio data of the media content before performing the first operation and ending the display of the text data.

In addition, the controller outputs a talk-back related to the second voice command included in the received second sound signal after entering the listening state, and after the output of the talk-back ends, the second related to the media content. An operation may be performed, but the controller may perform the second operation by adjusting the volume of the audio data of the media content before performing the first operation and ending the display of the text data.

In addition, the text data displayed first among the text data may include text data corresponding to audio data at a time when the volume is adjusted and text data corresponding to audio data before a predetermined time from a time when the volume is adjusted. have.

In addition, the text data that is displayed last among the text data may include text data corresponding to audio data at the time when the volume returns, and text data corresponding to audio data after a predetermined time from the time when the volume returns. You can.

 1. Smart device

 1.1. summary

Hereinafter, a smart device 1000 according to an embodiment of the present invention will be described.

The smart device 1000 according to an embodiment of the present invention may interact with a user through voice. Specifically, the smart device 1000 may receive a user voice and output feedback requested by a voice command included in the user voice. Here, the smart device 1000 may output audio-type or video-type feedback.

 1.2. Terms

Here, before further describing the smart device 1000 according to an embodiment of the present invention, some terms used in the specification will be defined.

 1.2.1. Smart device

In the present specification, the smart device 1000 may include any kind of device capable of interacting with a user and a voice as a medium using a voice assistant function. The smart device 1000 may be representatively provided in the form of a smart speaker, but in addition, the smart device 1000 is provided in the form of a smart phone, smart tablet, laptop, smart television, smart set-top box, smart display, smart projector, etc. It is also possible.

Looking at the component aspect, the smart device 1000 includes a voice input module 1200 for receiving user voice for interaction with a user using voice as a medium, and a voice output module for outputting audio-type feedback. It may include (1300). Also, the smart device 1000 may optionally further include a display module 1400 for outputting video-type feedback. In addition, the smart device 1000 may further include some components including a communication module 1020 for communicating with an external device (eg, a voice assistant server, etc.) in order to realize the voice assistant function, but specifics thereof The description will be described later.

 1.2.2. Voice assistant

The smart device 1000 according to an embodiment of the present invention may implement a voice assistant function. Here, the voice assistant function is a concept encompassing all functions that enable interaction between the user and the smart device 1000 using voice as a medium.

An example in which the voice assistant function is implemented in the present specification may be as follows. The user voice received by the smart device 1000 is delivered to the voice assistant server 10. The voice assistant server 10 interprets the voice command included in the user's voice, obtains feedback data regarding feedback requested by the voice command, and transmits it to the smart device 1000. The smart device 1000 outputs feedback based on feedback data received from the voice assistant server 10. In this way, the smart device 1000 can implement the voice assistant function. However, in the above, it has been described that the smart device 1000 implements the voice assistant function in cooperation with the voice assistant server 10, but in some cases, the smart device 1000 executes the voice assistant function locally with a standalone. It may also be possible.

 1.2.3. Voice assistant server

The voice assistant server 10 may collectively refer to a server that implements the voice assistant function in cooperation with the smart device 1000 according to an embodiment of the present invention. The voice assistant server 10 receives the user's voice from the smart device 1000, extracts the voice command included in the user's voice, interprets the voice command, and provides feedback to the smart device 1000 to output in response to the voice command. Responsible for generating feedback data and delivering it to the smart device 1000.

In the present specification, the voice assistant server 10 may be physically and / or functionally implemented as a single server or a plurality of servers as needed. For example, the voice assistant server 10 includes a voice recognition server that extracts voice commands from voices received from a smart device, an artificial intelligence server that interprets the extracted voice commands, a server that manages multimedia content for providing feedback, and the like. It can be a collection of multiple servers that are the same. That is, the voice assistant server 10 may be in the form of a single server that implements all of the above-mentioned functions and functions necessary for the implementation of other voice assistant functions, as well as a collective form of servers sharing each function.

 1.2.4. User voice

In this specification, the smart device 1000 may acquire a user voice through the voice input module 1200. Here, the user voice may mean a voice spoken by a user using the smart device 1000. For example, the voice input module 1200 of the smart device 1000 receives voice through the voice input module 1200 in a standby mode or a listening mode, which will be described later, so that the operating mode of the smart device 1000 is a standby mode or When the user speaks while in the listening mode, the smart device 1000 may acquire a user voice.

On the other hand, since the smart device 1000 according to an embodiment of the present invention cannot selectively receive a user's voice separately from other voices, the voice received by the smart device 1000 includes voices other than the user's voice. It may be. Examples of other voices may include voices due to feedback output from the smart device 1000 or voices caused by noise generated in the surroundings, etc. Hereinafter, these other voices will be distinguished from user voices and referred to as other voices. do. In addition, among other voices, voices due to feedback output from the smart device 1000 will be referred to as feedback voices.

 1.2.5. Voice command

The user voice may include a voice command for the user to perform / process a specific operation on the smart device 1000. Here, the user voice may be interpreted as a term defined from an acoustic point of view, and the voice command may be interpreted as a term defined from an information processing point of view. Therefore, in this specification, the user voice and the voice command will be used as distinct concepts. However, since the distinction between the user's voice and the voice command is not always clear, and in some cases, the practical benefit of the distinction may not be practical in some cases, some descriptions and patent claims to be described later in this specification are within the scope of understanding by those skilled in the art. It should be noted in advance that the two terms voice and voice command may be used interchangeably.

Meanwhile, the user does not necessarily have to give an instruction to the smart device 1000 only through a voice command. For example, the user may interact with the smart device 1000 in various forms such as button input, touch, and gesture.

 1.2.6. Wake up word

Since the smart device 1000 according to an embodiment of the present invention is difficult or unable to know in advance whether the received voice is a user voice spoken by the user to use the smart device 1000, it is difficult to perform a specific operation from the user. A standby mode may be used as a preliminary phase for entering a listening mode that receives a user voice requiring performance / processing.

The wake-up word may mean a trigger for entering the listening mode from the standby mode. Specifically, the smart device 1000 may detect a wake-up word from the user's voice received in the standby mode, and enter a listening mode when the wake-up word is detected. Accordingly, the wake-up word can be regarded as a special voice command instructing the smart device 1000 to enter the listening mode from the standby mode. When the wake-up word is used as a trigger for entering the listening mode, a malfunction that the smart device 1000 reacts to a user's voice spoken without the intention of the user to use the smart device 1000 can be prevented. .

Depending on the viewpoint, it is also possible to interpret the wake-up word as a kind of voice command, but in this specification, for convenience of description, the wake-up word and the voice command will be referred to separately. However, in some cases, it is revealed in advance that the wakeup word may be described as one type of voice command in this specification. Also, in the following description, the wakeup word may sometimes be replaced with the term hot word.

In the smart device 1000 according to an embodiment of the present invention, a single word or a phrase or a phrase may be set as a wake-up word. In a general usage environment of the smart device 1000, the wake-up word may be used as a call name or the like that the user calls the smart device 1000. For example, the wakeup word may be determined in the setting process of the smart device 1000, and may be determined as 'computer', 'hey speaker', and the like.

As will be described later, since the voice command may take the form of an arbitrary word, phrase, or sentence, a complicated operation is required in the process of recognizing and interpreting the voice command from the user's voice in order to perform the voice assistant function. Therefore, since the smart device 1000 is difficult to recognize any voice command itself, the process of processing the voice command from the user's voice is usually performed in the voice assistant server 10. On the other hand, since the wakeup word is only a single word or a set of several words, the process of detecting the wakeup word from the user's voice may be processed locally in the smart device 1000.

 1.2.7. feedback

In the present specification, the feedback may mean a response output by the smart device 1000 in response to an instruction, request, or request received from the user.

In this specification, feedback may include audio-type feedback and video-type feedback. Here, the audio-type feedback may mean audible feedback output through the voice output module 1300, and the audio-type feedback will be referred to as a talkback for convenience of description. In addition, the video-type feedback may mean visual feedback output through the display module 1400, and the video-type feedback will be referred to as a display back in the following description.

On the other hand, although the term talk-back includes the word 'talk', it is interpreted that the talk-back does not necessarily mean conversational feedback, but encompasses both auditory feedback such as music and sound effects. Should be.

In addition, although the display-back refers to video-type feedback, it is not necessarily in the form of a moving image and may include still images. Furthermore, although the display back is described as visual feedback and the talk back is audible feedback, for convenience of description, the display back may be interpreted as including visual feedback plus auditory feedback. For example, in the present specification, the smart device 1000 may describe that the smart device 1000 outputs a display back for playing multimedia content that provides an audio-visual experience such as a movie or a game. For example, the display back may include a TV program, a movie or music video, a YouTube streaming service, and the like.

 1.3. Components of smart devices

Hereinafter, components of the smart device 1000 according to an embodiment of the present invention will be described.

1 is a block diagram of a smart device 1000 according to an embodiment of the present invention.

The smart device 1000 according to an embodiment of the present invention may include a voice input module 1200, a voice output module 1300, a communication module 1020, a memory 1040, and a controller 1060. The smart device 1000 receives the user voice through the voice input module 1200, transmits the user voice to the voice assistant server 10 through the communication module 1020, and requests by the voice command included in the user voice The received feedback data can be received, and a talkback can be output through the voice output module 1300, and the controller 1060 can control each module required for the above-described process or process various information, and the memory 1040 Various information can be stored in. In addition, the smart device 1000 may optionally further include a display module 1400 to output a display back. In addition, the smart device 1000 may optionally further include a driving module 1500 for adjusting the direction of the display back.

The voice input module 1200 may receive various voices including a user voice. The voice input module 1200 may be provided with a single or multiple microphones 1202. When the voice input module 1200 is provided as a plurality of microphones 1202, the voice input module 1200 may be provided as a microphone array 1204 in which a plurality of microphones 1202 are formed and disposed in a predetermined shape.

The audio output module 1300 may output various sounds including a talkback. The audio output module 1300 may be provided as a single or multiple speakers 1302. The audio output module 1300 may be arranged in a structure having omnidireictional as necessary. Alternatively, the audio output module 1300 may be arranged in a structure that outputs sound in a directional manner as needed.

The display module 1400 may output various images including a display back. The display module 1400 may be implemented in the form of a display panel 1420 or a projector 1440.

In addition, the direction of the display module 1400 or the direction of the display back output by the display module 1400 may be adjusted by the driving module 1500. For example, the smart device 1000 outputs a display back through the projector 1440, and the display area or projection direction in which the display back is output may be dynamically adjusted around the user by the driving module 1500. Meanwhile, the driving module 1500 may further include a direction sensing sensor 1560 for sensing the direction of the current display back. For example, the direction detection sensor 1560 may sense the direction in which the projector 1440 is disposed.

The communication module 1020 communicates with an external device. For example, the smart device 1000 may transmit and receive information to and from the voice assistant server 10 through the communication module 1020, and more specifically, the smart device 1000 through the communication module 1020 voice assistant The user voice may be transmitted to the server 10, and feedback data may be received from the voice assistant server 10 through the communication module 1020.

The communication module 1020 can be roughly divided into a wired type and a wireless type. The wired type and the wireless type have advantages and disadvantages, and the smart device 1000 may be provided with a wired type communication module 1020 and / or a wireless type communication module 1020.

In the case of the wired type, a wired local area network (LAN), universal serial bus (USB) communication, and the like are representative examples, but other methods are also possible. In the case of the wireless type, a mobile communication method, a wireless personal area network (WPAN) communication method such as Bluetooth or Zigbee, and a wireless local area network (WLAN) communication method such as Wi-Fi And other known communication methods. Of course, it is revealed in advance that the wired / wireless communication method is not limited to the above-described example.

The memory 1040 can store various types of information. The memory 1040 may temporarily or semi-permanently store data. Examples of the memory 1040 include a hard disk drive (HDD), solid state drive (SSD), flash memory, read-only memory (ROM), and random access memory (RAM). This can be. The memory 1040 may be provided in a form built into the smart device 1000 or detachably attached to a feedback device.

The memory 1040 stores an operating program (OS: Operating System) for driving the smart device 1000, various applications installed in the smart device 1000, and various data necessary or used for the operation of the smart device 1000. Can be.

The controller 1060 may control the overall operation of the smart device 1000. For example, the smart device 1000 outputting the display back may be performed as the controller 1060 controls the display module 1400, and the smart device 1000 communicates with the voice assistant server 10. This may be performed as the controller 1060 controls the communication module 1020.

The control operation of the controller 1060 may be performed as the controller 1060 performs calculation and processing of various information. To this end, the controller 1060 may be implemented as a computer or similar device according to hardware or software or a combination thereof. In hardware, the controller 1060 may be provided in the form of an electronic circuit that processes an electrical signal and performs a control function. In software, the controller 1060 may be provided in the form of a program or code that drives a hardware circuit.

The controller 1060 may have a single physical configuration, but in some cases, may be provided in a physically separated form. In other words, the controller 1060 may be made of a single chip, but may be provided as a plurality of chips or substrates that are physically distributed, and at this time, a communication interface between each separated controller 1060 may be connected. have.

Meanwhile, in the following description, it is revealed that operations performed by the smart device 1000 may be interpreted as being performed by the controller 1060 unless otherwise specified.

 1.4. Smart device implementation

Hereinafter, some implementations of the smart device 1000 according to an embodiment of the present invention will be described. 2 to 8 are perspective views of implementations of the smart device 1000 according to an embodiment of the present invention.

According to an example, the smart device 1000 may be provided in a form of a smart speaker 1000a mounted on a horizontal surface, such as a table or floor, as illustrated in FIG. 2.

In this example, the housing 1100 may include a lower surface 1101 lying on a horizontal surface, an upper surface 1102 corresponding to the lower surface 1101, and a side surface 1103 connecting the lower surface 1101 and the upper surface 1102. have. 2 shows the housing 1100 in a circular column shape, but the housing 1100 may have various shapes other than a polygonal column, a circular or polygonal column having an inclined top surface 1102, a circle or polygonal horn, and the like. .

In this example, the smart device 1000 may be provided with an indicator 1106 indicating the operation mode of the smart device 1000. For example, the indicator 1106 may be a lamp or the like that displays a specific color or a specific pattern depending on the operation mode. In FIG. 2, the indicator 1106 is shown to be arranged to surround the rim of the side surface 1103 of the housing 1100, but this does not limit the shape or position of the indicator 1106 to FIG. 2.

In this example, a microphone array 1204 including a plurality of microphones 1202 may be provided in the smart device 1000. For example, the plurality of microphones 1202 may be disposed radially on the top surface 1102 of the housing 1100 or may be disposed along the side surface 1103 of the housing 1100 as illustrated in FIG. 2. Of course, it is also possible that a single microphone 1202 is provided in the smart device 1000. For example, when a single microphone 1202 is used as the voice input module 1200, the microphone 1202 is a point on the side 1103 of the housing 1100 where the smart device 1000 is mainly used, or It may be disposed on the upper surface 1102 of the housing 1100.

In this example, a single speaker 1302 may be provided in the smart device 1000 to output voice in an omni-directional manner. For example, the speaker 1302 is disposed inside the housing 1100 to output sound toward the lower surface 1101 of the housing 1100, and a cone-shaped protrusion is provided on the lower surface 1101 of the housing 1100 By doing so, the sound path can be output in the omni-directional direction of the housing 1100. Of course, it is also possible to provide a plurality of speakers 1302, and at this time, the smart device 1000 may be capable of directional voice output or multi-channel (eg, stereo sound or 5.1 channel) voice output.

In the implementation of the smart device 1000 according to the present example, since the display module 1400 is not mounted on the smart device 1000, the smart device 1000 cannot output a display back. Some of the smart device 1000 and a control method according to an embodiment of the present invention to be described later may be applied only to the smart device 1000 on which the display module 1400 is mounted, while the other part may include a display module 1400. It is revealed in advance that it can also be applied to the missing smart device 1000.

According to an example, the smart device 1000 is mounted on a horizontal surface, such as a table or a floor, as illustrated in FIG. 3, and may be provided in a form 1000b having a display panel 1420.

In this example, the display panel 1420 may be provided in the smart device 1000. The display panel 1420 may be mainly provided on one surface of the housing 1100. The smart device 1000 may output various images (eg, display back) through the display panel 1420. Meanwhile, as the display panel 1420 is provided as a touch panel, the display panel 1420 may function as a touch input interface.

In addition, since the display panel 1420 may replace the role of the indicator 1106, the indicator 1106 in the smart device 1000 may be selectively provided in this example. In addition, in the smart device 1000 of this example, the voice output module 1300 and the voice input module 1200 may be provided in various arrangements.

In the above, the smart device 1000 of this example has been described as being stationary, but the smart device 1000 may be provided in a wall-mounted manner. At this time, the housing 1100 may be provided with a mounting means for the smart device 1000 to be hung on the wall. For example, the mounting means may be provided as an adhesive layer, bracket, recess, or the like.

According to an example, the smart device 1000 may be provided in a form 1000c having a projector 1440 as illustrated in FIG. 4. Accordingly, the smart device 1000 may output various images through the projector 1440. The smart device 1000 illustrated in FIG. 4 may be mounted on a horizontal surface or used by being installed on a ceiling or a wall surface. When a usage environment with a short projection distance is desired, an ultra short throw (UST) projector may be used as the projector 1440 of the smart device 1000.

In this example, the smart device 1000 may optionally include a touch input interface, a gesture input interface, a gaze recognition interface, and / or a spatial recognition interface.

The touch input interface may detect a user's touch input on a surface on which the smart device 1000 is placed or a surface on which the smart device 1000 projects a display back. An example of a touch input interface is an infrared touch sensor. The infrared touch sensor includes a light-emitting means for irradiating infrared rays and a light-receiving means for receiving infrared rays, and a touch input may be obtained using infrared rays emitted by the light-emitting means reflected by a user's body and received as light-receiving means. . In the infrared touch sensor, an ordinary camera or an infrared camera may be used instead of the light receiving means. In the case of using an infrared camera, a touch input may be obtained by emitting infrared rays so that the light emitting means forms a predetermined pattern, and detecting a pattern deformation by the user's body in the infrared camera.

In addition, the gesture input interface may recognize gestures (eg, arm movements, finger movements, finger shapes, etc.) expressed by the user's body based on various images. Unlike the touch interface, the gesture input interface has an advantage of being able to receive a gesture for a space without receiving only a touch on a physical surface.

In addition, the gaze recognition interface may recognize the gaze of the user of the smart device 1000. For example, the gaze recognition interface may recognize a direction viewed by the user or a point viewed by the user as 2D or 3D information.

In addition, the spatial recognition interface may recognize spaces and objects around the smart device 1000. For example, the spatial recognition interface may recognize the structure of the room in which the smart device 1000 is placed, the location and shape of objects placed around it.

Of course, the touch input interface, the gesture input interface, the gaze recognition interface, and / or the space recognition interface of the smart device 1000 are not limited to the above-described examples, and various modifications apparent to those skilled in the art are possible. In addition, it is revealed that the touch input interface, the gesture input interface, the gaze recognition interface, and / or the spatial recognition interface can be applied to the smart device 1000 of this example as well as other implementations of the smart device 1000.

In this example, the smart device 1000 may be provided in a form (1000d) in which the projection direction is adjusted as its mounting direction is adjusted. For example, referring to FIG. 5, the housing 1100 of the smart device 1000 may have at least two mounting surfaces 1104 and 1105, and the housing 1100 through any one of the two mounting surfaces 1104 and 1105. Whether the projection direction faces the wall surface or the floor surface may be determined according to whether or not it is mounted. In other words, the projection direction or projection area of the smart device 1000 may be adjusted by the user manually adjusting the posture of the smart device 1000.

On the other hand, unlike the projection direction or the area is adjusted as the mounting posture is manually adjusted, in this example, the smart device 1000 may automatically adjust the projection direction. To this end, the smart device 1000 may further include a driving module 1500 that controls the direction of the projection or the area of the projection. For example, the smart device 1000 may be provided with a rotation motor 1520 for rotating the projector as a driving module 1500. For another example, the smart device 1000 may be provided with a reflection mirror or the like for adjusting the light path of the projection as the driving module 1500. As another example, the driving module 1500 may be provided in the form of a combination of a rotating motor 1520 and a reflection mirror. Here, the reflective mirror may be implemented in a solid-state type such as a MEMS mirror or a resonance mirror, or may be provided as a nodeing mirror or a polygonal mirror that is physically oriented. Referring to FIG. 6, the smart device 1000 is mounted on a table and outputs a display bag over the table. At this time, the smart device 1000 may use the driving module 1500 to move the projection direction of the display bag to the user direction. have. In addition, the smart device 1000 may appropriately adjust a wall surface or a table surface according to the projection direction by using the driving module 1500. The smart device 1000 that automatically adjusts the projection direction or the projection area is not limited to the type having the projector 1440. For example, the smart device 1000 of the type shown in FIG. 1 is equipped with a driving module 1500 that adjusts the direction of the display panel 1420 so that the smart device 1000 automatically displays through the driving module 15000. It is also possible to adjust the direction.

 1.5. Operating mode of smart device

Hereinafter, an operation mode of the smart device 1000 will be described.

The smart device 1000 according to an embodiment of the present invention may mainly receive a user voice and provide feedback in response to a voice command included in the user voice. Here, some of some of the technical factors considered in connection with the process of the smart device 1000 performing the above-described operation, that is, receiving the user's voice and providing feedback in response thereto may be as follows.

The first is to grasp the user's intention to speak. Even if the user is in an environment where the smart device 1000 can be used, the user's speech may not necessarily be to use the smart device 1000. If the smart device 1000 reacts to the user's voice that is spoken without the intention to use the smart device 1000, the smart device 1000 may malfunction, and accordingly, user convenience may deteriorate.

The second is the protection of personal information. The smart device 1000 mainly transmits the received user voice to the voice assistant server 10 in order to interpret the voice command, and all user voices received by the voice input module 1200 of the smart device 1000 are all voice assistants. If it is transmitted to the server 10, there is a possibility that personal information that the user does not want is transmitted to the operating subject of the voice assistant server 10. This can be accepted as a leak of personal information from the user's perspective.

In relation to at least the above two points, the smart device 1000 distinguishes and processes whether or not the voice received through the voice input module 1200 is uttered or not intended to use the smart device 1000. You should be able to.

However, in general, since the smart device 1000 does not have the ability to decode the received user's voice in the natural language form on its own, the voice input through the voice input module 1200 may prevent the user from using the smart device 1000. It can be difficult to determine whether or not you have spoken to use it.

Accordingly, the smart device 1000 according to an embodiment of the present invention is a smart device 1000 from the user voice prior to the phase of transmitting the received user voice to the voice assistant server 10 for interpretation of the voice command. By operating a phase of determining a user's intention as to whether or not a subsequent user voice is uttered from the intention to use the smart device 1000 based on whether a specific word reflecting the intention to use is detected, The user's voice uttered with the intention to use the smart device 1000 may be distinguished and processed.

Basically, an operation mode of the smart device 1000 according to an embodiment of the present invention may include a standby mode and a listening mode. Here, the standby mode is a mode for determining a user's intention to use the smart device 1000, and a listening mode is a mode for receiving a user's voice containing a voice command on the premise that the user has an intention to use the smart device 1000. to be.

Hereinafter, various operating modes of the smart device 1000 according to an embodiment of the present invention will be described in detail.

 1.5.1. Off state

The smart device 1000 may have an off state in which the device does not operate. In the off state, the device may be turned off because no power is supplied, or may be in a hibernation state using only minimal power.

 1.5.2. Standby mode

When the power is applied, the smart device 1000 may operate in a standby mode without being specifically instructed by the user. For example, the smart device 1000 may enter a standby mode when power is applied.

The standby mode is a phase in which the user intends to use the smart device 1000. For example, the smart device 1000 may perform an operation of detecting a wake-up word from a user's voice in standby mode, and determine a user's intention to use the smart device 1000 according to whether the wake-up word is detected. have.

In the standby mode, the smart device 1000 may receive voice through the voice input module 1200. At this time, the voice input module 1200 may receive the voice non-selectively. Since the smart device 1000 does not yet know whether the received voice is a user's voice uttered toward the smart device 1000, the smart device 1000 may not transmit the received voice to the voice assistant server 10 for interpretation of the voice command.

In the standby mode, the smart device 1000 may detect a wakeup word from the received voice. The wakeup word may be a specific word or phrase predetermined by the manufacturer of the smart device 1000 or a specific word group or phrase group predetermined by the manufacturer of the smart device 1000 by the user.

As described above, since the wake-up word is not an arbitrary natural language form, but only a few predetermined words or short phrases, detecting the wake-up word from the input voice does not require a lot of computation. Accordingly, the controller 1060 of the smart device 1000 can detect the wakeup word from the voice received directly locally without collaboration with the voice assistant server 10.

In the standby mode, the smart device 1000 may determine a user's intention to use the smart device 1000 based on whether a wakeup word is detected from the received voice. When a wakeup word is detected from the received voice, the smart device 1000 may be understood as having a user's intention to use the smart device 1000.

If the smart device 1000 determines that the user has an intention to use the smart device 1000, the smart device 1000 may enter a listening mode in preparation for receiving a user voice containing a voice command from the user. Conversely, when a wake-up word is not detected from the received voice, the smart device 1000 may understand that the user does not intend to use the smart device 1000, and the smart device 1000 may instead enter the listening mode. Standby mode can be maintained.

Meanwhile, the standby mode is a mode for detecting / recognizing wake-up words, and may be referred to as a wake-up word detection state or mode in some cases.

 1.5.3. Listening mode

As described above, when the wake-up word input is detected in the standby mode, the smart device 1000 may enter the listening mode. Meanwhile, a button for instructing the smart device 1000 to enter the listening mode is provided, and the smart device 1000 may enter the listening mode in response to a user input for the corresponding button.

In the listening mode, the smart device 1000 may receive a user voice through the voice input module 1200. The user voice input in the listening mode may include a voice command instructing a specific operation to the smart device 1000.

Here, although not necessarily, the voice command may mainly take a natural language form. Recognizing the voice command in the form of a natural language may be difficult to be processed locally due to a high amount of computation, so the smart device 1000 may interpret the voice command from the user's voice in cooperation with the voice assistant server 10.

Accordingly, in the listening mode, the smart device 1000 may transmit the input user voice to the voice assistant server 10 for interpretation of the voice command included in the user voice. The smart device 1000 may transmit the user's voice to the voice assistant server 10 in several ways. For example, the smart device 1000 may transmit the user voice received in the listening mode to the voice assistant server 10 in real time. As another example, the smart device 1000 may collect the received user voices and then transmit the collected user voices to the voice assistant server 10 collectively. The voice assistant server 10 may interpret the voice command from the user voice received from the smart device 1000, generate feedback data corresponding to the analyzed voice command, and return it to the smart device 1000.

In the listening mode, before the smart device 1000 transmits the user voice to the voice assistant server 10, the smart device 1000 may perform pre-processing of the user voice. Examples of preprocessing include noise canceling and voice data compression. For example, when the smart device 1000 outputting the talkback through the voice output module 1300 receives the user voice in the listening mode, the smart device 1000 receives the user voice including the talkback and voice commands together. You can. At this time, the smart device 1000 may extract the user's voice portion from the received voice by removing the talk back portion of the received voice by using the information about the talk bag output by itself.

In addition, in the listening mode, the smart device 1000 may perform an operation for creating a quiet surrounding environment in order to receive a clean user voice. For example, when entering the listening mode, the smart device 1000 may stop the output of the talkback that was previously being output or reduce the audio volume of the talkback. For example, the smart device 1000 playing radio news may pause the live news being played when entering the listening mode.

The smart device 1000 may maintain the listening mode for a certain time period. For example, when the smart device 1000 transmits a user input containing a voice command to the voice assistant server 10 and receives feedback data from the voice assistant server 10, the listening mode may be terminated. For another example, the smart device 1000 may terminate the listening mode when the input of the user's voice is completed or when the voice is not input for a predetermined time after the user's voice is input. As another example, after entering the listening mode, the smart device 1000 may terminate the listening mode when there is no user's voice input for a predetermined time. Hereinafter, a time interval in which the smart device 1000 maintains the listening mode will be referred to as a 'listening window'. In addition, in this connection, the operation in which the smart device 1000 enters the listening mode is referred to as an 'opening a listening window', and conversely, the operation in which the smart device 1000 exits the listening mode is referred to as the 'listening window'. It is referred to as closing (closing a listening window), and the state in which the smart device 1000 maintains the listening mode will be referred to as the 'listening a listening window' state.

Meanwhile, in some cases, the smart device 1000 may detect a wakeup word from the user's voice even in listening mode. When the wakeup word is detected in the listening mode, the smart device 1000 may restart the listening mode and initialize the listening window.

 1.5.4. Response mode

In the listening mode, when the smart device 1000 delivers the input user voice to the voice assistant server 10, the voice assistant server 10 extracts the voice command from the user voice, interprets the extracted voice command, and interprets the analyzed voice The feedback data may be generated based on the command and transmitted to the smart device 1000.

The smart device 1000 receiving the feedback data may output feedback using the feedback data. As such, the phase of receiving and outputting feedback is a response mode. In response mode, the smart device 1000 receives feedback data and outputs feedback accordingly. The smart device 1000 may output a talk back and / or a display back.

On the other hand, the smart device 1000 may receive feedback data and start outputting the feedback at the same time or immediately after starting outputting the feedback, and then enter the standby mode. In this case, the smart device 1000 may output feedback in a standby mode or a listening mode. Depending on the viewpoint, instead of prescribing that the smart device 1000 continues to output the feedback in the standby mode or the listening mode, it may be interpreted that the response mode and the standby mode are operated simultaneously.

For example, the smart device 1000 that is requested to stream live news from a user receives feedback data for live news from the voice assistant server 10 in response mode, starts playback of live news, and then enters standby mode. Stream live news. As another example, if a wake-up word is detected during live news streaming in the standby mode, the smart device 1000 enters a listening mode, but it is also possible to still maintain live news streaming.

 1.5.5. Operation in operation

The smart device 1000 according to an embodiment of the present invention does not have to operate all the above-described modes. For example, it is also possible that the response mode is omitted in the operation mode of the smart device 1000. In addition, not all of the above described in the smart device 1000 according to an embodiment of the present invention should be operated independently. For example, the standby mode, the listening mode, and the response mode of the smart device 1000 may have an operation mode in which all or a part thereof is overlapped.

7 and 8 are diagrams for an example of operation of an operation mode of the smart device 1000 according to an embodiment of the present invention, and FIG. 9 is a smart device 1000 and a voice assistant according to an embodiment of the present invention This is a diagram of communication between the servers 10.

7 and 9, the smart device 1000 may enter a standby mode from an off state when power is applied. The smart device 1000 in standby mode continuously receives voice through the voice input module 1200 and detects a wakeup word from the received voice.

When the wakeup word is detected, the smart device 1000 may enter a listening mode. Alternatively, even when a user input such as a touch / button / gesture indicating a listening mode is input, the smart device 1000 may enter the listening mode. In the listening mode, the smart device 1000 may receive a user voice containing a voice command.

The smart device 1000 may transmit the received user voice to the voice assistant server 10. The smart device 1000 may receive feedback data for the voice command from the voice assistant server 10 and output feedback based on the feedback data.

The smart device 1000 in the listening mode may return from the listening mode to the standby mode. For example, the smart device 1000 in the listening mode may return to the standby mode if the user's voice is not input after entering the listening mode. Also, for example, the smart device 1000 in the listening mode may return to the standby mode if no additional user voice is input after the user voice is input. In addition, for example, when the smart device 1000 in the listening mode transmits the user's voice to the voice assistant, receives feedback data of feedback related to the voice command from the voice assistant server 10 or based on the feedback data. When the output (or start of output) is performed, the standby mode may be returned.

Referring to FIGS. 8 and 9, when power is applied, the smart device 1000 may enter a standby mode from an off state. The smart device 1000 in the standby mode continuously receives voice through the voice input module 1200 and detects a wakeup word from the received voice.

When the wakeup word is detected, the smart device 1000 may enter a listening mode.

In the listening mode, the smart device 1000 may receive a user voice containing a voice command. The smart device 1000 may transmit the received user voice to the voice assistant server 10.

The smart device 1000 that transmits the received user voice to the voice assistant server 10 may enter a response mode and output feedback. The listening device smart device 1000 may enter a response mode when the user voice is transmitted to the voice assistant server 10 or when feedback data is received from the voice assistant server 10.

The smart device 1000 may receive feedback data for the voice command from the voice assistant server 10 and output feedback based on the feedback data. The smart device 1000 that ends the output of the feedback or initiates the output of the feedback may return to the standby mode.

On the other hand, the smart device 1000 in the listening mode may return to the standby mode instead of entering the response mode when the user voice is not received while in the listening mode.

 2. Voice processing during media content playback

 2.1. summary

In the above, it has been described that the smart device 1000 according to an embodiment of the present invention is equipped with a voice assistant function to receive a user voice and output feedback corresponding to a voice command included in the received user voice.

The process of recognizing the voice command in detail is as follows.

First, the smart device 1000 may receive a user voice including a voice command in a listening state, and transmit the input user voice to the voice assistant server 10. At this time, the user voice transmitted to the voice assistant server 10 may be a user voice that has undergone pre-processing such as noise canceling in the smart device 1000. When receiving the user's voice, the voice assistant server 10 performs speech-to-text (STT) to textize the voice signal and recognize the voice command based on the textualized content. When the voice command is recognized, the voice assistant server 10 may generate appropriate feedback data based on the recognized voice command and transmit it to the smart device 1000. When the feedback data is transmitted, the smart device 1000 may output it as a voice-type talkback or a video-type displayback.

The smart device 1000 according to an embodiment of the present invention has the characteristics of a voice interface device that interacts with a user through talk made by a user and a talkback output through a speaker, and at the same time plays content such as video or music. It has a personality as a content consuming device. Therefore, the smart device 1000 can be used to play multimedia content such as movies and music on a daily basis. Accordingly, it may be advantageous for the smart device 1000 according to an embodiment of the present invention to receive a user voice containing a voice command from a user even while playing media content.

When the smart device 1000 receives the user's voice during the playback of the media content, in addition to the voice according to the user's speech, the output of audio data according to the playback of the media content is transmitted to the voice input module 1200 of the smart device 1000. Can be entered together. In this case, the sound by audio data may act as a factor that makes it difficult to interpret the voice according to the user's speech. Particularly, when the smart device 1000 recognizes a certain voice command from the voice signal input in the listening state even if it is not difficult to detect the wake-up words corresponding to some predetermined words from the voice signal received in the standby mode. When the audio output of the media content is mixed with the input voice signal, there is a high possibility that the recognition rate of the voice command is dramatically reduced.

Accordingly, when the smart device 1000 according to an embodiment of the present invention enters a listening state during playback of media content and receives a voice signal including a voice command, the audio output of the media content being played is transmitted to the user's speech. In order to prevent the recognition rate of the voice command included in the voice signal from being lowered, a predetermined process may be performed on the media content.

 2.2. Standby mode during media content playback

10 is a diagram for operation of a standby mode during media content playback according to an embodiment of the present invention.

Referring to FIG. 10, the smart device 1000 may play media content. The operation mode of the smart device 1000 playing media content may be a standby mode. In other words, the smart device 1000 may receive a voice signal even while playing media content, and recognize a wakeup word from the received voice signal. Since the wake-up word is limited to a predetermined word unlike other voice commands that are randomly spoken by the user, the smart device 1000 wakes up from the input voice signal even if the voice output of the media content being played is mixed with the user's voice. Up word can be detected.

 2.3. Listening mode during media content playback

11 is a diagram for operation of a listening mode during media content playback according to an embodiment of the present invention.

Referring to FIG. 11, when the smart device 1000 detects a wakeup word, the operating state of the smart device 1000 may transition from the standby mode to the listening state. The smart device 1000 that is playing the media content may control the playback of the media content when its operation mode is switched to the listening mode.

For example, the smart device 1000 may maintain the playback of the media content, but may lower or remove the voice volume of the media content. As another example, the smart device 1000 may stop playing the media content.

By controlling the reproduction of the media content as described above, when the user speaks, the smart device 1000 may receive only the voice signal of the user whose audio data of the media content is not mixed. Accordingly, when the operating state of the smart device 1000 is in the listening state, the recognition rate of the voice command of the user can be improved.

When entering the listening mode, if the media content stops playing, the user must stop watching the media content, but when entering the listening mode, if the volume is lowered or eliminated while maintaining the media content playing, the user can use the voice command. It has the advantage of continuing the appreciation of media content while igniting. However, since the media content is played in a state in which there is no or low voice volume of the media content, the smart device 1000 may output subtitles corresponding to audio data to compensate for the volume of the media content being erased or reduced. In addition, the smart device 1000 may perform various operations so that the user can continue to enjoy the media content that was being played in the listening mode, and detailed description thereof will be described later.

 3. Control methods of smart devices

Hereinafter, examples of a method of operating a listening mode or a method of processing a media content that is currently playing when entering the listening mode will be described with reference to an embodiment of the present invention. On the other hand, in the description of the examples of the method to be described later, each example is described as being performed by the above-described smart device 1000, and thus the methods described below can be implemented as a control method of the smart device 1000. Do. However, since this is merely for convenience of description, it is revealed in advance that methods for outputting directional feedback according to an embodiment of the present invention are not necessarily limited by the above-described smart device 1000.

 3.1. Example 1

12 is a flowchart of a first example of a method for controlling a smart device according to an embodiment of the present invention.

Referring to FIG. 12, a method for controlling a smart device according to an embodiment of the present invention includes a media content playback step (S1110), a step of receiving a voice signal during media content playback (S1120), and a wake-up word to the received voice signal If included, it may include entering a listening state (S1130) and adjusting the volume of media content when entering the listening state (S1140).

Hereinafter, each step will be described in more detail.

The smart device 1000 may play the media content (S1110).

Here, the media content may include at least audio data. For example, the media content may include video data and audio data. As another example, the media content may include only audio data excluding video data. The controller 1060 may play media content by outputting audio data of the media content through the audio output module 1300.

In addition, the media content may be provided as feedback for the voice command of the user. Therefore, when the media content includes only audio data, it may be a talk back, and when the media content further includes video data, it may be a display back. However, the media content does not necessarily have to be feedback by voice commands.

The smart device 1000 may receive a voice signal during media content playback (S1120). For example, the voice signal may include a user's voice received from the outside of the smart device 1000. At this time, the voice of the user may include a wake-up word. The smart device 1000 may recognize a wakeup word from the user's voice.

The smart device 1000 may enter a listening state upon receiving a user voice including a wake-up word (S1130).

When the operating state of the smart device 1000 is switched to the listening state, the smart device 1000 may adjust the volume of the media content to increase the recognition rate of the user's voice command (S1140). For example, the smart device 1000 may adjust by lowering or removing the volume of audio data of the media content being played in order to remove the sound excluding the user's voice.

For example, the smart device 1000 may adjust the volume of the audio data of the media content in various ways. For example, the volume of the audio data may be reduced to a predetermined size. Here, the predetermined size may be 0.

As another example, the smart device 1000 may end the output of audio data. Accordingly, the sound of the media content being played can be removed.

As another example, the smart device 1000 may prevent the sound of media content from being output by deactivating the audio output module 1300.

As described above, when the smart device 1000 adjusts the volume of audio data, the smart device 1000 may gradually or rapidly adjust the volume.

Accordingly, the smart device 1000 may continuously receive the media content to the user while receiving the voice command without noise due to the audio data of the media content being played back.

Meanwhile, the smart device 1000 may restore the volume of the media content to a certain point in time.

When the listening mode ends, the smart device 1000 may adjust the volume of the media content as it is.

For example, the smart device 1000 may return to the standby mode when a predetermined time elapses without inputting a voice command in the listening mode. When returning to the standby mode, the volume of media content may be adjusted as it is. For another example, the smart device 1000 may adjust the volume of the media content as it is when outputting feedback on the voice command received in the listening mode. In this case, the feedback may be a display back without audio data. For another example, the smart device 1000 may adjust the volume of the media content to the original when output of the feedback is terminated for the voice command received in the listening mode. In this case, the feedback may be a talk back or a display back including audio data.

 3.2. Example 2

13 is a flowchart of a second example of a method for controlling a smart device according to an embodiment of the present invention.

Referring to FIG. 13, a method for controlling a smart device according to an embodiment of the present invention includes a media content playback step (S1210), a step of receiving a voice signal during media content playback (S1220), and a wake-up word to the received voice signal When included, the step of entering into the listening state (S1230), adjusting the volume of the media content when entering the listening state (S1240) and displaying the text data when entering the listening state may further include (S1250). . Here, since steps S1210 to S1240 may be performed similarly to steps S1110 to S1140 described above, a description thereof will be omitted below.

When the smart device 1000 lowers or removes the volume of the audio data of the media content being played in the listening mode, the user may not be able to fully appreciate the media content. For example, when live news or the like is provided, the user may not know the content of the live news in the listening mode. Accordingly, the smart device 1000 may output text data corresponding to the audio data of the media content in order to supplement the voice of the reduced or removed media content (S1250).

Here, the smart device 1000 may receive text data from the voice assistant server 10. For example, when entering the listening mode, the smart device 1000 may request text data from the voice assistant server 10.

14 and 15 are diagrams for displaying text data according to an embodiment of the present invention.

Referring to FIG. 14, the smart device 1000 may recognize a wakeup word during media content playback, enter a listening state, and display an indication that the operating state is a listening state. When the operating state of the smart device 1000 is switched to the listening state, the smart device 1000 may adjust the volume of audio data of the media content being played and output text data corresponding to the volume-adjusted audio data. At this time, the smart device 1000 may display an indication indicating the volume status of the audio data.

Referring to FIG. 15, the smart device 1000 may display text data corresponding to audio data before the volume is adjusted as well as text data corresponding to the audio data whose volume is adjusted. Specifically, in FIG. 15, the volume of the "highest temperature is 6 degrees in Seoul and 8 degrees in Busan ..." part of the "highest temperature is 6 degrees in Seoul" is the audio data output before the wake-up word is input. When the wake-up word is input and the volume is adjusted after the wake-up word is input, the smart device 1000 may output text data even for the portion where the volume is not adjusted. have. That is, the smart device 1000 may output text data corresponding to audio data having a volume control and text data corresponding to audio data associated with a volume-controlled audio data.

Here, the audio data related to the volume-controlled audio data may be a part contextually related to the volume-adjusted audio data. For example, it may be a portion in which the volume is adjusted and a sentence or a phrase. Alternatively, the audio data related to the volume-adjusted audio data may be audio data from a point in time before a predetermined time before the volume is adjusted to a point in time when the volume is adjusted. For example, it may be audio data output for the previous 2 seconds from the time when the volume is adjusted.

On the other hand, when the smart device 1000 returns the volume of the media content to the original, the display of text data may also be ended.

For example, the smart device 1000 may end the display of text data at a time point when the volume of the media content is restored. For another example, the smart device 1000 may maintain the text data until the output of audio data corresponding to the displayed text data ends and then display the text data.

 3.3. Example 3

16 is a flowchart of a third example of a method for controlling a smart device according to an embodiment of the present invention.

Referring to FIG. 16, a method for controlling a smart device according to an embodiment of the present invention includes playing a media content (S1310), receiving a voice signal while playing the media content (S1320), and entering a listening state. (S1330), adjusting the volume of media content (S1340), displaying text data (S1350), outputting feedback (S1360), stopping playing media content (S1370) and playing media content It may include the step of resuming (S1380).

Hereinafter, each step will be described in more detail. However, since steps S1310 to S1360 may be performed similarly to the other steps described above, a detailed description thereof will be omitted.

When entering the listening mode, the smart device 1000 maintains playback of the media content that was previously playing, but controls the volume of the audio data of the media content and outputs text data corresponding to the volume-controlled audio data. In this state, the smart device 1000 may receive a voice command in the listening mode and output feedback in response (S1360).

At this time, the smart device 1000 may stop playing the media content in order to output feedback (S1370). For example, if the feedback is a display back or talk white containing audio data, the smart device 1000 may stop playing the media content to output it. Then, when the feedback output ends, the smart device 1000 may resume playback of the media content (S1380). Specifically, assuming a smart device 1000 that receives a voice command that asks for today's weather while playing live news, the smart device 1000 normally displays live news while the user enters a wake-up word before asking for today's weather. When the wake-up word is recognized and enters the listening mode, the volume of live news is adjusted, and while the weather information corresponding to the voice command asking for weather is output as a talkback, the live news is stopped, and when the talkback ends Live news can be resumed.

 3.4. Example 4

17 is a flowchart of a fourth example of a method for controlling a smart device according to an embodiment of the present invention.

Referring to FIG. 17, a method for controlling a smart device according to an embodiment of the present invention includes playing a media content (S1410), receiving a voice signal while playing the media content (S1420), and entering a listening state. (S1430), determining the type of media content (S1440) and maintaining or stopping the playback of the media content according to the type of media content (S1450).

Hereinafter, each step will be described in more detail. However, since steps S1410 to S1430 may be performed similarly to the other steps described above, detailed description thereof will be omitted.

In other examples of the method for controlling a smart device according to an embodiment of the present invention described above, when the smart device 1000 enters a listening mode during media content playback, the smart device 1000 maintains playback of the media content. In other words, the smart device 1000 may stop playing the media content.

At this time, the smart device 1000 may determine whether to maintain or stop the playback of the media content when entering the listening mode, according to the type of the media content being played (S1440). Here, the type of media content may include, for example, live content and non-live content.

For example, the smart device 1000 may determine that the media content is being played when the media content being played is live content such as news or real-time sports broadcast. In this case, the smart device 1000 may maintain the playback of the media content when entering the listening mode, but may also adjust the volume and output text data (S1450).

As another example, the smart device 1000 may determine to stop playing the media content when the media content being played is non-live content, such as a replay or movie. At this time, when entering the listening mode, the smart device 1000 may stop playback of the media content (S1450), and then resume playback.

 3.5. Example 5

18 is a flowchart of a fifth example of a method for controlling a smart device according to an embodiment of the present invention.

Referring to FIG. 18, a method for controlling a smart device according to an embodiment of the present invention includes receiving a voice signal in a standby mode (S1510), entering a listening state (S1520), and receiving a voice command ( S1530) and outputting feedback corresponding to the voice command in consideration of whether the media content is played in the standby mode (S1540).

Hereinafter, each step will be described in more detail.

First, the smart device 1000 may receive a voice signal in standby mode (S1510), and may enter a listening mode when a wakeup word is included in the voice signal (S1520). In the listening mode, the smart device 1000 receives a voice command (S1530), delivers it to the voice assistant server 10, receives feedback data from the voice assistant server 10, and outputs feedback using the received feedback data can do.

At this time, the smart device 1000 may output feedback considering whether to play the media content in the standby mode (S1540).

For example, when the media content is being played in the standby mode, the smart device 1000 may output the feedback shorter than the case where it is not. To this end, the smart device 1000 may omit some of the feedback content to be output from the received feedback data or output it at a high speed. Specifically, a part of the talk bag may be converted into a display bag to reduce the content of the talk bag or to speed up the talk bag.

For another example, the smart device 1000 delivers information indicating whether media content was already playing in standby mode when the user voice containing the voice command is transmitted to the voice assistant server 10, and the voice assistant server (10) may generate feedback data shorter than that in the case of not based on information indicating whether or not the media content was previously played in the standby mode.

For another example, when there is media content that is being played, the smart device 1000 may adjust the data type of the output feedback in consideration of the data type of the media content that is being played. For example, when the pre-played media content is in the form of audio data, feedback can be output to the display back so that the user can listen to the pre-played media content audibly and visually view the newly requested feedback. Or vice versa, if the media content that is being played is in the form of video data, the feedback may be output as a talkback to allow the user to visually watch the media content that is currently playing and audibly listen to the newly requested feedback. To this end, the smart device 1000 may determine a feedback type in consideration of the type of media content that is being played from the received feedback data. Alternatively, the smart device 1000 may transmit the form of the media content that is being played to the voice assistant server 10 to allow the voice assistant server to determine the feedback type in consideration of this, and then receive feedback data accordingly.

The methods according to the embodiments of the present invention described above may be used alone or in combination with each other. In addition, since each step described in each method is not essential, it is possible that each method includes all the steps as well as a part. In addition, since the order in which the steps are described is only for convenience of description, the steps in the above methods are not necessarily performed in the order described.

In addition, the method according to the above-described embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. Here, the computer-readable medium may include program instructions, data files, data structures, or the like alone or in combination. In addition, the program instructions recorded on the computer-readable medium may be specially designed and configured for an embodiment or may be known and available to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs, DVDs, and magnetic media such as floptical disks. -Hardware devices specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language code that can be executed by a computer using an interpreter, etc., as well as machine language codes produced by a compiler. The hardware device described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

The above description is merely illustrative of the technical idea of the present invention, and those skilled in the art to which the present invention pertains may make various modifications and variations without departing from the essential characteristics of the present invention. Therefore, the embodiments of the present invention described above may be implemented separately or in combination with each other.

Therefore, the embodiments disclosed in the present invention are not intended to limit the technical spirit of the present invention, but to explain, and the scope of the technical spirit of the present invention is not limited by these embodiments. The scope of protection of the present invention should be interpreted by the claims below, and all technical spirits within the scope equivalent thereto should be interpreted as being included in the scope of the present invention.

10: Voice Assistant server
1000: smart device
1020: communication module
1040: memory
1060: controller
1100: housing
1106: indicator
1120: lower frame
1140: upper frame
1160: transparent window
1200: voice input module
1202: microphone
1204: microphone array
1300: audio output module
1302: speaker
1400: display module
1420: display panel
1440: projector
1500: drive module
1520: rotary motor
1540: rotating plate
1560: Direction sensor
1600: power module
1700: User position detection module
1720: (mono / stereo) camera
1800: heat dissipation module

Claims (15)

A wake-up word detection state for determining whether a sound signal received from the outside includes a wake-up word, and a listening state for recognizing voice commands included in the sound signal ( A media content control method performed by a smart device having an operating state including a listening state,
Play the media content related to the first voice command input by the user, output audio data of the media content through the audio output module of the smart device, and video data of the media content through the video output module of the smart device Playing the media content by displaying;
Entering the wakeup word detection state while maintaining the playback of the media content after starting the playback of the media content;
Determining whether the wakeup word is included in a first sound signal received during playback of the media content by the smart device in the wakeup word detection state;
If the wakeup word is included in the first sound signal, entering the listening state while maintaining playback of the media content;
When entering the listening state, performing a first operation related to the media content;
Outputting a talk-back related to a second voice command included in the received second sound signal after entering the listening state; and
And after the output of the talk-back ends, performing a second operation related to the media content.
The performing of the first operation may include adjusting the volume of the audio data by reducing or removing the volume of the audio data, and text data corresponding to the volume-adjusted audio data of the media content. Displaying with the media content's video data,
The step of performing the second operation includes adjusting the volume of the audio data of the media content before performing the first operation and ending the display of the text data.
Control method of smart devices.
delete According to claim 1,
The outputting of the talk-back includes outputting feedback related to the second voice command.
Control method of smart devices.
delete According to claim 1,
The first text data displayed among the text data includes text data corresponding to audio data at a time when the volume is adjusted and text data corresponding to audio data before a predetermined time from a time when the volume is adjusted.
Control method of smart devices.
According to claim 1,
The text data that is displayed last among the text data includes text data corresponding to audio data at the time when the volume returns and text data corresponding to audio data after a predetermined time from the time when the volume returns.
Control method of smart devices.
A wake-up word detection state for determining whether a sound signal received from the outside includes a wake-up word, and a listening state for recognizing voice commands included in the sound signal ( A media content control method performed by a smart device having an operating state including a listening state,
Play the media content related to the first voice command input by the user, output audio data of the media content through the audio output module of the smart device, and video data of the media content through the video output module of the smart device Playing the media content by displaying;
Entering the wakeup word detection state while maintaining the playback of the media content after starting the playback of the media content;
Determining whether the wakeup word is included in a first sound signal received during playback of the media content by the smart device in the wakeup word detection state;
If the wakeup word is included in the first sound signal, entering the listening state while maintaining playback of the media content; And
When entering the listening state, depending on the type of the media content, performing the operation of any one of the pause of the playback of the media content or volume control of the audio data of the media content;
If the media content is real-time streaming content, adjust the volume of the audio data of the media content,
And if the media content is not real-time content, pausing the media content or performing an operation of reducing the volume of audio data of the media content.
Control method of smart devices.
delete The method of claim 7,
And when adjusting the volume of the audio data, displaying text data corresponding to the volume-adjusted audio data of the media content together with video data of the media content.
Control method of smart devices.
A wake-up word detection state for determining whether a sound signal received from the outside includes a wake-up word, and a listening state for recognizing voice commands included in the sound signal ( Listening state) as a smart device having an operating state,
A voice input module for receiving a sound signal;
A voice output module for outputting voice;
An image output module for displaying an image; And
Play the media content related to the first voice command input by the user, output audio data of the media content through the audio output module of the smart device, and video data of the media content through the video output module of the smart device Play the media content by displaying, enter the wake-up word detection state while maintaining the playback of the media content after starting the playback of the media content, and the smart device in the wake-up word detection state displays the media It is determined whether the wake-up word is included in the first sound signal received through the voice input module during playback of the content, and if the wake-up word is included in the first sound signal, the media content is maintained. While listening When entering the state and entering the listening state, a first operation related to the media content is performed, and after entering the listening state, a talk-back related to a second voice command included in the received second sound signal And a controller that outputs and performs a second operation related to the media content after the output of the talk-back ends.
The controller adjusts the volume of the audio data by reducing or removing the volume of the audio data, and displays text data corresponding to the volume adjusted audio data of the media content together with video data of the media content. By performing the first operation,
The controller performs the second operation by adjusting the volume of the audio data of the media content before performing the first operation and ending the display of the text data,
Smart device.
delete The method of claim 10,
When outputting the talk-back, the controller outputs feedback related to the second voice command.
Smart device.
delete The method of claim 10,
The first text data displayed among the text data includes text data corresponding to audio data at a time when the volume is adjusted and text data corresponding to audio data before a predetermined time from a time when the volume is adjusted.
Smart device.
The method of claim 10,
The text data that is displayed last among the text data includes text data corresponding to audio data at the time when the volume returns and text data corresponding to audio data after a predetermined time from the time when the volume returns.
Smart device.

Images