KR102062524B1 - Voice recognition and translation method and, apparatus and server therefor - Google Patents

Abstract

According to embodiments of the present invention, the present invention relates to a voice recognition and translation method and a terminal device and a server therefor, and more particularly, to a voice recognition and translation method using machine learning. The voice recognition and translation method of the present invention, which recognizes and translates voice mixed with a dialect, comprises the steps of: obtaining a first machine learning model which learns a relationship between first language dialect voices and first language standard texts on the basis of a plurality of first language dialect voices and a plurality of first language standard texts corresponding to the first language dialect voices; obtaining a second machine learning model which learns a relationship between first language dialect texts and the first language standard texts on the basis of a plurality of first language dialect texts and the first language standard texts corresponding to the first language dialect texts; receiving a first dialect input voice of a first language and a second language standard text corresponding to the first dialect input voice of the first language; obtaining a first standard input text of the first language by applying the first dialect input voice of the first language to the first machine learning model; converting the first dialect input voice of the first language into a first dialect input text of the first language; obtaining a second standard input text of the first language by applying the first dialect input text of the first language to the second machine learning model; and obtaining a third machine learning model by performing machine learning based on the first standard input text of the first language, the second standard input text of the first language, and the second language standard text.

Description

VOICE RECOGNITION AND TRANSLATION METHOD AND, APPARATUS AND SERVER THEREFOR}

Embodiments of the present invention relate to a voice recognition and translation method, and a terminal apparatus and a server therefor, and more particularly, to a translation method for recognizing speech using machine learning and a terminal apparatus and a server therefor.

Recently, voice translation has been activated online from one's own language to another's language. However, the language has different attributes, which brings difficulties. In particular, it is more difficult to translate dialects in different languages, such as Korean and Chinese, into other languages.

For example, in the case of Korean, the dialects and phonemes that are unique in Gyeonggi-do, Jeolla-do, Gyeongsang-do, Gangwon-do, and Jeju-do are different from those of the standard Korean. Voice recognition is not easy.

A method for recognizing and translating a dialect of a mixed dialect of the present disclosure is based on a standard language text of a plurality of first languages corresponding to a dialect voice of a plurality of first languages and a dialect voice of a plurality of first languages. Acquiring a first machine learning model for learning a relationship between a dialect of a speech and standard language text of a first language, a plurality of first languages corresponding to a dialect text of a plurality of first languages and a dialect text of a plurality of first languages Acquiring a second machine learning model learning the relationship between the dialect text of the first language and the standard language text of the first language based on the standard language text of the first language, the first dialect input speech of the first language and the first language of the first language; Receiving a standard language text of a second language corresponding to the dialect input speech, applying the first dialect input speech of the first language to the first machine learning model 1 obtaining a standard language input text, converting a first dialect input voice of a first language into a first dialect input text of a first language, and applying the first dialect input text of a first language to a second machine learning model Acquiring a second standard language input text of the first language, performing machine learning based on the first standard language input text of the first language, the second standard language input text of the first language, and the standard language text of the second language 3 acquiring a machine learning model.

The method for recognizing and translating a dialect of a mixed dialect of the present disclosure may include receiving a second dialect input voice in a first language, and applying a second dialect input voice in a first language to the first machine learning model. Obtaining a third standard language input text of the language, converting the second dialect input voice of the first language into a second dialect input text of the first language, and converting the second dialect input text of the first language to the second machine learning Obtaining a fourth standard language input text of the first language by applying to the model, applying the third standard language input text of the first language and the fourth standard language input text of the first language to the third machine learning model Obtaining a standard language input text, and generating an output speech of the second language based on the standard language input text and the text-to-speech model of the second language.

The method for recognizing and translating a dialect of a mixed dialect of the present disclosure may include analyzing at least one hidden state of a first machine learning model and a second machine learning model to determine whether the second dialect input voice of the first language is dialect. Obtaining information indicative of, wherein the text-to-speech model generates an output speech of a second language that includes any dialect accent of the second language, if the information represents a dialect, and text-to-speech The model includes generating an output speech of the second language that includes the standard language accent of the second language if the information represents the standard language.

A method for recognizing and translating a mixed speech of the present disclosure includes receiving accuracy information for an output voice of a second language from a user, and accuracy information, third standard language input text of the first language, and a first language Updating the third machine learning model based on the fourth standard word input text of.

An apparatus for recognizing and translating a mixed dialect of the present disclosure includes a processor and a memory, the processor comprising a dialect voice of a plurality of first languages and a dialect voice of a plurality of first languages according to instructions included in the memory. Acquiring a first machine learning model that learns a relationship between a dialect voice of a first language and a standard language text of a first language based on the standard language texts of a plurality of first languages corresponding to And obtaining a second machine learning model that learns a relationship between the dialect text of the first language and the standard language text of the first language based on the standard language texts of the first language corresponding to the dialect texts of the first language. Receiving standard language text of a second language corresponding to a first dialect input voice of the first language and a first dialect input voice of the first language, and Applying a first dialect input speech in one language to the first machine learning model to obtain a first standard language input text in the first language, wherein the first dialect input speech in the first language is in the first dialect input text in the first language Converting to a second language, applying a first dialect input text of a first language to a second machine learning model, and obtaining a second standard language input text of a first language, a first standard language input text of a first language, and a first language Performing machine learning based on the second standard language input text and the standard language text of the second language to obtain a third machine learning model.

The processor of the apparatus for recognizing and translating a mixed dialect of the present disclosure may include receiving a second dialect input voice in a first language according to a command included in a memory, and generating a second dialect input voice in a first language. 1 obtaining a third standard language input text of a first language by applying to a machine learning model, converting a second dialect input voice of a first language into a second dialect input text of a first language, a first language of a first language 2 applying the dialect input text to the second machine learning model to obtain a fourth standard language input text of the first language, a third standard language input text of the first language, and a fourth standard language input text of the first language to the third machine Obtaining a standard language input text of the second language by applying to the learning model, and outputting the second language based on the standard language input text and the text-to-speech model of the second language; It performs the step of generating speech.

The processor of the apparatus for recognizing and translating a mixed dialect of the present disclosure analyzes a hidden state of at least one of the first machine learning model and the second machine learning model according to an instruction included in a memory, Obtaining information indicating whether the second dialect input speech is dialect, the text-to-speech model generates an output speech in a second language that includes any dialect accent in the second language when the information indicates dialect And a text-to-speech model, when the information represents a standard language, generates an output speech of a second language that includes a standard language accent of the second language.

A processor of an apparatus for recognizing and translating a mixed speech of the present disclosure may include receiving, from a user, accuracy information about an output voice of a second language, according to instructions included in a memory, and accuracy information, the accuracy of the first language. Updating the third machine learning model based on the third standard language input text and the fourth standard language input text of the first language.

In addition, a program for implementing a method for recognizing and translating a speech having a mixed dialect as described above may be recorded in a computer-readable recording medium.

1 is a block diagram of a voice translation apparatus 100 according to an embodiment of the present disclosure.
2 is a diagram illustrating a voice translation system according to an embodiment of the present disclosure.
3 is a flowchart illustrating a voice translation method according to an embodiment of the present disclosure.
4 illustrates a data learner included in a voice translation device according to an embodiment of the present disclosure.
5 illustrates a data learner included in a voice translation device according to an embodiment of the present disclosure.
6 illustrates a data learner included in a voice translation device according to an embodiment of the present disclosure.
7 is a flowchart illustrating a voice translation method according to an embodiment of the present disclosure.
8 is a diagram illustrating a screen of a user terminal according to an exemplary embodiment of the present disclosure.

Advantages and features of the disclosed embodiments, and methods of achieving them will be apparent with reference to the embodiments described below in conjunction with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below, but may be implemented in various forms, and the present embodiments are merely provided to make the present disclosure complete, and those of ordinary skill in the art to which the present disclosure belongs. It is merely provided to fully inform the scope of the invention.

Terms used herein will be briefly described, and the disclosed embodiments will be described in detail.

The terminology used herein has been selected among general terms that are currently widely used while considering the functions of the present disclosure, but may vary according to the intention or precedent of a person skilled in the relevant field, the emergence of a new technology, and the like. In addition, in certain cases, there is also a term arbitrarily selected by the applicant, in which case the meaning will be described in detail in the description of the invention. Therefore, the terms used in the present disclosure should be defined based on the meanings of the terms and the contents throughout the present disclosure, rather than simply the names of the terms.

A singular expression in this specification includes a plural expression unless the context clearly indicates that it is singular. Also, the plural expressions include the singular expressions unless the context clearly indicates the plural.

When any part of the specification is to "include" any component, this means that it may further include other components, except to exclude other components unless otherwise stated.

Also, as used herein, the term "part" means a software or hardware component, and "part" plays certain roles. However, "part" is not meant to be limited to software or hardware. The “unit” may be configured to be in an addressable storage medium and may be configured to play one or more processors. Thus, as an example, a "part" refers to components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, procedures, Subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays and variables. The functionality provided within the components and "parts" may be combined into a smaller number of components and "parts" or further separated into additional components and "parts".

According to an embodiment of the present disclosure, the “unit” may be implemented with a processor and a memory. The term “processor” should be interpreted broadly to include general purpose processors, central processing units (CPUs), microprocessors, digital signal processors (DSPs), controllers, microcontrollers, state machines, and the like. In some circumstances, a “processor” may refer to an application specific semiconductor (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), or the like. The term "processor" refers to a combination of processing devices such as, for example, a combination of a DSP and a microprocessor, a combination of a plurality of microprocessors, a combination of one or more microprocessors in conjunction with a DSP core, or a combination of any other such configuration. May be referred to.

The term "memory" should be interpreted broadly to include any electronic component capable of storing electronic information. The term memory refers to random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erase-programmable read-only memory (EPROM), electrical May also refer to various types of processor-readable media, such as erasable PROM (EEPROM), flash memory, magnetic or optical data storage, registers, and the like. If the processor can read information from and / or write information to the memory, the memory is said to be in electronic communication with the processor. The memory integrated in the processor is in electronic communication with the processor.

DETAILED DESCRIPTION Hereinafter, exemplary embodiments will be described in detail with reference to the accompanying drawings so that those skilled in the art may easily implement the embodiments. In the drawings, parts irrelevant to the description are omitted in order to clearly describe the present disclosure.

1 is a block diagram of a voice translation apparatus 100 according to an embodiment of the present disclosure.

Referring to FIG. 1, the voice translation apparatus 100 may include a data learner 110 and a data recognizer 120. The voice translation apparatus 100 as described above may include a processor and a memory.

The data learner 110 may receive the first data and the second data to obtain a machine learning model for the relationship between the first data and the second data. The machine learning model acquired by the data learner 110 may be a model for generating second data using the first data. For example, it is possible to learn the relationship between a dialect of a plurality of first languages and standard language text of a plurality of first languages. In addition, the data learning unit 110 may learn the relationship between the dialect text of the plurality of first languages and the standard language text of the plurality of first languages. The data learning unit 110 may learn a criterion regarding which standard language text is output according to the dialect voice or text.

The data recognition unit 120 may output the standard word text by applying the received dialect or text to the machine learning model. The machine learning model may be information about a predetermined criterion for obtaining the standard language text based on the dialect speech or text. In addition, the data recognition unit 120 may use the result output by the received dialect or text and the machine learning model to update the machine learning model.

At least one of the data learner 110 and the data recognizer 120 may be manufactured in the form of at least one hardware chip and mounted on the electronic device. For example, at least one of the data learner 110 and the data recognizer 120 may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or a conventional general purpose processor (eg, a CPU). Alternatively, the electronic device may be manufactured as a part of an application processor or a graphics dedicated processor (eg, a GPU) and mounted on various electronic devices.

In addition, the data learner 110 and the data recognizer 120 may be mounted in separate electronic devices, respectively. For example, one of the data learner 110 and the data recognizer 120 may be included in the electronic device, and the other may be included in the server. In addition, the data learner 110 and the data recognizer 120 may provide the model information constructed by the data learner 110 to the data recognizer 120 through a wired or wireless connection. The data inputted to 120 may be provided to the data learner 110 as additional learning data.

Meanwhile, at least one of the data learner 110 and the data recognizer 120 may be implemented as a software module. When at least one of the data learner 110 and the data recognizer 120 is implemented as a software module (or a program module including instructions), the software module may be a memory or computer readable non-readable. It may be stored in a non-transitory computer readable media. In this case, at least one software module may be provided by an operating system (OS) or by a predetermined application. Alternatively, some of the at least one software module may be provided by an operating system (OS), and others may be provided by a predetermined application.

The data learner 110 according to an exemplary embodiment of the present disclosure may include a data acquirer 111, a preprocessor 112, a training data selector 113, a model learner 114, and a model evaluator 115. It may include.

The data acquirer 111 may acquire data necessary for machine learning. Since a lot of data is required for learning, the data obtaining unit 111 may receive a plurality of dialect or text and standard language text corresponding to the plurality of dialect or text.

The preprocessor 112 may preprocess the obtained data so that the received data may be used for machine learning. The preprocessor 112 may process the acquired data into a preset format so that the model learner 114, which will be described later, may use. The preprocessor 112 may separate the text and the voice into at least one unit of phoneme, syllable, morpheme, word, word, or sentence.

In addition, the preprocessor 112 may obtain unit embedding by analyzing text and voice for each unit. That is, the preprocessor 112 may acquire embedding of at least one of a phoneme, a syllable, a morpheme, a word, a word, or a sentence. Embedding can represent representing the data as a unique vector. For example, morpheme embedding may mean that each morpheme is represented by a unique vector. The preprocessor 112 may use a distributed memory (DM) or distributed bag of words (DBOW) algorithm to obtain embedding.

The training data selector 113 may select data necessary for learning from the preprocessed data. The selected data may be provided to the model learner 114. The training data selector 113 may select data required for learning from preprocessed data according to a preset criterion. In addition, the training data selector 113 may select data according to preset criteria by learning by the model learner 114 to be described later.

The model learner 114 may learn a criterion about which standard language text to output according to the dialect voice or text based on the training data. In addition, the model learning unit 114 may learn by using a learning model that outputs standard language text according to the dialect voice or text as the training data. In this case, the data learning model may be a previously built model. For example, the data learning model may be a model built in advance by receiving basic training data (eg, sample dialect text, voice and standard language text, etc.).

The data learning model may be constructed in consideration of the application field of the learning model, the purpose of learning, or the computer performance of the device. The data learning model may be, for example, a model based on a neural network. For example, models such as Deep Neural Network (DNN), Recurrent Neural Network (RNN), Long Short-Term Memory models (LSTM), Bidirectional Recurrent Deep Neural Network (BRDNN), and Convolutional Neural Networks (CNN) can be used as data learning models. But it is not limited thereto.

According to various embodiments of the present disclosure, when there are a plurality of pre-built data learning models, the model learning unit 114 may determine a data learning model having a high correlation between the input learning data and the basic learning data as a data learning model to be trained. have. In this case, the basic training data may be previously classified by data type, and the data learning model may be pre-built for each data type. For example, the basic training data is classified based on various criteria such as the region where the training data is generated, the time at which the training data is generated, the size of the training data, the genre of the training data, the creator of the training data, and the types of objects in the training data. It may be.

In addition, the model learner 114 may train the data learning model using, for example, a learning algorithm including an error back-propagation method or a gradient descent method.

In addition, the model learner 114 may learn the data learning model through, for example, supervised learning using the learning data as an input value. In addition, the model learning unit 114 learns data through unsupervised learning that finds a criterion for situation determination by, for example, self-learning a type of data necessary for situation determination without any guidance. You can train the model. In addition, the model learner 114 may learn the data learning model through, for example, reinforcement learning using feedback on whether the result of the situation determination according to the learning is correct.

In addition, when the data learning model is trained, the model learner 114 may store the learned data learning model. In this case, the model learner 114 may store the learned data learning model in a memory of the electronic device including the data recognizer 120. Alternatively, the model learner 114 may store the learned data learning model in a memory of a server connected to the electronic device through a wired or wireless network.

In this case, the memory in which the learned data learning model is stored may store, for example, commands or data related to at least one other element of the electronic device. The memory may also store software and / or programs. The program may include, for example, a kernel, middleware, an application programming interface (API) and / or an application program (or “application”), and the like.

The model evaluator 115 may input the evaluation data into the data learning model, and if the result output from the evaluation data does not satisfy a predetermined criterion, the model evaluator 114 may relearn. In this case, the evaluation data may be preset data for evaluating the data learning model.

For example, the model evaluator 115 may not satisfy a predetermined criterion when the number or ratio of the evaluation data whose recognition result is not accurate among the results of the learned data learning model for the evaluation data exceeds a preset threshold. It can be evaluated as. For example, when a predetermined criterion is defined as a ratio of 2%, when the trained data learning model outputs an incorrect recognition result for more than 20 evaluation data out of a total of 1000 evaluation data, the model evaluation unit 115 learns. The data learning model can be evaluated as not suitable.

On the other hand, when there are a plurality of trained data learning models, the model evaluator 115 evaluates whether each learned video learning model satisfies a predetermined criterion, and uses the model satisfying the predetermined criterion as the final data learning model. You can decide. In this case, when there are a plurality of models satisfying a predetermined criterion, the model evaluator 115 may determine any one or a predetermined number of models that are preset in the order of the highest evaluation score as the final data learning model.

At least one of the data acquirer 111, the preprocessor 112, the training data selector 113, the model learner 114, and the model evaluator 115 in the data learner 110 is at least one. May be manufactured in the form of a hardware chip and mounted on an electronic device. For example, at least one of the data acquirer 111, the preprocessor 112, the training data selector 113, the model learner 114, and the model evaluator 115 may be artificial intelligence (AI). It may be manufactured in the form of a dedicated hardware chip, or may be manufactured as a part of an existing general purpose processor (eg, a CPU or an application processor) or a graphics dedicated processor (eg, a GPU) and mounted on the aforementioned various electronic devices.

In addition, the data acquirer 111, the preprocessor 112, the training data selector 113, the model learner 114, and the model evaluator 115 may be mounted in one electronic device or may be separate. Each of the electronic devices may be mounted on the electronic device. For example, some of the data acquirer 111, the preprocessor 112, the training data selector 113, the model learner 114, and the model evaluator 115 are included in the electronic device, and the rest of the data is included in the electronic device. Can be included on the server.

In addition, at least one of the data acquirer 111, the preprocessor 112, the training data selector 113, the model learner 114, and the model evaluator 115 may be implemented as a software module. A program in which at least one of the data acquirer 111, the preprocessor 112, the training data selector 113, the model learner 114, and the model evaluator 115 includes a software module (or instruction). Module may be stored on a computer readable non-transitory computer readable media. In this case, at least one software module may be provided by an operating system (OS) or by a predetermined application. Alternatively, some of the at least one software module may be provided by an operating system (OS), and others may be provided by a predetermined application.

The data recognizer 120 according to an exemplary embodiment of the present disclosure may include a data acquirer 121, a preprocessor 122, a recognition data selector 123, a recognition result provider 124, and a model updater 125. It may include.

The data acquirer 121 may acquire dialect text or speech necessary to generate standard word text. The preprocessor 122 may preprocess the acquired data so that the acquired dialect or text may be used. The preprocessing unit 122 may process the acquired data into a preset format so that the recognition result providing unit 124, which will be described later, may use the acquired data to output voice or text.

The recognition data selector 123 may select necessary data from the preprocessed data. The selected data may be provided to the recognition result provider 124. The recognition data selector 123 may select some or all of the preprocessed data according to a preset criterion. In addition, the recognition data selector 123 may select data according to a predetermined criterion by learning by the model learner 114.

The recognition result provider 124 may output the voice or text by applying the selected data to the data learning model. The recognition result provider 124 may apply the selected data to the data learning model by using the data selected by the recognition data selector 123 as an input value. In addition, the recognition result may be determined by the data learning model.

The model updater 125 may cause the data learning model to be updated based on the evaluation of the recognition result provided by the recognition result provider 124. For example, the model updater 125 may provide the model learning unit 114 with the recognition result provided by the recognition result providing unit 124 so that the model learning unit 114 may update the data learning model. have.

Meanwhile, at least one of the data acquisition unit 121, the preprocessing unit 122, the recognition data selection unit 123, the recognition result providing unit 124, and the model updating unit 125 in the data recognition unit 120 is at least It may be manufactured in the form of one hardware chip and mounted on an electronic device. For example, at least one of the data acquirer 121, the preprocessor 122, the recognition data selector 123, the recognition result provider 124, and the model updater 125 may be artificial intelligence (AI). ) May be manufactured in the form of a dedicated hardware chip, or may be manufactured as a part of an existing general purpose processor (eg, a CPU or an application processor) or a graphics dedicated processor (eg, a GPU) and mounted on the aforementioned various electronic devices.

In addition, the data acquirer 121, the preprocessor 122, the recognition data selector 123, the recognition result provider 124, and the model updater 125 may be mounted in one electronic device or may be separate. May be mounted on the electronic devices. For example, some of the data obtaining unit 121, the preprocessor 122, the recognition data selecting unit 123, the recognition result providing unit 124, and the model updating unit 125 are included in the electronic device, and some of the remaining portions are included in the electronic device. May be included in the server.

In addition, at least one of the data acquirer 121, the preprocessor 122, the recognition data selector 123, the recognition result provider 124, and the model updater 125 may be implemented as a software module. At least one of the data obtaining unit 121, the preprocessor 122, the recognition data selecting unit 123, the recognition result providing unit 124, and the model updating unit 125 may include a software module (or instruction). If implemented as a program module, the software module may be stored in a computer readable non-transitory computer readable media. In this case, at least one software module may be provided by an operating system (OS) or by a predetermined application. Alternatively, some of the at least one software module may be provided by an operating system (OS), and others may be provided by a predetermined application.

2 is a diagram illustrating a voice translation system according to an embodiment of the present disclosure.

The voice translation system 200 may include an input unit 210, a voice translation device 220, and an output unit 230. The input unit 210 may be included in the user terminal. The input unit 210 may receive voice or text of a first language. The input unit 210 may transmit the received voice or text of the first language to the voice translation apparatus 220 through wire or wirelessly. The input unit 210 may compress the voice or text of the first language based on a predetermined algorithm in order to reduce the volume of data.

The voice translation apparatus 220 may convert the received voice or text of the first language into voice or text of the second language. The voice translation apparatus 220 may correspond to the voice translation apparatus 100 of FIG. 1. The voice translation apparatus 220 may correspond to a user terminal or a server. The voice translation apparatus 220 may include a data learner 110 and a data recognizer 120. The voice translation apparatus 220 may convert voice or text of a first language into voice or text of a second language using a machine learning model.

The voice translation apparatus 220 may transmit the generated voice or text of the second language to the output unit 230 via wire or wirelessly. The voice translation apparatus 220 may compress the generated voice or text of the second language by using a predetermined algorithm to reduce the volume of data required for data transmission and to speed up the transmission.

The output unit 230 may correspond to a user terminal. The output unit 230 may output text or voice translated in a second language. The output unit 230 may display the text translated in the second language on the display unit. Alternatively, the output unit 230 may output the voice translated in the second language through the speaker.

3 is a flowchart illustrating a voice translation method according to an embodiment of the present disclosure.

The speech translation device 220 may use the dialect of the first language and the standard language of the first language based on the dialect of the first language and the standard language text of the first language corresponding to the dialect of the first language. An operation 310 may be performed to obtain a first machine learning model that learns the relationship of text. The speech translation device 220 may use the dialect text of the first language and the dialect text of the first language based on the dialect text of the plurality of first languages and the dialect text of the first language corresponding to the dialect text of the plurality of first languages. An operation 320 may be performed to obtain a second machine learning model that learns the relationship of text. The voice translation apparatus 220 may perform an operation 330 of receiving a standard dialect text of a second language corresponding to the first dialect input voice of the first language and the first dialect input voice of the first language. The speech translation apparatus 220 may perform an operation 340 of obtaining a first standard language input text of the first language by applying the first dialect input speech of the first language to the first machine learning model. The voice translation apparatus 220 may perform a step 350 of converting the first dialect input voice of the first language into the first dialect input text of the first language. The voice translation apparatus 220 may perform the operation 360 of obtaining the second standard language input text of the first language by applying the first dialect input text of the first language to the second machine learning model. The voice translation apparatus 220 performs a machine learning based on the first standard language input text of the first language, the second standard language input text of the first language, and the standard language text of the second language to obtain a third machine learning model. 370 may be performed. Each step will be described in detail with reference to FIGS. 4 to 6.

4 illustrates a data learner included in a voice translation device according to an embodiment of the present disclosure.

The data learning unit 410 included in the voice translation apparatus 220 may generate a first language based on standard language texts of a plurality of first languages corresponding to a dialect of a plurality of first languages and a dialect of a plurality of first languages. An operation 310 may be performed to obtain a first machine learning model that learns a relation between a dialect of a dialect and a standard language text of a first language. The first machine learning model may be a previously learned machine learning model.

The data learner 410 included in the voice translation apparatus 220 may receive a dialect 421 of a plurality of first languages and standard language text 422 of a plurality of first languages. The data learner 410 may divide the dialect 421 of the first language or the standard language text 422 of the first language into at least one of a phoneme, a syllable, a morpheme, a word, a word, or a sentence. The voice translation apparatus 220 may acquire the first machine learning model by learning a relationship between the dialect 421 of the first language and the standard language text 422 of the first language.

The dialect voice of the first language may be a dialect of a plurality of regions. For example, if the first language is Korean, the dialect may include dialects of Seoul, Gyeonggi-do, Chungcheong-do, Gyeongsang-do, Jeolla-do, Gangwon-do, and Jeju-do. In addition, the dialect voice of the first language may be voice data obtained by recording text read by a speaker of each region.

Also, the standard language text of the first language corresponding to the dialect sounds of the plurality of first languages may be one local dialect. That is, when the first language is Korean, the standard language text of the first language may be text in modern Seoul that is written by educated people. The standard language text of the plurality of first languages may include at least one letter.

The dialect voice of the plurality of first languages may include a plurality of sentences. Also, the standard language text of the plurality of first languages corresponding to the dialect sounds of the plurality of first languages may include a plurality of sentences. The voice translation apparatus 220 may acquire dialect sounds of a plurality of first languages and standard language texts of a plurality of first languages in a TV broadcast or a movie. For example, a television broadcast or a movie may have subtitles in a standard language even though the voice is in dialect. The voice translation apparatus 220 may receive a dialect of a first language and standard language text of a plurality of first languages from a TV broadcast or a movie and perform machine learning.

The voice translation apparatus 220 may acquire the first machine learning model 431 as a machine learning result. The voice translation apparatus 220 may store the first machine learning model 431 in a memory. The voice translation apparatus 220 may transmit the first machine learning model 431 to another voice translation apparatus 220. The speech translation apparatus 220 may translate the dialect of the first language into standard language text of the first language based on the first machine learning model 431. Also, when the voice translation apparatus 220 receives the standard language voice of the first language, the voice translation apparatus 220 may generate the standard language text of the first language based on the first machine learning model 431. The voice translation apparatus 220 may acquire the first machine learning model 431 in advance in order to generate the third machine learning model.

5 illustrates a data learner included in a voice translation device according to an embodiment of the present disclosure.

The data learning unit 510 included in the voice translation apparatus 220 includes the dialect text 521 of the first language and the standard language text 522 of the first language corresponding to the dialect speech of the first language. An operation 310 may be performed to obtain a second machine learning model 531 that learns the relationship between the dialect text 521 of the first language and the standard language text 522 of the first language. The second machine learning model may be a machine learning model learned in advance.

The data learner 510 included in the voice translation apparatus 220 may receive the dialect text 521 of the first language and the standard language text 522 of the first language. The data learner 510 may divide the dialect text 521 of the first language or the standard language text 522 of the first language into at least one of a phoneme, a syllable, a morpheme, a word, a word, or a sentence. The voice translation apparatus 220 may acquire a second machine learning model by learning a relationship between the dialect text 521 of the first language and the standard language text 522 of the first language.

The dialect text of the first language may be dialect of a plurality of regions. For example, if the first language is Korean, the dialect may include dialects of Seoul, Gyeonggi-do, Chungcheong-do, Gyeongsang-do, Jeolla-do, Gangwon-do, and Jeju-do. In addition, the standard language text of the first language corresponding to the dialect texts of the plurality of first languages may be a language used by specific people in a specific region. That is, when the first language is Korean, the standard language text of the first language may be text in modern Seoul that is written by educated people.

The dialect text of the plurality of first languages may include a plurality of words or a plurality of sentences. Also, the standard language text of the plurality of first languages corresponding to the dialect texts of the plurality of first languages may include a plurality of words or a plurality of sentences. The user may obtain a second machine learning model 531 by inputting a speech, a dictionary, or various materials uploaded on the Internet into the voice translation apparatus 220.

The voice translation apparatus 220 may acquire the second machine learning model 531 as a machine learning result. The voice translation apparatus 220 may store the second machine learning model 531 in a memory. The voice translation apparatus 220 may transmit the second machine learning model 531 to another voice translation apparatus 220. The speech translation device 220 may translate the dialect text of the first language into standard language text of the first language based on the second machine learning model 531. Also, when the voice translation apparatus 220 receives the standard language text of the first language, the voice translation apparatus 220 may generate the standard language text of the same or the same language based on the first machine learning model 531. The voice translation apparatus 220 may acquire the second machine learning model 531 in advance in order to generate the third machine learning model.

When the speakers of the dialect generate voice data by recording a voice that reads the same text, each voice data may reflect different voice characteristics. The voice feature may include different utterance sections, silent sections, voice level and pronunciation stress. Since the first machine learning model learns the relationship between the dialect of the first language and the standard language text of the first language, the first machine learning model is based on the phonetic features included in the dialect speech of the first language. Can generate standard word text. However, dialect speakers will pronounce the same text differently. For example, a speaker in a particular area may pronounce "ㅆ" similarly to "ㅅ". Also, speakers of dialects can express the same content in different words. Speakers in certain areas, for example, call "leek" a "pot." If based only on the dialect of the first language, there is a possibility that the generated standard language text of the first language is incorrect.

Therefore, the voice translation apparatus 220 may further use a second machine learning model that learns the relationship between the dialect text of the first language and the standard language text of the first language. That is, the speech translation apparatus 220 may generate standard language text of the first language based on not only the speech feature of the speaker who speaks the dialect but also the text using the second machine learning model.

Hereinafter, a process of obtaining a third machine learning model together with FIG. 6 will be described.

6 illustrates a data learner included in a voice translation device according to an embodiment of the present disclosure.

The voice translation apparatus 220 may perform an operation 330 of receiving a standard language text 623 of a second language corresponding to the first dialect input voice of the first language and the first dialect input voice of the first language. . The speech translation device 220 includes phonemes, syllables, morphemes, words, phrases, or sentences of the standard language text 623 of the second language corresponding to the first dialect input voice of the first language and the first dialect input voice of the first language. Can be divided into at least one of

The first dialect input voice of the first language may be voice data obtained by the voice translation apparatus 220 from a microphone or a memory. The first dialect input voice of the first language may be voice data received by the voice translation apparatus 220 from an external device such as a user terminal.

The standard language text 623 of the second language may be text having the same meaning as the first dialect input voice of the first language. The standard language text 623 of the second language may be at least one word or at least one sentence.

The speech translation apparatus 220 may perform an operation 340 of obtaining a first standard language input text 621 of the first language by applying the first dialect input speech of the first language to the first machine learning model. As already explained, the first machine learning model is a model that learns the relationship between the dialect speech of the first language and the standard language text of the first language. Step 340 may be a test stage of the first machine learning model.

The voice translation apparatus 220 may perform a step 350 of converting the first dialect input voice of the first language into the first dialect input text of the first language. In order to convert the first dialect input speech of the first language into the first dialect input text of the first language, it may be based on a speech-to-text model. The speech-to-text model can be implemented by various algorithms. For example, the speech-to-text model may be a model that has been previously machined based on the plurality of voices and the plurality of texts.

The speech translation apparatus 220 may perform the operation 360 of obtaining the second standard language input text 622 of the first language by applying the first dialect input text of the first language to the second machine learning model. As described above, the second machine learning model may be a model that learns the relationship between the dialect text of the first language and the standard language text of the first language. Step 360 may be a test stage of the second machine learning model.

The voice translation apparatus 220 performs machine learning based on the first standard language input text 621 of the first language, the second standard language input text 622 of the first language, and the standard language text 623 of the second language. An operation 370 of obtaining a third machine learning model may be performed. Step 370 may be a learning stage of the third machine learning model 631. The third machine learning model 631 may be a machine learning model for generating a second standard language input text based on the first standard language input text of the first language and the second standard language input text of the first language. The speech translation apparatus 220 may use an attention algorithm to obtain a third machine learning model. The attention algorithm will be described below in conjunction with FIG. 7.

The voice translation apparatus 220 may store the third machine learning model 631 in a memory. Alternatively, the voice translation apparatus 220 may transmit the third machine learning model 631 to another voice translation apparatus 220. The voice translation apparatus 220 may accurately translate the voice of the first language into the text of the second language based on the third machine learning model 631. The speech translation apparatus 220 may repeatedly perform the steps 330 to 370 to increase the accuracy of the third machine learning model 631.

7 is a flowchart illustrating a voice translation method according to an embodiment of the present disclosure.

7 may represent a test stage of a third machine learning model of the voice translation apparatus 220.

The voice translation apparatus 220 may perform a step 710 of receiving a second dialect input voice of a first language. The voice translation apparatus 220 may directly receive a second dialect input voice of a first language from an input unit such as a speaker. In addition, the voice translation apparatus 220 may receive voice data stored in a memory or voice data from a user terminal.

The speech translation apparatus 220 may perform an operation 720 of obtaining a third standard language input text of the first language by applying the second dialect input speech of the first language to the first machine learning model. In addition, the voice translation apparatus 220 may perform step 730 of converting the second dialect input voice of the first language into the second dialect input text of the first language. In addition, the voice translation apparatus 220 may perform a step 740 of applying a second dialect input text of the first language to the second machine learning model to obtain a fourth standard language input text of the first language. Steps 720 to 740 of FIG. 7 may be the same as steps 340 to 360 of FIG. 3.

In operation 750, the voice translation apparatus 220 obtains the standard language input text of the second language by applying the third standard language input text of the first language and the fourth standard language input text of the first language to the third machine learning model. Can be done.

The speech translation apparatus 220 may generate the standard language input text of the second language in at least one unit of phoneme, syllable, morpheme, word, word, or sentence based on the third machine learning model. That is, the voice translation apparatus 220 may generate some of the standard language input text of the next second language by applying some of the generated standard language input text of the second language to the third machine learning model. The voice translation apparatus 220 may generate a standard language input text of the entire second language by merging a part of the standard language input text of the second language.

In addition, the third machine learning model of the speech translation apparatus 220 may use an attention algorithm. The speech translation device 220 is included in the standard language input text of the second language based on a part of phoneme, syllable, morpheme, or word included in the third standard language input text of the first language or the fourth standard language input text of the first language. You can create phonemes, syllables, morphemes or words. For example, the third machine learning model of the speech translation device 220 may pay attention to the subject portion of the first language in order to generate the subject portion of the second language. Also, the third machine learning model of the voice translation apparatus 220 may pay attention to the verb part of the first language in order to generate the verb part of the second language. When the third machine learning model uses an attention algorithm, the text of the first language can be converted to the text of the second language more accurately.

In addition, according to the attention algorithm, the speech translation apparatus 220 generates the second language text based on the front sentence of the first language text at the beginning of the generation of the second language text, and as the second language text is generated. Increasingly, the second language text can be generated based on the back sentence of the first language text.

The standard language input text of the second language is generated based on both the third standard language input text of the first language and the fourth standard language input text of the first language, and thus the third standard language input text of the first language and the fourth language of the first language. It may be more accurate than the text of the second language generated based on any one of the standard language input text.

The speech translation device 220 may perform an operation 760 of generating the output speech of the second language based on the standard language input text and the text to speech model of the second language. The text-to-speech model can be implemented based on various algorithms.

The text-to-speech model may generate the output speech of the second language by applying various speech speeds, pronunciation accents, pitches, lengths of speech sections, lengths of silence sections, emotions, and tones based on additional information. .

The speech translation apparatus 220 analyzes at least one hidden state of the first machine learning model and the second machine learning model to obtain information indicating whether the second dialect input speech of the first language is dialect. can do. For example, the first machine learning model and the second machine learning model may output standard word text by applying weights to the received voice or text. The speech translation apparatus 220 may analyze whether the weight applied to the received speech or text is a weight for a dialect or a weight for a standard word. Also, when the weight applied to the received speech or text is a weight highly related to the dialect, the speech translation apparatus 220 may generate information indicating that the received speech or text is dialect.

If the text-to-speech model of the speech translation device 220 indicates the dialect, the text-to-speech model performs the step of generating an output speech of the second language that includes any dialect accent of the second language. can do. In addition, the text-to-speech model of the speech translation apparatus 220 may perform the step of generating an output speech of the second language including the standard language intonation of the second language when the information represents the standard language. That is, when the voice or text of the first language applied to the first machine learning model and the second machine learning model is a dialect, the voice translation device 220 may output a dialect of the second language. In addition, when the voice or text of the first language applied to the first machine learning model and the second machine learning model is a standard language, the voice translation apparatus 220 may output the voice of the standard language of the second language.

The voice translation apparatus 220 may perform a step of receiving accuracy information on the output voice of the second language from the user. For example, a speaker of the second language or a speaker of the first language can evaluate whether the output voice of the second language is correct. The voice translation device 220 may receive the accuracy information.

The third machine learning model may be updated based on the accuracy information of the voice translation apparatus 220, the third standard language input text of the first language, and the fourth standard language input text of the first language. The voice translation apparatus 220 may update the third machine learning model to increase accuracy information. The speech translation apparatus 220 may be repeatedly updated to obtain a third machine learning model that is increasingly mature. The speech translation apparatus 220 may generate an output speech of the correct second language based on the third machine learning model.

8 is a diagram illustrating a screen of a user terminal according to an exemplary embodiment of the present disclosure.

The user terminal 810 may correspond to the voice translation apparatus 220 of FIG. 2. The user terminal 810 may include a display unit. The display unit may display a voice input button 821, a voice input result display area 822, a dialect division display area 823, and a translation result display area 824. The user may press the voice input button 821. The user terminal 810 may be switched to a mode for receiving a voice from the user. The user terminal 810 may receive a voice from the user.

The user terminal 810 may convert the received voice into text. The user terminal 810 may convert the received voice into text. In other words, the text may not be a standard word. The user terminal 810 may display the converted text in the voice input result display area 822. The user may check the text of the voice input result display area 822 to determine whether the user terminal 810 properly recognizes the voice.

The user terminal 810 may determine which local dialect is spoken based on the intonation or words of the voice. In addition, the user terminal 810 may display in which area dialect the voice corresponds to the area division display area 823 of the dialect.

The user terminal 810 may include a first machine learning model or a second machine learning model. The user terminal 810 may analyze the hidden state of at least one of the first machine learning model and the second machine learning model to obtain information indicating whether the second dialect input voice of the first language is dialect. Can be. For example, the first machine learning model and the second machine learning model may output standard word text by applying weights to the received voice or text. The user terminal 810 may analyze whether the weight applied to the received voice or text is a weight for a specific local dialect or a weight for a standard word. In addition, when the weight applied to the received voice or text is a weight highly related to a specific local dialect, the user terminal 810 may generate information indicating that the received voice or text is a specific local dialect. However, the present invention is not limited thereto, and the user terminal 810 may determine which local dialect includes voice based on a predetermined algorithm.

According to an embodiment of the present disclosure, the display unit of the user terminal 810 may further include a translation button (not shown). When the user presses a translation button, the user terminal 810 may obtain the third standard language input text of the first language by applying the input voice to the first machine learning model. In addition, the user terminal 810 may convert the input voice into the second dialect input text of the first language. In addition, the user terminal 810 may obtain the fourth standard language input text of the first language by applying the second dialect input text of the first language to the second machine learning model. In addition, the user terminal 810 may obtain the standard language input text of the second language by applying the third standard language input text of the first language and the fourth standard language input text of the first language to the third machine learning model. The user terminal 810 may display the translated text in the translation result display area 824.

According to an embodiment of the present disclosure, when a user presses a translation button, the user terminal 810 may transmit an input voice to a server (not shown). The server may obtain the third standard language input text of the first language by applying the input voice to the first machine learning model. The server may also convert the input voice into a second dialect input text of the first language. In addition, the server may obtain the fourth standard language input text of the first language by applying the second dialect input text of the first language to the second machine learning model. In addition, the server may obtain the standard language input text of the second language by applying the third standard language input text of the first language and the fourth standard language input text of the first language to the third machine learning model. The server may also transmit the standard language input text of the second language to the user terminal 810. The user terminal 810 may display the translated text in the translation result display area 824.

In addition, the user terminal 810 may generate an output voice of the second language based on the standard language input text and the text-to-speech model of the second language. In addition, the user terminal 810 may output an output voice of a second language based on a voice output unit (not shown) such as a speaker.

So far I looked at the center of the various embodiments. Those skilled in the art will appreciate that the present invention can be implemented in a modified form without departing from the essential features of the present invention. Therefore, the disclosed embodiments should be considered in descriptive sense only and not for purposes of limitation. The scope of the present invention is shown in the claims rather than the foregoing description, and all differences within the scope will be construed as being included in the present invention.

Meanwhile, the above-described embodiments of the present invention can be written as a program that can be executed in a computer, and can be implemented in a general-purpose digital computer that operates the program using a computer-readable recording medium. The computer-readable recording medium includes a storage medium such as a magnetic storage medium (eg, a ROM, a floppy disk, a hard disk, etc.) and an optical reading medium (eg, a CD-ROM, a DVD, etc.).

Claims (8)

As a method for recognizing and translating dialects,
Studying the relationship between the dialect of the first language and the standard language text of the first language based on the dialect voices of the first language and the standard language texts of the first language corresponding to the dialect voices of the first language Obtaining a first machine learning model;
Studying the relationship between the dialect text of the first language and the standard language text of the first language based on the dialect text of the first language and the standard language text of the first language corresponding to the dialect text of the first language Obtaining a second machine learning model;
Receiving standard language text of a second language corresponding to a first dialect input voice of a first language and a first dialect input voice of the first language;
Obtaining a first standard language input text of a first language by applying the first dialect input speech of the first language to the first machine learning model;
Converting the first dialect input speech of the first language into a first dialect input text of the first language;
Applying a first dialect input text of the first language to the second machine learning model to obtain a second standard language input text of a first language;
Obtaining a third machine learning model by performing machine learning based on the first standard language input text of the first language, the second standard language input text of the first language, and the standard language text of the second language;
Receiving a second dialect input voice of a first language;
Obtaining a third standard language input text of a first language by applying a second dialect input voice of the first language to the first machine learning model;
Converting a second dialect input speech of the first language into a second dialect input text of the first language;
Applying a second dialect input text of the first language to the second machine learning model to obtain a fourth standard language input text of a first language;
Obtaining a standard language input text of a second language by applying the third standard language input text of the first language and the fourth standard language input text of the first language to the third machine learning model;
Generating an output speech of a second language based on the standard language input text and the text-to-speech model of the second language;
Acquiring information indicating whether a second dialect input speech of the first language is dialect based on a weight included in at least one hidden state of the first machine learning model and the second machine learning model;
The text-to-speech model is further configured to generate an output speech of the second language that includes any dialect accent of the second language if the information represents dialect; And
And the text-to-speech model includes generating an output speech of the second language that includes a standard language accent of a second language if the information represents a standard language.
delete delete The method of claim 1,
Receiving accuracy information about an output voice of a second language from a user; And
Updating the third machine learning model based on the accuracy information, the third standard language input text of the first language, and the fourth standard language input text of the first language.
As a device for recognizing and translating a mixed dialect,
The apparatus includes a processor and a memory, the processor according to the instructions contained in the memory,
Studying the relationship between the dialect of the first language and the standard language text of the first language based on the dialect voices of the first language and the standard language texts of the first language corresponding to the dialect voices of the first language Obtaining a first machine learning model;
Studying the relationship between the dialect text of the first language and the standard language text of the first language based on the dialect text of the first language and the standard language text of the first language corresponding to the dialect text of the first language Obtaining a second machine learning model;
Receiving standard language text of a second language corresponding to a first dialect input voice of a first language and a first dialect input voice of the first language;
Obtaining a first standard language input text of a first language by applying the first dialect input speech of the first language to the first machine learning model;
Converting the first dialect input speech of the first language into a first dialect input text of the first language;
Applying a first dialect input text of the first language to the second machine learning model to obtain a second standard language input text of a first language;
Obtaining a third machine learning model by performing machine learning based on the first standard language input text of the first language, the second standard language input text of the first language, and the standard language text of the second language;
Receiving a second dialect input voice of a first language;
Obtaining a third standard language input text of a first language by applying a second dialect input voice of the first language to the first machine learning model;
Converting a second dialect input speech of the first language into a second dialect input text of the first language;
Applying a second dialect input text of the first language to the second machine learning model to obtain a fourth standard language input text of a first language;
Obtaining a standard language input text of a second language by applying the third standard language input text of the first language and the fourth standard language input text of the first language to the third machine learning model;
Generating an output speech of a second language based on the standard language input text and the text-to-speech model of the second language;
Obtaining information indicating whether a second dialect input speech of the first language is dialect based on a weight of at least one hidden state of the first machine learning model and the second machine learning model;
The text-to-speech model is further configured to generate an output speech of the second language that includes any dialect accent of the second language if the information represents dialect; And
And the text-to-speech model performs the step of generating an output speech of the second language that includes a standard language accent of a second language when the information represents a standard language.
delete delete The method of claim 5,
The processor according to the instructions contained in the memory,
Receiving accuracy information about an output voice of a second language from a user; And
And updating the third machine learning model based on the accuracy information, the third standard language input text of the first language, and the fourth standard language input text of the first language.

Images