KR102374525B1 - Keyword Spotting Apparatus, Method and Computer Readable Recording Medium Thereof - Google Patents

Abstract

A keyword spotting apparatus, method, and computer-readable recording medium are disclosed. A keyword spotting method according to an embodiment of the present invention is a keyword spotting method using an artificial neural network, comprising the steps of obtaining an input feature map from an input voice, and the same channel as the input feature map. ), a first convolution operation with the input feature map is performed for each of n different filters having a length w1 - a length of w1 is smaller than the width of the input feature map. performing a second convolution operation with the result of the first convolution operation for each of the different filters having the same channel length as the input feature map, and outputting the result of the previous convolution operation as an output feature It includes the steps of storing as a map (Output Feature Map) and extracting a voice keyword by applying the output feature map to a learned machine learning model.

Description

Keyword Spotting Apparatus, Method and Computer Readable Recording Medium Thereof

The present invention relates to a keyword spotting apparatus, method, and computer-readable recording medium, and more particularly, to a keyword spotting apparatus, method, and computer-readable recording medium capable of spotting keywords very quickly while maintaining high accuracy. it's about

Keyword Spotting (KWS) plays a very important role in speech-based user interaction in smart devices. Recent technological advances in the field of deep learning are leading the application of CNN to the keyword spotting field because of the accuracy and robustness of convolutional neural networks (CNNs).

The most important challenge facing keyword spotting systems is resolving the trade-off between high accuracy and low latency. This has become a very important issue since it is known that the traditional convolution-based keyword spotting approach requires a very large amount of computation to achieve a suitable level of performance.

Nevertheless, studies on the actual latency of keyword spotting models in mobile devices are not active.

An object of the present invention is to provide a keyword spotting apparatus, method, and computer-readable recording medium capable of rapidly extracting voice keywords with high accuracy.

A keyword spotting method according to an embodiment of the present invention is a keyword spotting method using an artificial neural network, comprising the steps of obtaining an input feature map from an input voice, and the same channel as the input feature map. ), a first convolution operation with the input feature map is performed for each of n different filters having a length w1 - a length of w1 is smaller than the width of the input feature map. performing a second convolution operation with the result of the first convolution operation for each of the different filters having the same channel length as the input feature map, and outputting the result of the previous convolution operation as an output feature It includes the steps of storing as a map (Output Feature Map) and extracting a voice keyword by applying the output feature map to a learned machine learning model.

Also, a stride value of the first convolution operation may be 1.

In addition, the performing of the second convolution operation includes performing a first sub-convolution operation between m different filters having a width of w2 and a result of the first convolution operation, m having a width of w2 performing a second sub-convolution operation between different filters and a result of the first sub-convolution operation, and a third sub-convolution operation between m different filters having a width of 1 and the result of the first sub-convolution operation The method may include performing a convolution operation and summing results of the second and third sub-convolution operations.

Also, the stride values of the first to third sub-convolution operations may be 2, 1, and 2, respectively.

In addition, the method further includes: performing a third convolution operation, wherein the performing the third convolution operation comprises l different filters having a width of w2 and a result of the second convolution operation 4 performing a subconvolution operation, performing a fifth subconvolution operation with l different filters of width w2 and the result of the fourth subconvolution operation, l different filters having a width of 1 The method may include performing a sixth sub-convolution operation between a filter and a result of the second convolution operation and summing the results of the fifth and sixth sub-convolution operations.

The method may further include performing a fourth convolution operation, wherein the performing the fourth convolution operation comprises performing a seventh operation between m different filters having a width w2 and a result of the second convolution operation. performing a sub-convolution operation, performing an eighth sub-convolution operation with m different filters of width w2 and the result of the seventh sub-convolution operation, and the result of the second convolution operation The method may include summing the results of the eighth sub-convolution operation.

Also, the stride value of the seventh and eighth sub-convolution operations may be 1.

In addition, in the step of obtaining the input feature map, it is possible to obtain an input feature map having a size of t × 1 × f (width × height × channel) from the result of MFCC (Mel Frequency Cepstral Coefficient) processing for the input voice, , where t is time and f is frequency.

Meanwhile, a computer-readable recording medium in which a program for performing the keyword spotting method according to the present invention is recorded may be provided.

On the other hand, the keyword spotting apparatus according to an embodiment of the present invention is an apparatus for spotting keywords using an artificial neural network, a memory in which at least one program is stored and the at least one program are executed, thereby generating an artificial neural network. a processor for extracting a speech keyword using A first convolution operation with the input feature map is performed on each of n different filters, which is smaller than the width of A second convolution operation with the result of the convolution operation is performed, the result of the previous convolution operation is stored as an output feature map, and a speech keyword is extracted by applying the output feature map to the learned machine learning model.

Also, the stride value of the first convolution operation may be 1.

Also, in performing the second convolution operation, the processor performs a first sub-convolution operation with m different filters having a width of w2 and a result of the first convolution operation, and a width of w2 A second sub-convolution operation is performed between m different filters of , and a result of the first sub-convolution operation, and a third result of m different filters having a width of 1 and the result of the first convolution operation is performed. A sub-convolution operation may be performed, and results of the second and third sub-convolution operations may be summed.

Also, the stride values of the first to third sub-convolution operations may be 2, 1, and 2, respectively.

In addition, the processor further performs a third convolution operation, and in performing the third convolution operation, the processor is configured to perform a comparison between l different filters having a width of w3 and a result of the second convolution operation. A fourth sub-convolution operation is performed, and a fifth sub-convolution operation is performed with l different filters having a width of w3 and the result of the fourth sub-convolution operation, and l different filters having a width of 1 and a sixth sub-convolution operation with a result of the second convolution operation may be performed, and the results of the fifth and sixth sub-convolution operations may be summed.

In addition, the processor further performs a fourth convolution operation, and in performing the fourth convolution operation, the processor is configured to perform m different filters having a width of w2 and a result of the second convolution operation. A seventh sub-convolution operation is performed, an eighth sub-convolution operation is performed with m different filters of width w2 and the result of the seventh sub-convolution operation, and the result of the second convolution operation is The results of the eighth sub-convolution operation may be summed.

Also, the stride value of the seventh and eighth sub-convolution operations may be 1.

In addition, the processor may obtain the input feature map having a size of t×1×f (width×height×channel) from a result of MFCC processing on the input voice, where t is time and f is frequency. it means.

The present invention can provide a keyword spotting apparatus, method, and computer-readable recording medium capable of rapidly extracting voice keywords with high accuracy.

1 is a block diagram illustrating a convolutional neural network according to an embodiment.
2 is a diagram schematically illustrating a convolution operation method according to the prior art.
3 is a diagram schematically illustrating a convolution operation method according to an embodiment of the present invention.
4 is a flowchart schematically illustrating a keyword spotting method according to an embodiment of the present invention.
5 is a flowchart schematically illustrating a convolution operation process according to an embodiment of the present invention.
6 is a block diagram schematically illustrating a configuration of a first convolution block according to an embodiment of the present invention.
7 is a flowchart schematically illustrating a keyword spotting method according to another embodiment of the present invention.
8 is a flowchart schematically illustrating a keyword spotting method according to another embodiment of the present invention.
9 is a block diagram schematically illustrating a configuration of a second convolution block according to an embodiment of the present invention.
10 is a flowchart schematically illustrating a keyword spotting method according to another embodiment of the present invention.
11 is a flowchart schematically illustrating a keyword spotting method according to another embodiment of the present invention.
12 is a performance comparison table for explaining the effect of the keyword spotting method according to the present invention.
13 is a diagram schematically showing the configuration of a keyword spotting apparatus according to an embodiment of the present invention.

Advantages and features of the present invention and methods of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be implemented in a variety of different forms, and only these embodiments allow the disclosure of the present invention to be complete, and common knowledge in the technical field to which the present invention belongs It is provided to fully inform the possessor of the scope of the invention, and the present invention is only defined by the scope of the claims. Like reference numerals refer to like elements throughout.

Although "first" or "second" is used to describe various elements, these elements are not limited by the above terms. Such terms may only be used to distinguish one component from another. Accordingly, the first component mentioned below may be the second component within the spirit of the present invention.

The terminology used herein is for the purpose of describing the embodiment and is not intended to limit the present invention. In this specification, the singular also includes the plural unless specifically stated otherwise in the phrase. As used herein, “comprises” or “comprising” implies that the stated component or step does not exclude the presence or addition of one or more other components or steps.

Unless otherwise defined, all terms used herein may be interpreted with meanings commonly understood by those of ordinary skill in the art to which the present invention pertains. In addition, terms defined in a commonly used dictionary are not to be interpreted ideally or excessively unless specifically defined explicitly.

1 is a block diagram illustrating a convolutional neural network according to an embodiment.

A convolutional neural network (CNN) is a type of artificial neural network (ANN), and may be mainly used to extract features of matrix data or image data. The convolutional neural network may be an algorithm that learns features from historical data.

In the convolutional neural network, the processor may acquire features by applying a filter to the input image 110 through the first convolutional layer 120 . The processor may reduce the size by subsampling the filtered image through the first pooling layer 130 . The processor may filter the image through the second convolution layer 140 and the second pooling layer 150 to extract features, and reduce the size by subsampling the filtered image. Thereafter, the processor may acquire the output data 170 by completely concatenating the processed images through the hidden layer 160 .

In a convolutional neural network, the convolution layers 120 and 140 perform a convolution operation between input activation data, which is three-dimensional input data, and weight data, which is four-dimensional data representing learnable parameters. Output activation data that is 3D output data may be obtained. Here, the obtained output activation data may be used as input activation data in the next layer.

On the other hand, since thousands of multiplication and addition operations are required to calculate one pixel on the output activation data, which is the three-dimensional output data, most of the time for data processing in the convolutional neural network is spent in the convolution layer.

2 is a diagram schematically illustrating a convolution operation method according to the prior art.

First, FIG. 2( a ) exemplarily shows an input voice that is a target of voice keyword extraction. In the input voice, the horizontal axis represents time and the vertical axis represents a feature. Considering that the input voice is a voice signal, the feature may be understood to mean a frequency. FIG. 2(a) shows features or frequencies having a constant size over time for convenience of explanation, but it should be understood that voice has a characteristic in which the frequency changes with time. Therefore, it can be understood that the voice signal in the general case is a signal having different frequencies along the time axis.

Meanwhile, the graph shown in FIG. 2A may be a result of MFCC (Mel Frequency Cepstral Coefficient) processing on the input voice. MFCC is one of the techniques for extracting the characteristics of a sound. It is a technique for dividing an input sound by a certain section (short time) and extracting the feature through spectrum analysis for each section. And, the input data (I) expressed in the graph can be displayed as follows.

FIG. 2( b ) shows an input feature map generated from the input voice shown in FIG. 2( a ) and a convolution filter K1 . The input feature map may be generated from the MFCC data shown in Fig. 2(a). As shown in Fig. 2(b), width (w, width) x height (h, height) x channel (c, channel) may be generated in the size of t×f×1 based on the direction. And, the input data X _2d corresponding to the input feature map may be expressed as follows.

Since the channel direction length of the input feature map is 1, it can be understood as two-dimensional data having a size of t×f. The filter K1 may perform a convolution operation while moving in the width direction and the height direction on the t×f plane of the input feature map. Here, assuming that the size of the filter K1 is 3×3, the weight data W _2d corresponding to the filter K1 can be expressed as follows.

When a convolution operation is performed between the input data X _2d and the weight data W _2d , the number of operations may be expressed as 3×3×1×f×t . And, assuming that the number of filters K1 is c, the number of operations may be expressed as 3×3×1×f×t×c.

FIG. 2( c ) exemplarily illustrates an output feature map obtained as a result of a convolution operation between the input data X _2d and the weight data W _2d . Previously, it can be seen that the channel direction length of the output feature map is c since the number of filters K1 is assumed to be c. Accordingly, data corresponding to the output feature map may be expressed as follows.

A convolutional neural network is known as a neural network that performs continuous transformation from a low-level to a high-level. On the other hand, since modern convolutional neural networks use small filters, it may be difficult to extract useful features from both low-frequency and high-frequency in a relatively shallow network. there is.

When image analysis is performed through a convolutional neural network, adjacent pixels among a plurality of pixels constituting an image have a high probability of having similar characteristics, and pixels at distant locations have a relatively high probability of having different characteristics. Therefore, convolutional neural networks can be a good tool for image analysis and training.

On the other hand, when a conventional convolutional neural network used for image analysis and learning is applied to speech analysis and learning, efficiency may be reduced. This is because the magnitude of the frequency of the voice signal changes with the passage of time, and even the temporally adjacent voice signals may have a large difference in the magnitude of the frequency.

3 is a diagram schematically illustrating a convolution operation method according to an embodiment of the present invention.

3A illustrates an input feature map and a filter K according to an embodiment of the present invention. The input feature map shown in FIG. 3(a) may have substantially the same size as the input feature map of FIG. 2(b), but may have a height of 1 and a length in the channel direction may be defined as f. Meanwhile, the length in the channel direction of the filter K according to the present invention is the same as the length in the channel direction of the input feature map. Therefore, only one filter K can cover all frequency data included in the input feature map.

Meanwhile, the input data X _1d and weight data W _1d corresponding to the input feature map of FIG. 3A and the filter K may be expressed as follows, respectively.

Since the channel direction lengths of the input feature map and the filter K are the same, the filter K needs to perform a convolution operation while moving only in the width w direction. A solution operation can be understood as a one-dimensional operation.

Meanwhile, when the shape of the input feature map is the same as that of the input feature map of FIG. 2B , the length in the channel direction of the filter K may be defined as 1 and the height may be defined as f. Accordingly, the filter K according to the present invention is not limited to the length in the channel direction. Since the height of the input feature map or the length in the channel direction may be 1, in the filter (K) according to the present invention, the length in the other directions except for the width direction is the same as the length in the direction corresponding to the input feature map do it with

When a convolution operation is performed between the input data X _1d and the weight data W _1d , the number of operations may be expressed as 3×1×f×t×1 . And, assuming that the number of filters K is c`, the number of operations may be expressed as 3×1×f×t×1×c′.

3B exemplarily illustrates an output feature map obtained as a result of a convolution operation between the input data X _1d and the weight data W _1d . Previously, it can be seen that the channel direction length of the output feature map is c' since the number of filters K is assumed to be c'. Accordingly, data corresponding to the output feature map may be expressed as follows.

When the size of the filter K1 used in the prior art is compared with the size of the filter K according to the present invention, the size of the filter K according to the present invention is larger and the convolution operation is performed with one filter. The number of filters (K) according to the present invention is also larger. Accordingly, the convolution operation according to the present invention can perform the convolution operation with only a smaller number of filters compared to the prior art.

That is, since the relationship c>c' is established, the total number of convolution operations is further reduced when the convolution operation method according to the present invention is used.

4 is a flowchart schematically illustrating a keyword spotting method according to an embodiment of the present invention.

Referring to FIG. 4 , the keyword spotting method according to an embodiment of the present invention includes obtaining an input feature map from an input voice (S110), performing a first convolution operation (S120), and a second convolution It includes a step of performing a solution operation (S130), a step of storing the output feature map (S140), and a step of extracting a voice keyword (S150).

The keyword spotting method according to the present invention is a keyword spotting method using an artificial neural network, and in step S110, an input feature map is obtained from an input voice. Voice keyword extraction aims to extract a predefined keyword from an audio signal stream. For example, the keyword spotting method may be used even when keywords such as "Hey Siri" and "Okay Google" are identified from a human voice and used as a wake-up language of the device.

In step S110, an input feature map having a size of t×1×f (width×height×channel) may be obtained from the result of MFCC (Mel Frequency Cepstral Coefficient) processing for the input voice. Here, t is time and f is frequency.

The input feature map may express an input voice acquired for a predetermined time t as a frequency with respect to a time t. In addition, each frequency data in the input feature map may be expressed as a floating point.

In step S120, a first convolution operation with the input feature map is performed on each of n different filters having the same channel length as the input feature map. The n different filters may discriminate different voices, respectively. For example, the first filter may determine a voice corresponding to 'a', and the second filter may determine a voice corresponding to 'o'. Through the convolution operation with the n different filters, voices corresponding to the characteristics of each filter may have their sound characteristics further reinforced.

The width of the n filters may be defined as w1. If the width of the input feature map is w, the relationship w>w1 may be established. For example, in FIG. 3A , the width of the input feature map may be t and the width of the filter K1 may be 3 . Meanwhile, in the first convolution operation, when the number of filters is n, a total of n outputs exist. The convolution operation may be performed through a processor such as a CPU, and a result of the convolution operation may be stored in a memory or the like.

In the first convolution operation performed in step S120, a stride value of '1' may be applied. The stride means the interval at which the filter skips the input data in the convolution operation. If '1' is applied as the stride value, the convolution operation is performed with all input data without skipping data.

In step S130, a second convolution operation is performed with the result of the first convolution operation for each of the different filters having the same channel length as the input feature map. The number of filters used for the second convolution operation may be m, and the width of the filter may be defined as w2. In this case, the width of the filter used for the second convolution operation and the width of the filter used for the first convolution operation may be the same or different from each other.

However, the length in the channel direction of the filter used in the second convolution operation is the same as the length in the channel direction of the filter used in the first convolution operation, and is the same as the length in the channel direction of the input feature map.

The m different filters used in the second convolution operation may discriminate different voices, respectively, like n different filters used in the first convolution operation.

In step S140, the result of the previous convolution operation is stored as an output feature map. The output feature map is a result that is finally obtained before extracting the voice keyword in step S150, and may have a form as shown in FIG. 3(b). The output feature map stored in step S140 may be understood as a result of the convolution operation performed in the immediately preceding step. Accordingly, in step S140 , the result of the second convolution operation performed in step S130 may be stored as the output feature map.

In step S150, a voice keyword is extracted by applying the output feature map to a learned machine learning model. The machine learning model may include a pooling layer, a full-connect layer, a softmax operation, and the like, which will be described in detail with reference to the accompanying drawings.

Meanwhile, a pooling operation may be performed following the convolution operation described in this specification, and a zero padding technique may be applied to prevent data reduction that may occur during the pooling operation.

5 is a flowchart schematically illustrating a convolution operation process according to an embodiment of the present invention.

Referring to FIG. 5 , the steps of performing the first convolution operation ( S120 ) and performing the second convolution operation ( S130 ) are shown in more detail. In the step in which the first convolution operation is performed, a convolution operation is performed between the input feature map and the filter. In the embodiment shown in FIG. 5, the filter has a size of 3×1, which means a filter having a width of 3 and a height of 1. In addition, as described with reference to FIG. 4 , the channel direction length of the filter is the same as the channel direction length of the input feature map.

Meanwhile, 1 is applied to the stride value of the first convolution operation, and the number of the filters may be 16k. Here, k is a multiplier for the number of channels, and when k is 1, it can be understood that a convolution operation is performed between a total of 16 different filters and the input feature map. That is, when the value of k is 1, the value of n is 16.

In the step of performing the second convolution operation, a convolution operation may be performed between the result of the first convolution operation and the first convolution block. For the second convolution operation, filters having a different width from the filter used in the step in which the first convolution operation is performed may be used. In addition, in the embodiment shown in FIG. 5 , a stride value of 2 may be applied to the second convolution operation. In addition, 16k different filters may be used in the first convolution operation, but 24k different filters may be used in the second convolution operation. Therefore, it can be understood that a filter that is 1.5 times larger than that of the first convolution operation is used in the second convolution operation. That is, when the value of k is 1, the value of m becomes 24.

When the second convolution operation is performed, a final result may be extracted after performing a pooling layer, a full connect layer, and a softmax operation.

One model including the first and second convolution operations, pooling, full connect, and softmax of FIG. 5 may be understood as an artificial neural network model according to an embodiment of the present invention.

6 is a block diagram schematically illustrating a configuration of a first convolution block according to an embodiment of the present invention.

The first convolution block is a convolution block used for the second convolution operation described with reference to FIGS. 4 and 5 . The second convolution operation using the first convolution block ( S130 ) includes performing a first sub-convolution operation between m different filters having a width of w2 and the result of the first convolution operation ( S131 ). ) is included. In this case, the width of the filter used may be different from the width of the filter used for the first convolution operation or the second convolution operation. In the exemplary embodiment shown in FIG. 6 , the width of the filter used in step S131 , that is, the w2 value may be 9 . In addition, as described with reference to FIG. 5 , the number of filters used in step S131 is 16k, and a total of 16k convolution operations may be performed. Meanwhile, in step S131 , 2 may be applied as a stride value.

When step S131 is performed, batch normalization and an activation function may be applied to the convolution result. In FIG. 6, an embodiment using a Rectified Linear Unit (ReLU) as the activation function is disclosed. Batch Normalization can be applied to solve the Gradient Vanishing or Gradient Exploding problem that may occur as the convolution operation is repeatedly performed. Activation Function can also be applied to solve the problem of gradient vanishing or gradient exploding, and can be a means to solve the problem of over fitting. There are various types of functions in the activation function, and the present invention discloses a method of applying the ReLU function as an embodiment. When the convolutional data is normalized through the batch normalization process, the ReLU function can be applied to the normalized data.

Batch Normalization and Activation Function are used to solve gradient vanishing (or exploding) and over fitting problems that may occur while going through multiple convolution layers. Embodiments of the invention may be variously changed.

When the application of the batch normalization and the activation function is completed, a second sub-convolution operation may be performed (S132). In step S132, a second sub-convolution operation is performed between m different filters having a width of w2 and the result of the first sub-convolution operation. In this case, the width of the filter used may be different from the width of the filter used for the first convolution operation or the second convolution operation. In the embodiment shown in FIG. 6 , the width of the filter used in step S132 may be 9. In addition, as described with reference to FIG. 5 , the number of filters used in step S132 is 16k, and a total of 16k convolution operations may be performed. Meanwhile, in step S132 , 1 may be applied as a stride value. Meanwhile, when the second sub-convolution operation is performed in step S132, batch normalization may be performed.

In step S133, a third sub-convolution operation may be performed. In the third sub-convolution operation, a convolution operation is performed between m different filters having a width of 1 and the result of the convolution operation performed in step S120. And, in this case, 2 may be applied as the stride value. After the third sub-convolution operation is performed in step S133, batch normalization and activation function may be applied thereafter.

Meanwhile, step S130 may further include adding the result of the second sub-convolution operation performed in step S132 with the result of the third sub-convolution operation performed in step S133. As shown in FIG. 6 , the results of the second and third sub-convolution operations are summed, and when an activation function is finally applied to the summed result, data may be transferred to a next step, for example, a pooling layer.

7 is a flowchart schematically illustrating a keyword spotting method according to another embodiment of the present invention.

Referring to FIG. 7 , in a keyword spotting method according to another embodiment of the present invention, performing a first convolution operation (S220), performing a second convolution operation (S230), and a third convolution operation and performing a solution operation (S240). In steps S220 and S230, the first convolution operation and the second convolution operation described with reference to FIGS. 4 to 6 may be performed.

In step S240, the third sub-convolution operation may be performed using the first convolution block including the configuration shown in FIG. 6 . 6 and 7 together, the step of performing the third sub-convolution operation ( S240 ) includes l different filters having a width of w2 and a fourth result of the second convolution operation. performing a sub-convolution operation, performing a fifth sub-convolution operation with l different filters of width w2 and the result of the fourth sub-convolution operation, l different filters having a width of 1 and performing a sixth sub-convolution operation with a result of the second convolution operation, and summing the results of the fifth and sixth sub-convolution operations.

That is, in step S240, it can be understood that the same type of convolution operation as in step S230 is performed once more. However, the number of filters used in step S240 may be different from the number of filters used in step S230. Referring to FIG. 7 , it can be seen that 24k filters may be used in step S230 and 32k filters may be used in step S240 .

8 is a flowchart schematically illustrating a keyword spotting method according to another embodiment of the present invention.

Referring to FIG. 8 , in the keyword spotting method according to another embodiment of the present invention, performing a first convolution operation (S320), performing a second convolution operation (S330), and a fourth It may include performing a convolution operation (S340). In operation S340, a fourth convolution operation may be performed using the second convolution block.

The number of filters used in step S330 and the number of filters used in step S340 may be the same as 24k, but the number of filters is the same, but the size of the filter and data included in the filter (eg, weight activation) ) can be different.

9 is a block diagram schematically illustrating a configuration of a second convolution block according to an embodiment of the present invention.

The second convolution block may be performed in step S340 of performing a fourth convolution operation. Referring to FIG. 9 , the step S340 includes performing a seventh sub-convolution operation with m different filters having a width of w2 and the result of the second convolution operation ( S341 ), and m filters having a width of w2 are performed. The method may include performing an eighth sub-convolution operation between different filters and a result of the seventh sub-convolution operation (S342).

In FIG. 9 , the width of the filter used in steps S341 and S342 may be set to '9' and the stride value may be set to '1'. Also, batch normalization may be applied to the result of the seventh sub-convolution operation performed in step S341, and then the ReLU function may be applied as an activation function.

Batch normalization may also be applied to the result of the eighth sub-convolution operation performed in step S342. In addition, a step of summing the result of the second convolution operation performed in step S330 and the result of the eighth sub-convolution operation may be performed.

The result of the second convolution operation performed in step S330 is summed with the result of the eighth sub-convolution operation without additional convolution operation, and a corresponding path may be defined as an Identity Shortcut. The reason that Identity Shortcut is applied differently from the first convolution block is that the stride values applied to the seventh and eighth sub-convolution operations included in the second convolution block are all 1, so the dimension in the convolution operation process is Because change doesn't happen.

Meanwhile, data to which the ReLU function is applied as an activation function for the summing result may be transferred to a next step, for example, a pooling step.

10 is a flowchart schematically illustrating a keyword spotting method according to another embodiment of the present invention.

Referring to FIG. 10 , in a keyword spotting method according to another embodiment of the present invention, a first convolution operation is performed as in step S120 of FIG. 4 , and a convolution using a first convolution block The operation may include three consecutive steps.

For the configuration of the first convolution block, refer to FIG. 6 . In FIG. 10 , 24k different filters may be used in the first first convolution block, and 32k and 48k different filters may be used in the second and last first convolution blocks, respectively.

Data to which the ReLU function of the last step is applied in the last first convolution block may be transferred to a next step, for example, a Pooing layer.

11 is a flowchart schematically illustrating a keyword spotting method according to another embodiment of the present invention.

Referring to FIG. 11 , a keyword spotting method according to another embodiment of the present invention includes performing a first convolution operation as in step S120 of FIG. 4 . The step of performing the convolution operation using the first convolution block and the step of performing the convolution operation using the second convolution block may include repeating three times.

Refer to FIG. 6 for the configuration of the first convolution block, and refer to FIG. 9 for the configuration of the second convolution block. 11 , 24k different filters are used in the first first and second convolution blocks, 32k different filters are used in the second first and second convolution blocks, and 48k different filters are used in the last first and second convolution blocks. Several different filters may be used.

Data to which the ReLU function of the last step is applied in the last first convolution block may be transferred to a next step, for example, a Pooing layer.

12 is a performance comparison table for explaining the effect of the keyword spotting method according to the present invention.

In the table of FIG. 12, models CNN-1 to Res15 show the results of using the speech keyword extraction model according to the conventional method. Among the voice keyword extraction models according to the conventional method, the CNN-2 model showed the best results (1.2ms) in terms of extraction time, and the Res15 model showed the best results (95.8%) in terms of accuracy.

The TC-ResNet8 model to the TC-ResNet14-1.5 model represents a model to which the keyword spotting method according to the present invention is applied. First, the TC-ResNet8 model is a model using the method shown in FIG. 10, and is a model to which 1 is applied as a k value. Accordingly, it can be understood that 16 different filters are used in the stage in which the first convolution operation is performed. Also, it can be understood that 24, 32, and 48 different filters are used in the first, second, and last first convolution blocks, respectively.

The TC-ResNet8-1.5 model is a model using the method shown in FIG. 10, and 1.5 is applied as a k value. Therefore, it can be understood that 24 different filters are used in the stage in which the first convolution operation is performed. Also, it can be understood that 36, 48, and 72 different filters are used in the first, second, and last first convolution blocks, respectively.

The TC-ResNet14 model is a model using the method shown in FIG. 11, and 1 is applied as a k value. Therefore, it can be understood that 16 different filters are used in the stage in which the first convolution operation is performed. And, sequentially, 24, 24, 32, 32, 48, and 48 different filters are used in the first, second, first, second, first, and second convolution blocks, respectively. can be understood as being

The TC-ResNet14-1.5 model is a model using the method shown in FIG. 11, and 1.5 is applied as a k value. Therefore, it can be understood that 24 different filters are used in the stage in which the first convolution operation is performed. Then, sequentially 36, 36, 48, 48, 72, and 72 different filters are used in the first, second, first, second, first, and second convolution blocks, respectively. can be understood as being

Referring to the table shown in Fig. 12, the TC-ResNet8 model recorded a keyword extraction time of 1.1 ms, which is superior to the fastest recording (1.2 ms) when using the model according to the conventional method. In addition, when using the TC-ResNet8 model, it can be understood that the accuracy of 96.1% can be secured, which is superior to the conventional method in terms of accuracy.

On the other hand, the TC-ResNet14-1.5 model recorded an accuracy of 96.6%, which is the most accurate value among the models according to the conventional method and the models according to the present invention.

Considering the accuracy of voice keyword extraction as the most important criterion, the keyword extraction time of the Res15 model, which is the best among conventional methods, is 424 ms. And, the keyword extraction time of the TC-ResNet14-1.5 model corresponding to an embodiment of the present invention is 5.7 ms. Therefore, even using the TC-ResNet14-1.5 model, which has the slowest keyword extraction time among embodiments of the present invention, it is possible to extract voice keywords about 74.8 times faster than the conventional method.

Meanwhile, in the case of using the TC-ResNet8 model, which has the fastest keyword extraction time among embodiments of the present invention, it is possible to extract voice keywords at a speed 385 times faster than that of the Res15 model.

13 is a diagram schematically showing the configuration of a keyword spotting apparatus according to an embodiment of the present invention.

Referring to FIG. 13 , the keyword spotting apparatus 20 may include a processor 210 and a memory 220 . Those of ordinary skill in the art related to the present embodiment may know that other general-purpose components other than those shown in FIG. 13 may be further included.

The processor 210 controls the overall operation of the keyword spotting apparatus 20 and may include at least one processor such as a CPU. The processor 210 may include at least one specialized processor corresponding to each function or may be an integrated processor.

The memory 220 may store a program, data, or file related to a convolution operation performed in the convolutional neural network. The memory 220 may store instructions executable by the processor 210 . The processor 210 may execute a program stored in the memory 220 , read data or files stored in the memory 220 , or store new data. In addition, the memory 220 may store program commands, data files, data structures, etc. alone or in combination.

The processor 210 may include a plurality of low-precision operators (eg, 8-bit operators) in which high-precision operators (eg, 32-bit operators) are designed in a hierarchical structure. In this case, the processor 210 may support an instruction for high-precision operation and a single instruction multiple data (SIMD) instruction for low-precision operation. If the bit-width is quantized to fit the input of the low-precision operator, the processor 210 performs a plurality of small-bit-width operations in parallel instead of performing the large-bit-width operation within the same time. By performing as , the convolution operation can be accelerated. The processor 210 may accelerate a convolution operation on a convolutional neural network through a predetermined binary operation.

The processor 210 may obtain an input feature map from the input voice. The input feature map may be obtained with a size of t×1×f (width×height×channel) from a result of MFCC (Mel Frequency Cepstral Coefficient) processing on the input voice.

The input feature map may express an input voice acquired for a predetermined time t as a frequency with respect to a time t. In addition, each frequency data in the input feature map may be expressed as a floating point.

Also, the processor 210 performs a first convolution operation with the input feature map for each of n different filters having the same channel length as the input feature map. In this case, the width w1 of the n different filters may be set to be smaller than the width of the input feature map.

The n different filters may discriminate different voices, respectively. For example, the first filter may determine a voice corresponding to 'a', and the second filter may determine a voice corresponding to 'o'. Through the convolution operation with the n different filters, voices corresponding to the characteristics of each filter may have their sound characteristics further reinforced.

The width of the n filters may be defined as w1. If the width of the input feature map is w, the relationship w>w1 may be established. When the number of the filters in the first convolution operation is n, when a total of n outputs exist, the result of the convolution operation may be stored in the memory 220 .

In the first convolution operation, '1' may be applied as a stride value. The stride means the interval at which the filter skips the input data in the convolution operation. If '1' is applied as the stride value, the convolution operation is performed with all input data without skipping data.

In addition, the processor 210 performs a second convolution operation with the result of the first convolution operation for each of the different filters having the same channel length as the input feature map.

The number of filters used for the second convolution operation may be m, and the width of the filter may be defined as w2. In this case, the width of the filter used for the second convolution operation and the width of the filter used for the first convolution operation may be the same or different from each other.

However, the length in the channel direction of the filter used in the second convolution operation is the same as the length in the channel direction of the filter used in the first convolution operation, and is the same as the length in the channel direction of the input feature map.

The m different filters used in the second convolution operation may discriminate different voices, respectively, like n different filters used in the first convolution operation.

In addition, the processor 210 stores the result of the previous convolution operation as an output feature map in the memory 220, and applies the output feature map to the learned machine learning model to extract a voice keyword.

The output feature map is a result that is finally obtained before extracting the voice keyword, and may have a form as shown in FIG. 3(b). The output feature map stored in the memory 220 may be understood as a result of the convolution operation performed in the immediately preceding step. Accordingly, in an embodiment of the present invention, the result of the second convolution operation may be stored as the output feature map.

Meanwhile, the machine learning model may include a pooling layer, a full connect layer, a softmax operation, and the like. In addition, a pooling operation may be performed following the convolution operation described in this specification, and a zero padding technique may be applied to prevent data reduction that may occur in the pooling operation process.

The embodiments described above may also be implemented in the form of a recording medium including instructions executable by a computer, such as a program module executed by a computer. Computer-readable media can be any available media that can be accessed by a computer and can include both volatile and non-volatile media, removable and non-removable media.

In addition, computer-readable media may include computer storage media. Computer storage media may include both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules, or other data.

Although embodiments of the present invention have been described with reference to the accompanying drawings in the above, those of ordinary skill in the art to which the present invention pertains can implement the present invention in other specific forms without changing the technical spirit or essential features. You will understand that there is Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive.

20: keyword spotting device
210: processor
220: memory

Claims (17)

A method of operating a keyword spotting device, comprising:
obtaining an input feature map in which frequency data corresponding to the same time is expressed in a preset direction from the input voice;
performing a convolution operation between the input feature map and at least one filter;
storing the result of the convolution operation as an output feature map; and
extracting keywords in the input speech based on the output feature map. According to claim 1,
The preset direction indicates a channel direction,
The input feature map represents data corresponding to a size tХ1Хf (width Хheight Хchannel), and the at least one filter represents data corresponding to a size t'Х1Хf (width Хheight Хchannel) ( t and t' denotes the number of activations arranged in the axial direction corresponding to time, and f denotes the number of activations arranged in the axial direction corresponding to frequency), the method. delete According to claim 1,
The step of performing a convolution operation between the input feature map and at least one filter comprises:
performing a first convolution operation between the input feature map and each of n different filters comprising a range of frequencies in the input feature map. 5. The method of claim 4,
The method further comprising: performing a second convolution by using a first convolution block that performs a plurality of sub-convolution operations in which at least one or more stride values are respectively applied to a result of the first convolution operation. 5. The method of claim 4,
The method further comprising: performing a second convolution based on the plurality of first convolution blocks to which at least one of the number of strides and filters is applied differently. 6. The method of claim 5,
Performing the second convolution comprises:
Performing the second convolution by using a second convolution block that performs a plurality of preset sub-convolution operations based on the result of the operation of the first convolution block and the stride value , method. According to claim 1,
In the obtaining of the input feature map, an input feature map having a size of t × 1 × f (width × height × channel) is obtained from a result of MFCC (Mel Frequency Cepstral Coefficient) processing for the input speech.
Here, t is time and f is frequency. A computer-readable recording medium in which a program for performing the method according to any one of claims 1 to 2 and 4 to 8 is recorded. An apparatus for extracting a voice keyword using an artificial neural network, the apparatus comprising:
a memory in which at least one program is stored; and
and a processor for extracting a voice keyword using an artificial neural network by executing the at least one program,
The processor is
obtaining an input feature map in which frequency data corresponding to the same time is expressed in a preset direction from the input voice;
performing a convolution operation between the input feature map and at least one filter;
storing the result of the convolution operation as an output feature map,
and extracting a keyword in the input voice based on the output feature map. 11. The method of claim 10,
The preset direction indicates a channel direction,
The input feature map represents data corresponding to a size tХ1Хf (width Хheight Хchannel), and the at least one filter represents data corresponding to a size t'Х1Хf (width Хheight Хchannel) ( t and t' denotes the number of activations arranged in the axial direction corresponding to time, f denotes the number of activations arranged in the axial direction corresponding to frequency), a keyword spotting device. delete 12. The method of claim 11,
The processor is
performing a first convolution operation between the input feature map and each of n different filters including the frequency range of the input feature map. 14. The method of claim 13,
The processor is

A keyword spotting apparatus for performing a second convolution by using a first convolution block that performs a plurality of sub-convolution operations to which a result of the first convolution operation and at least two or more stride values are respectively applied. 14. The method of claim 13,
The processor is
A keyword spotting apparatus for performing a second convolution based on a plurality of first convolution blocks to which at least one of the number of strides and filters is applied differently. 15. The method of claim 14,
The processor is
A keyword spotting device that performs the second convolution by using a second convolution block that performs a plurality of preset sub-convolution operations based on a result of the operation of the first convolution block and a stride value . 11. The method of claim 10,
The processor is configured to obtain the input feature map having a size of t×1×f (width×height×channel) from a result of MFCC processing on the input speech.
Here, t is time and f is frequency.

Images