JP2020126492A - Information processing device, speech recognition system, and speech recognition program - Google Patents

Abstract

An object of the present invention is to improve recognition accuracy of utterance content. An input unit to which moving image data captured by an imaging device is input, and an area indicating a person's lips is recognized from an image for each frame included in the moving image data, and continuous lip images of the person are generated. a lip region extracting unit for extracting the lip region image data shown in FIG. It has a recognition model selection unit, an utterance recognition unit that recognizes the utterance content of the person using the selected recognition model, and an output unit that outputs the recognition result of the utterance content. [Selection drawing] Fig. 4

Description

The present invention relates to an information processing device, a speech recognition system, and a voice recognition program.

In a recent voice recognition system, a machine lip-reading technology (lip reading) for recognizing the utterance content from the movement of the lips of a speaker using image information in order to complement the voice information is already known.

In addition, as one of the techniques of using image information for voice recognition, a face image captured by a wide-angle photographing device is converted into a plane regular image, and a lip region is extracted according to the distance between the participant and the wide-angle photographing device. There is known a technique for setting a magnification at the time.

In a meeting or the like, the distance between the imaging device and the speaker changes depending on the position and posture of the speaker sitting, the movement of the speaker, and the like. Therefore, in the image information input as the interpolation of the voice information, there is no guarantee that the size of the lip area of the speaker is always constant, and the resolution of the image information input to the recognizer varies, and the recognition of the utterance content It was difficult to improve the accuracy of.

The disclosed technology aims to improve the recognition accuracy of utterance content.

The disclosed technology recognizes a region indicating a person's lip from an input unit to which moving image data captured by an image capturing device is input and an image for each frame included in the moving image data, and detects a continuous lip image of the person. Based on the lip area extraction unit that extracts the lip area image data indicating, and the attribute information added to the lip area image data, a recognition model used to recognize the utterance content of the person is selected from a plurality of recognition models. An information processing apparatus including: a recognition model selection unit, a speech recognition unit that recognizes the speech content of the person using the selected recognition model, and an output unit that outputs a recognition result of the speech content.

It is possible to improve the recognition accuracy of the utterance content.

It is a figure explaining the speech recognition system of 1st embodiment. It is a figure which shows an example of the hardware constitutions of the information processing apparatus of 1st embodiment. It is a figure explaining the function of the information processing apparatus of 1st embodiment. 6 is a first flowchart illustrating a process of the information processing device according to the first embodiment. 6 is a second flowchart illustrating the processing of the information processing apparatus according to the first embodiment. It is a figure explaining the lip image of 1st embodiment. It is a figure explaining selection of a recognition model of a first embodiment. It is a 1st figure explaining the recognition model of 1st embodiment. It is a 2nd figure explaining the recognition model of 1st embodiment. It is a figure explaining the function of the information processor of a second embodiment. 9 is a first flowchart illustrating a process of the information processing device according to the second embodiment. It is a 1st figure explaining the recognition model of 2nd embodiment. It is a 2nd figure explaining the recognition model of 2nd embodiment. It is a figure explaining selection of a recognition model of a third embodiment.

(First embodiment)
The first embodiment will be described below with reference to the drawings. FIG. 1 is a diagram illustrating a speech recognition system according to the first embodiment.

The utterance recognition system 100 of this embodiment includes an information processing device 200 and an imaging device 300. In the utterance recognition system 100, the information processing device 200 and the imaging device 300 are connected by wire or wirelessly.

FIG. 1 shows an example in which a conference is held by three participants A, B, and C, and the utterance recognition system 100 recognizes the utterance contents of each of the participants A, B, and C.

The information processing apparatus 200 performs basic control of the system including recognition of utterance content. The information processing apparatus 200 is connected to, for example, a network or the Internet, and the image data captured by the imaging apparatus 300 may be transmitted to a server on the network or a cloud server on the Internet. In this case, the information processing device 200 may receive the recognition result of the utterance content made in the server or the cloud server.

The information processing apparatus 200 according to the present embodiment visualizes the utterance content by displaying the recognition result of the utterance content of the participant on the display device 400. Further, the information processing apparatus 200 of the present embodiment may hold the recognition result of the utterance content as text data and output the text data as the minutes at an arbitrary timing. The arbitrary timing is, for example, when the conference ends.

The display device 400 may be, for example, an electronic blackboard or a display. Further, in the example of FIG. 1, the display device 400 is included in the utterance recognition system 100, but the present invention is not limited to this, and the display device 400 may not be included in the utterance recognition system 100.

In the utterance recognition system 100 of the present embodiment, a recognition model (recognizer) according to the distance between the imaging device 300 and the participant is used for the voice recognition of the participant.

The recognition model is learned for each distance using image data of the lip regions of speakers having different resolutions, which are captured in advance at various distances, and may be included in the information processing apparatus 200.

The information processing apparatus 200 according to the present embodiment can improve the recognition accuracy of the utterance content at the distance by using the recognition model according to the distance between the participant and the imaging apparatus 300.

FIG. 2 is a diagram illustrating an example of a hardware configuration of the information processing device according to the first embodiment. The information processing apparatus 200 of the present embodiment includes an input device 21, an output device 22, a drive device 23, an auxiliary storage device 24, a memory device 25, an arithmetic processing device 26, and an interface device 27, which are connected to each other by a bus B. Including.

The input device 21 is a device for inputting various kinds of information, and is realized by, for example, a keyboard, a pointing device, or the like. The input device 21 may be an interface or the like for inputting image data captured by the image capturing device 300.

The output device 22 is for outputting various kinds of information, and may be, for example, a display or the like, or may be an interface for outputting information to the display device 400. The interface device 27 includes a LAN card and the like, and is used to connect to a network.

The speech recognition program according to the present embodiment is at least a part of various programs that control the information processing device 200. The utterance recognition program is provided by, for example, distributing the storage medium 28 or downloading it from a network. The storage medium 28 recording the speech recognition program is a storage medium for optically, electrically or magnetically recording information such as a CD-ROM, a flexible disk or a magneto-optical disk, or information such as a ROM or a flash memory. It is possible to use various types of storage media such as a semiconductor memory that electrically records the data.

Further, the utterance recognition program is installed in the auxiliary storage device 24 from the storage medium 28 via the drive device 23 when the storage medium 28 recording the utterance recognition program is set in the drive device 23. The speech recognition program downloaded from the network is installed in the auxiliary storage device 24 via the interface device 27.

The auxiliary storage device 24 stores the installed speech recognition program and also stores necessary files and data. The memory device 25 reads the utterance recognition program from the auxiliary storage device 24 and stores it when the information processing device 200 is activated. Then, the arithmetic processing unit 26 realizes various processes as described later according to the speech recognition program stored in the memory unit 25.

Next, functions of the information processing apparatus 200 according to the present embodiment will be described with reference to FIG. FIG. 3 is a diagram illustrating the functions of the information processing apparatus according to the first embodiment.

The information processing apparatus 200 according to the present embodiment includes a video input unit 210, a person area recognition unit 211, an image correction unit 212, a face area recognition unit 213, a lip area extraction unit 214, a lip pixel number calculation unit 215, and a recognition model selection unit 216. A lip pixel number conversion unit 217, a lip feature amount calculation unit 218, an utterance content recognition unit 219, and a text output unit 220.

These units are realized by the arithmetic processing unit 26 of the information processing device 200 reading and executing the utterance recognition program stored in the memory unit 25.

The information processing device 200 also includes a storage unit 230. The storage unit 230 is realized by, for example, the memory device 25 of the information processing device 200, the auxiliary storage device 24, or the like.

Recognition models 231, 232 and 233 are stored in the storage unit 230. The recognition model 231 is used when the distance between the imaging device 300 and the speaker is short. The recognition model 232 is used when the distance between the imaging device 300 and the speaker is a medium distance. The recognition model 233 is used when the distance between the imaging device 300 and the speaker is long.

The video input unit 210 of the present embodiment acquires video data (moving image data) captured by the imaging device 300. The person area recognition unit 211 recognizes an area where a person is present in continuous frame images in the acquired video data and extracts the area as image data. In the following description, the image data extracted by the person area recognition unit 211 is called person area image data, and the image represented by the person area image data is called a person image.

The image correction unit 212 performs brightness correction when the person area image data is too bright or too dark. An existing general technique may be used as the brightness correction method.

When the imaging device 300 is a spherical camera, it is common to combine two super wide-angle images acquired by two lenses and treat them as one image. It is common that the image is in the Equirectangular format. In that case, if perspective correction is performed with the designated correction position as the center, it can be processed as an image without distortion.

The image correction unit 212 of the present embodiment performs perspective correction centered on the coordinates recognized by the person area recognition unit 211, and acquires the image represented by the person area image data as a person's area without distortion. ..

The face area recognition unit 213 recognizes a person's face from the person area image data whose distortion is corrected by the image correction unit 212, and extracts image data of the face area. In the following description, the image data extracted by the face area recognition unit 213 is called face image data, and the image indicated by the face image data is called a face image.

As an algorithm of face recognition by the face area recognition unit 213, there are various existing methods such as a Haar-Like feature quantity classifier and a classifier using HOG feature quantity, and these may be used.

The lip area extracting unit 214 extracts image data of the lip area from the face image data. In the following description, the image data extracted by the lip area extraction unit 214 will be referred to as lip area image data, and the image represented by the lip area image data will be referred to as a lip image. The lip area image data may be data indicating a plurality of lip images.

The lip area extracting unit 214 can extract lip area image data from the recognition result by using a discriminator that can recognize the number of landmarks in the lip area for face recognition by the face area recognizing unit 213, for example. it can.

In the present embodiment, the lip image can be obtained as a continuous image from the video data input to the video input unit 210 by continuously executing the above-described processing for each frame.

The lip pixel number calculation unit 215 calculates the average value of the number of pixels of each horizontal width of the continuous lip image, and adds this average value to the lip area image data as attribute information of the lip area image data. That is, the lip pixel number calculation unit 215 functions as an attribute addition unit that adds attribute information to the lip area image data.

The continuous lip image is a group of lip images extracted from the image of each frame of video data (moving image data).

The recognition model selection unit 216 uses the recognition model used to recognize the utterance content from the recognition models 231, 232, and 233 stored in the storage unit 230 according to the average value calculated by the lip pixel number calculation unit 215. Select. In other words, the recognition model selection unit 216 selects the recognition model based on the attribute information given to the lip area image data.

The average value of the number of horizontal pixels of the continuous lip image corresponds to the distance between the imaging device 300 and the speaker (participant). Therefore, the recognition model selection unit 216 selects the recognition model according to the distance between the imaging device 300 and the speaker (participant).

The lip pixel number conversion unit 217 converts the pixel number of consecutive lip images so as to match the recognition model selected by the recognition model selection unit 216.

The lip feature amount calculation unit 218 acquires spatial information and temporal information as feature amounts from consecutive lip images. Specifically, the feature amount of the present embodiment is an 8-bit RGB value of an image indicated by the number of horizontal pixels and the number of vertical images of continuous lip images for a certain period.

The utterance content recognition unit 219 recognizes the utterance content of the speaker based on the feature amount acquired by the lip feature amount calculation unit 218 and the recognition model selected by the recognition model selection unit 216.

The text output unit 220 outputs the recognition result by the utterance content recognition unit 219 as text data to the display device 400 or the like.

In the example of FIG. 3, the recognition models 231, 232, 233 are stored in the storage unit 230 of the information processing apparatus 200, but the present invention is not limited to this. The recognition models 231, 232, 233 may be stored in a device other than the information processing device 200.

Next, processing of the information processing apparatus 200 according to the first embodiment will be described with reference to FIG. FIG. 4 is a first flowchart illustrating the process of the information processing device according to the first embodiment.

The information processing apparatus 200 of the present embodiment acquires the video data captured by the imaging apparatus 300 by the video input unit 210 (step S401).

Then, the information processing apparatus 200 starts a loop that repeats the processing from step S403 N times (step S402).

In the information processing apparatus 200, the person area recognition unit 211 acquires image data of one frame from the video data acquired by the video input unit 210 (step S403). Subsequently, the person area recognition unit 211 recognizes an area where a person is present from the image data of one frame and extracts the person area image data (step S404).

The person area recognition unit 211 of this embodiment extracts the image data of the rectangular area of the person area image data. Further, although only the recognition process for one person will be described here for simplification of description, it is also assumed that a plurality of persons are recognized. In that case, this series of recognition processing and extraction processing are processed sequentially or in parallel for the number of people.

Subsequently, the information processing apparatus 200 corrects the distortion or the like of the person area image data by the image correction unit 212 (step S405). Subsequently, in the information processing apparatus 200, the face area recognition unit 213 recognizes the face area from the corrected person area image data and extracts the face area image data (step S406).

Subsequently, the information processing apparatus 200 causes the lip area extraction unit 214 to recognize the lip area from the face area image data and extract the lip area image data (step S407). Then, the lip area extracting unit 214 adds the lip image data to the buffer (step S408).

The information processing apparatus 200 repeats the processing from step S403 to step S409 N times (step S409).

Specifically, for example, the information processing apparatus 200 repeats the processing of steps S403 to S409 about 150 times. In this case, for example, when the frame rate is 30 fps, 4 seconds of continuous lip area image data is stored in the buffer. Details of the processing from step S403 to step S409 will be described later. The lip area image data of this embodiment includes a plurality of image data showing a plurality of continuous lip images.

Subsequently, in the information processing apparatus 200, the lip pixel number calculation unit 215 calculates and acquires the average value w of the horizontal pixel numbers of the continuous lip images stored in the buffer (step S410).

Then, the information processing apparatus 200 causes the recognition model selection unit 216 to select the recognition model according to the average value w.

That is, in the present embodiment, the average value w of the number of pixels in the horizontal width of the continuous lip image is the attribute information referred to when the recognition model selection unit 216 selects the recognition model. The average value w may be added to and held in the lip area image data indicating continuous lip images.

Specifically, in the information processing apparatus 200, the recognition model selection unit 216 determines whether the average value w is less than 10 pixels (step S411).

In step S411, when the average value w is less than 10 pixels, the recognition model selection unit 216 determines that the lip image is too small and thus cannot be recognized, and resets the buffer storing continuous lip images (step S412). , And returns to step S402. The case where the lip image is too small is a case where the speaker is too far from the imaging device 300.

When the average value w is less than 10 pixels in step S411, the recognition model selection unit 216 determines whether the average value w is 10 pixels or more and less than 25 pixels (step S413).

When the average value w is 10 pixels or more and less than 25 pixels in step S413, the recognition model selection unit 216 sets the recognition model 231 among the recognition models stored in the storage unit 230 (step S414), which will be described later. It proceeds to step S418. In other words, the recognition model selection unit 216 selects the recognition model 231 according to the average value w that is the attribute information given to the lip area image data indicating the continuous lip images stored in the buffer.

The recognition model 231 is a recognition model for long distances, which is selected when the size of the lip region is small and the distance from the speaker to the imaging device 300 is long, but the recognition is possible.

In step S413, if the average value w is not 10 pixels or more and less than 25 pixels, that is, if the average value w is 25 pixels or more, the recognition model selection unit 216 determines whether the average value w is 25 pixels or more and less than 40 pixels. It is determined whether or not (step S415).

When the average value w is 25 pixels or more and less than 40 pixels in step S415, the recognition model selection unit 216 sets the recognition model 232 of the recognition models stored in the storage unit 230 (step S416), which will be described later. It proceeds to step S418.

The recognition model 232 is a recognition model for the medium distance, which is selected when the size of the lip region is medium and the distance from the speaker to the imaging device 300 is medium.

In step S415, when the average value w is not 10 pixels or more and less than 25 pixels, that is, when the average value w is 40 pixels or more, the recognition model selection unit 216 sets the recognition model 233 (step S417), which will be described later. It proceeds to step S418.

The recognition model 233 is a short-distance recognition model that is selected when the lip area is large and the distance from the speaker to the imaging device 300 is short.

Subsequently, the information processing device 200 causes the lip pixel number conversion unit 217 to resize the continuous lip images stored in the buffer according to the selected recognition model (step S418).

The recognition models 231, 232, and 233 of the present embodiment are network parameters adjusted by deep learning using a long-distance image, a medium-distance image, and a short-distance image.

The horizontal width of the image represented by the image data input to the long-distance recognition model 231 needs to be 10 pixels. Similarly, the horizontal width of the image represented by the image data input to the recognition model 232 for medium distance needs to be 30 pixels, and the horizontal width of the image represented by the image data input to the recognition model 233 needs to be 50 pixels.

The lip pixel number conversion unit 217 of the present embodiment performs resizing of the lip image so that the image data indicating the lip image can be input to the selected recognition model. Specifically, the lip pixel number conversion unit 217 may convert the resolution of the lip image.

Subsequently, the information processing apparatus 200 causes the lip feature amount calculation unit 218 to acquire the feature amount of the lip image (step S419).

Subsequently, the information processing apparatus 200 recognizes the utterance content by inputting the resized lip image data and the feature amount to the selected recognition model by the utterance content recognition unit 219 (step S420).

Subsequently, the information processing apparatus 200 causes the text output unit 220 to output the recognition result as text data to the display device 400 or the like (step S421) and reset the buffer (step S422).

Subsequently, the information processing apparatus 200 determines whether or not a processing end instruction has been received (step S423). When the processing end instruction is received in step S423, the information processing apparatus 200 ends the processing. If the end instruction is not accepted in step S423, the information processing apparatus 200 returns to step S402.

Next, the loop processing shown in FIG. 4 will be further described with reference to FIG. FIG. 5 is a second flowchart illustrating the process of the information processing device according to the first embodiment.

The information processing apparatus 200 according to the present embodiment initializes the value of the counter for counting the number of lip images required for input to the recognition model (step S501).

Then, the information processing apparatus 200 acquires the video data imaged by the imaging device 300 by the video input unit 210 (step S502).

Subsequently, the information processing apparatus 200 acquires an image for one frame by the person area recognition unit 211 and recognizes a person in the image (step S503).

Subsequently, the information processing apparatus 200 determines whether or not the person is recognized in step S503 (step S504). If the person is not recognized in step S504, the person area recognition unit 211 determines whether or not the speaker final position information can be referred to (step S505). The speaker final position information is information indicating the position at which the speaker was last recognized in any of the images included in the video data.

In step S505, if the speaker last position information cannot be referred to, that is, if the speaker last position information is the initial value, the information processing apparatus 200 determines that the speaker is not in the vicinity thereof and returns to step S501.

If the speaker final position information can be referred to in step S505, the information processing apparatus 200 proceeds to step S507 described later.

When a person is recognized in step S504, the person area recognition unit 211 updates the speaker final position information corresponding to this person (step S506).

Subsequently, the person area recognition unit 211 specifies the person area from the image data based on the speaker final position information, and extracts the person area image data indicating the person image (step S507). The information processing apparatus 200 performs the correction by the image correction unit 212 after extracting the person area image data.

Subsequently, in the information processing apparatus 200, the face area recognition unit 213 performs face recognition on the person area image data (step S508), and determines whether a face is recognized (step S509).

If the face is not recognized in step S509, it is determined whether the face final position information can be referred to (step S510). The face final position information is information indicating the final position of the face of the speaker in the person image. If the face final position information cannot be referred to in step S510, the information processing apparatus 200 returns to step S501.

When it is possible to refer to the face final position information in step S510, the information processing apparatus 200 proceeds to step S512 described below.

When a face is recognized in step S509, the face area recognition unit 213 extracts face area image data indicating a face image from the person area image data based on the face final position information (step S512).

Then, the information processing apparatus 200 causes the lip area extraction unit 214 to extract lip area image data indicating a lip image from the face area image data (step S513). Subsequently, the information processing apparatus 200 adds 1 to the value of the counter to count the number of acquired current frames (step S514), and adds the lip area image data to the buffer (step S515).

Subsequently, the information processing apparatus 200 determines whether or not the number of acquired frames has reached the number of frames required for input to the recognition model (step S516). In other words, the information processing device 200 determines whether or not the value of the counter has reached the number of frames required for input to the recognition model. In the example of FIG. 5, the number of frames required for input to the recognition model is 150, but the number is not limited to this.

In step S516, when the required number of frames has not been reached, the information processing apparatus 200 returns to step S502.

When the required number of frames has been reached in step S516, the information processing apparatus 200 initializes the speaker final position information in order to align the data of the next speaker (step S517). Subsequently, the information processing apparatus 200 initializes the face final position information (step S518), and ends the processing for one utterance recognition.

As described above, in the present embodiment, the recognition model used for speech recognition is selected and speech recognition is performed for each speech according to the width of the lip image indicating the distance between the speaker and the image capturing apparatus 300. The accuracy of speech recognition can be improved. Further, by combining the present embodiment with speech recognition using voice information, the accuracy of speech recognition can be improved.

Next, the lip image of the present embodiment will be further described with reference to FIG. FIG. 6 is a diagram illustrating a lip image according to the first embodiment.

An image 61 shown in FIG. 6 shows an example of an image captured by the image capturing apparatus 300, which is a spherical camera. This image 61 is a distorted image in the Equirectangular format.

In the image 61, the three participants (speakers) A, B, and C of the conference surround the table, and the participant A is located near the imaging device 300, the participant B is located in the middle position, and the participant is located far from the imaging device 300. Participant C is seated.

In this embodiment, the person area recognition unit 211 performs the person area recognition process on the image 61 to extract the person area image data indicating the rectangular person images 611, 612, and 613.

Here, since the human images 611, 612, and 613 are distorted images, the image correction unit 212 performs perspective correction based on the center coordinates of the human image. By this correction, the distorted person images 611, 612, and 613 become the distorted corrected person images 611A, 612A, and 613A.

In the present embodiment, the face area recognition unit 213 performs face area recognition processing on the corrected person images 611A, 612A, and 613A to extract face area image data indicating a face image, and further, the face area image data. Then, the lip area recognition processing is performed by the lip area extracting unit 214.

As a result, the lip area extraction unit 214 extracts lip area image data indicating the lip images 621, 622, and 623.

Next, selection of a recognition model by the recognition model selection unit 216 will be described with reference to FIG. 7. FIG. 7 is a diagram illustrating selection of a recognition model according to the first embodiment.

In the present embodiment, if the average value of the number of pixels in the horizontal width of continuous lip images is less than 10 pixels, it is regarded as unrecognizable and is outside the applicable range of the recognition model.

Further, in the present embodiment, when the average value of the number of horizontal pixels of the continuous lip image is 10 pixels or more and less than 25 pixels, the recognition model selection unit 216 selects the recognition model 231 for long distance. Then, in the present embodiment, the lip pixel number conversion unit 217 reduces the average number of pixels in the lateral width of the lip image indicated by the lip region image data input to the recognition model 231 to 10 pixels.

Since human lips are horizontally long, the number of pixels in the vertical direction may be 5 pixels. When the number of pixels in the vertical direction is 5, the recognition model 231 has been learned using image data of 10×5 pixels.

Further, in the present embodiment, the recognition model selection unit 216 selects the recognition model 232 for medium distance when the average value of the number of horizontal pixels of the continuous lip image is 25 pixels or more and less than 50 pixels. .. Then, when the recognition model 232 is selected, the lip pixel number conversion unit 217 performs resizing of enlarging or reducing the lip image to 30×15 pixels.

Further, in the present embodiment, the recognition model selection unit 216 selects the short-distance recognition model 233 when the average value of the number of pixels in the horizontal width of the continuous lip image is 50 pixels or more. Then, when the recognition model 233 is selected, the lip pixel number conversion unit 217 resizes the lip image so that the lip image has a size of 50×25 pixels.

The resizing method by the lip pixel number conversion unit 217 may be an existing method such as a nearest neighbor method, a bilinear interpolation method, or a bicubic interpolation method.

Next, the recognition model of the present embodiment will be described with reference to FIG. FIG. 8 is a first diagram illustrating the recognition model of the first embodiment.

In FIG. 8, the vertical axis represents the recognition accuracy, and the horizontal axis represents the average value of the number of pixels in the horizontal width of the input continuous lip image.

In FIG. 8, a lip region showing a continuous lip image having an average horizontal pixel count of 150 pixels or less is obtained for a recognition model learned using a continuous lip image having an average horizontal pixel count of 150 pixels. This shows the recognition accuracy when image data is input.

As can be seen from the results, when the average value of the horizontal width pixels of the lip image indicated by the input lip area image data is 50 pixels or more, the average value of the horizontal width pixels of the lip image is recognized as 50 pixels. There is no difference in recognition accuracy from the case.

However, when the lip area image data indicating the lip image with the average number of horizontal pixels of the lip image being less than 50 pixels is input to this recognition model, the feature amount of the image data is lost and the gap with the model is lost. It can be seen that the accuracy of the recognition is lowered.

Therefore, attention is paid to the case where the lip area image data in which the average value of the number of horizontal pixels of the lip image is less than 50 pixels is input.

FIG. 9 is a second diagram illustrating the recognition model of the first embodiment. In FIG. 9, when lip area image data indicating a lip image having a different average width pixel number is input to a recognition model learned using a lip image having an average width pixel number less than 50 pixels. Is shown.

In FIG. 9, for example, when the lip area image data indicating a lip image having an average horizontal pixel count of 25 pixels is input, learning is performed using a lip image having an average horizontal pixel count of 25 pixels. Using the recognized recognition model, the recognition accuracy is highest.

Further, when the lip area image data indicating the lip image having an average width of 50 pixels or more is input, a recognition model learned using a lip image having an average width of 50 pixels. Is the most accurate recognition.

As described above, in the present embodiment, the recognition model used for recognizing the utterance content is selected for each utterance, according to the average value of the number of pixels in the width of the lip image corresponding to the distance between the imaging device 300 and the speaker. As a result, even in a situation where the distance between the speaker and the camera changes, such as in a meeting place, it is possible to improve the accuracy of recognition of the utterance content performed in real time.

(Second embodiment)
The second embodiment will be described below with reference to the drawings. The second embodiment is different from the first embodiment in that the recognition model is selected according to the frame rate when acquiring the lip region image data. Therefore, in the following description of the second embodiment, the differences from the first embodiment will be described, and those having the same functional configuration as the first embodiment will be described in the first embodiment. The same reference numerals as those used are given and the description thereof is omitted.

FIG. 10 is a diagram illustrating the functions of the information processing apparatus according to the second embodiment.

The information processing apparatus 200A of the present embodiment includes a video input unit 210, a person area recognition unit 211, an image correction unit 212, a face area recognition unit 213, a lip area extraction unit 214, a lip pixel number calculation unit 215, and a recognition model selection unit 216A. In addition to the lip pixel number conversion unit 217, the lip feature amount calculation unit 218, the utterance content recognition unit 219, and the text output unit 220, a frame rate calculation unit 221 and a frame complementation unit 222 are included.

Further, the information processing device 200A of the present embodiment has a storage unit 230A. Recognition models 241, 242, and 243 are stored in the storage unit 230A.

The frame rate calculation unit 221 of the present embodiment calculates the value of the frame rate that changes from moment to moment and adds the frame rate as attribute information of the lip region image data. That is, the frame rate calculation unit 221 functions as an attribute addition unit that adds attribute information to the lip area image data.

The frame rate indicates the number of frames to be processed per unit time in the moving image acquired by the imaging device 300, and changes depending on the overall processing load of the utterance recognition system 100, the specifications of the information processing device 200A, and the like.

The recognition model selection unit 216A selects a recognition model according to the frame rate calculated by the frame rate calculation unit 221. In other words, the recognition model selection unit 216A selects the recognition model based on the attribute information given to the lip area image data indicating the continuous lip images.

The frame complementing unit 222 of this embodiment complements a frame according to the recognition model selected by the recognition model selecting unit 216. As the thinning-out method and the complementing method performed by the frame complementing unit 222, simple methods such as simply copying the previous frame and excluding unnecessary portions can be considered. Further, the frame complementing unit 222 may use an existing method of obtaining an intermediate value from the difference between the pixel values of the preceding and following frame images and generating a new likely intermediate frame.

The recognition models 241, 242, 243 stored in the storage unit 230A of the present embodiment are recognition models learned with the lip area image data indicating continuous lip images acquired at different frame rates as an input.

Specifically, the recognition model 241 is a recognition model learned by inputting lip region image data indicating continuous lip images acquired at a high frame rate. Further, the recognition model 242 is a recognition model learned by inputting lip area image data indicating continuous lip images acquired at a medium frame rate. Further, the recognition model 243 is a recognition model learned by inputting lip area image data indicating continuous lip images acquired at a low frame rate.

Next, with reference to FIG. 11, processing of the information processing apparatus 200A of the present embodiment will be described. FIG. 11 is a first flowchart illustrating the process of the information processing device according to the second embodiment.

The information processing apparatus 200A of the present embodiment acquires the video data captured by the imaging apparatus 300 by the video input unit 210 (step S1101).

Subsequently, the information processing device 200A starts a timer (step S1102). In the present embodiment, for example, the frame rate is calculated by the number of times the loop from step S1103 to step S1110, which will be described later, is repeated while the timer measures 4 seconds. For example, when the loop is repeated 150 times, the frame rate is 30 fps, and when the loop is repeated 50 times, the frame rate is 10 fps. The calculation of the frame rate is performed in step S1112 described later.

The processing from step S1103 to step S1110 in FIG. 11 is the same as the processing from step S402 to step S409 in FIG.

Following step S1110, the information processing apparatus 200A stops the timer when the time set in the timer has elapsed (step S1111).

Subsequently, in the information processing apparatus 200A, as described above, the frame rate calculation unit 221 calculates and acquires the frame rate based on the number of times the loop is repeated within the time measured by the timer (step S1112).

The frame rate calculated by the frame rate calculation unit 221 is attribute information referred to when the recognition model selection unit 216A selects the recognition model, and is attached to the lip area image data and held.

Subsequently, the information processing apparatus 200A determines whether the frame rate is lower than 3 fps by the recognition model selection unit 216A (step S1113).

When the frame rate is less than 3 fps in step S1113, the recognition model selection unit 216A determines that recognition at this frame rate is impossible, resets the timer and the buffer (step S1114), and returns to step S1101.

In step S1113, if the frame rate is not lower than 3 fps, that is, if the frame rate is 3 fps or higher, the recognition model selection unit 216A determines whether the frame rate is 3 fps or higher and lower than 5 fps (step S1115).

When the frame rate is 3 fps or more and less than 5 fps in step S1115, the recognition model selection unit 216A sets the recognition model 243 as the frame rate is considered to be low (step S1116), and proceeds to step S1120 described later.

In step S1115, when the frame rate is not 3 fps or more and less than 5 fps, that is, when the frame rate is 5 fps or more, the recognition model selection unit 216A determines whether the frame rate is 5 fps or more and less than 10 fps (step S1117). ).

When the frame rate is 5 fps or more and less than 10 fps in step S1117, the recognition model selection unit 216A sets the recognition model 242 with the frame rate being medium (step S1118), and proceeds to step S1120 described below.

In step S1117, if the frame rate is not 5 fps or more and less than 10 fps, that is, if the frame rate is 10 fps or more, the recognition model selection unit 216A sets the recognition model 241 as a high frame rate (step S1119), and will be described later. Proceed to step S1120.

Subsequently, the information processing apparatus 200A causes the lip pixel number conversion unit 217 to resize the continuous lip images stored in the buffer according to the selected recognition model (step S1120).

In the present embodiment, the resolution when resizing the lip image may be constant regardless of the selected recognition model, or may be combined with the processing of the first embodiment.

Subsequently, the information processing apparatus 200A uses the frame complementing unit 222 to complement the lip area image data indicating continuous lip images in the buffer according to the selected recognition model and the acquired frame rate (step S1121). , And proceeds to step S1122. Note that the complement of the present embodiment includes a process of thinning out image data.

The processing of steps S1122 to S1124 of FIG. 11 is the same as the processing of steps S419 to event 421 of FIG. 4, so description thereof will be omitted.

The information processing apparatus 200A resets the timer and the buffer following step S1124 (step S1125), and determines whether or not a processing end instruction has been received (step S1126).

When the processing end instruction is received in step S1126, the information processing apparatus 200A ends the processing. If the end instruction is not accepted in step S1126, the information processing apparatus 200A returns to step S1101.

The information processing apparatus 200A of the present embodiment continuously recognizes the utterance content using the lip image by continuously repeating the processing of FIG.

Next, with reference to FIG. 12, the recognition model of this embodiment will be described. FIG. 12 is a first diagram illustrating the recognition model of the second embodiment.

In FIG. 12, the vertical axis represents the recognition accuracy, and the horizontal axis represents the frame rate when the input continuous lip images are acquired.

In FIG. 12, in the case of inputting lip area image data showing continuous lip images when the frame rate is 10 fps or less, with respect to the recognition model learned using continuous lip images, obtained at a frame rate of 30 fps. Shows the recognition accuracy of.

In the recognition model shown in FIG. 12, when the frame rate of the input lip area image data is 10 fps or more, there is no difference in recognition accuracy.

However, when the frame rate of the input lip area image data is set to less than 10 fps, the feature amount and temporal information of the lip area image data are lost, a gap with the model is generated, and recognition accuracy is lowered.

Therefore, attention is paid to the case where the frame rate of the input lip area image data is set to less than 10 fps.

FIG. 13 is a second diagram illustrating the recognition model of the second embodiment. FIG. 13 shows a case in which lip region image data having different frame rates are input to the recognition model learned by setting the frame rate of the input lip region image data to be less than 10 fps.

In FIG. 13, for example, when the lip area image data having a frame rate of 5 fps is input, the recognition accuracy obtained by using the recognition model learned using the lip area image data having a frame rate of 5 fps is the highest. Get higher

Further, when the lip area image data having a frame rate of 10 fps or higher is input, there is no difference in recognition accuracy regardless of whether the frame rate is 10 fps or 30 fps. Further, it can be seen that the recognition accuracy is highest when the recognition model learned using the lip area image data having the frame rate of 10 fps is used. Therefore, when the lip area image data having a frame rate of 10 fps or higher is input, a recognition model learned using the lip area image data having a frame rate of 10 fps may be used.

Further, when the lip area image data having a frame rate of less than 1 fps is input, the recognition accuracy is extremely low, and thus the recognition is impossible in this embodiment. In this case, it is conceivable to recognize the utterance content by voice information.

As described above, in the present embodiment, since the recognition model used for recognizing the utterance content is selected according to the frame rate when acquiring the lip image, the utterance content of the utterance recognition system 100 is selected according to the communication situation of the utterance recognition system 100. The accuracy of recognition can be improved.

(Third embodiment)
The third embodiment will be described below with reference to the drawings. The third embodiment is different from the first embodiment in that the recognition model is selected according to the orientation of the speaker's face. The third embodiment will be described below with reference to FIG.

FIG. 14 is a diagram illustrating selection of a recognition model according to the third embodiment.

The speaker does not always speak in the direction of the imaging device 300, but often speaks by looking at the direction of the display device 400 or another speaker.

In that case, the lip image has a smaller width than the image 143 in the case where the speaker faces the imaging device 300, as in the images 141 and 142 shown in FIG. 14. In this case, the width does not become narrow due to the distance between the imaging device 300 and the speaker.

Therefore, in the present embodiment, for example, a recognition model learned in advance by using continuous lip images facing rightward and leftward is prepared in advance, and when face recognition is performed by the face area recognition unit 213, the orientation of the face is estimated, Select a recognition model suitable for each orientation.

In this embodiment, in this way, by selecting the recognition model according to the orientation of the face, it is possible to suppress a decrease in recognition accuracy.

Although the present invention has been described above based on the respective embodiments, the present invention is not limited to the requirements shown in the above embodiments. With respect to these points, the gist of the present invention can be modified within a range that does not impair the invention, and can be appropriately determined according to the application mode.

100 Speech recognition system 200, 200A Information processing device 210 Video input unit 211 Human region recognition unit 212 Image correction unit 213 Face region recognition unit 214 Lip region extraction unit 215 Lip pixel number calculation unit 216, 216A Recognition model selection unit 217 Lip pixel number Conversion unit 218 Lip feature amount calculation unit 219 Speech content recognition unit 220 Text output unit 221 Frame rate calculation unit 222 Frame complement unit 230, 230A Storage unit 231, 232, 233, 241, 242, 243 Recognition model 300 Imaging device 400 Display device

Japanese Unexamined Patent Application Publication No. 2015-019162

Claims (9)

An input unit for inputting moving image data captured by the image capturing device;
From each image included in the moving image data, a lip area extracting unit that recognizes an area indicating a lip of a person and extracts lip area image data indicating a continuous lip image of the person,
A recognition model selection unit that selects a recognition model used to recognize the utterance content of the person from among a plurality of recognition models based on attribute information given to the lip area image data,
A speech recognition unit that recognizes the speech content of the person using the selected recognition model;
An information processing device comprising: an output unit that outputs a recognition result of the utterance content. A lip pixel number calculation unit for calculating an average value of the number of horizontal pixels of the continuous lip image,
The information processing apparatus according to claim 1, wherein the average value is attribute information added to the lip area image data. A storage unit in which the plurality of recognition models are stored,
The plurality of recognition models are
The information processing apparatus according to claim 2, wherein each of the models is a model learned by using lip area image data extracted from moving image data of the person captured with different distances between the image capturing apparatus and the person. A frame rate calculation unit that calculates a frame rate in the moving image indicated by the moving image data,
The information processing apparatus according to claim 1, wherein the frame rate is used as the attribute information. A storage unit in which the plurality of recognition models are stored,
The plurality of recognition models are
The information processing apparatus according to claim 4, wherein each of the models is a model in which lip area image data showing continuous lip images acquired at different frame rates is learned as an input. 6. The lip pixel number conversion unit for converting the resolution of the continuous lip image into different resolutions so that the lip region image data becomes input data of the selected recognition model. The information processing device according to item. A feature amount calculation unit that calculates an 8-bit RGB value of an image indicated by the number of horizontal pixels and the number of vertical images of the continuous lip image for a certain period as a feature amount;
The speech recognition unit,
The information processing apparatus according to claim 1, wherein the utterance content is recognized using the selected recognition model and the feature amount. A speech recognition system having an imaging device and an information processing device,
The information processing device,
An input unit for inputting moving image data captured by the image capturing device;
From each image included in the moving image data, a lip area extracting unit that recognizes an area indicating a lip of a person and extracts lip area image data indicating a continuous lip image of the person,
A recognition model selection unit that selects a recognition model used to recognize the utterance content of the person from among a plurality of recognition models based on attribute information given to the lip region image data,
A speech recognition unit that recognizes the speech content of the person using the selected recognition model;
An utterance recognition system, comprising: an output unit that outputs a recognition result of the utterance content. In the information processing device,
A process of inputting moving image data captured by the image capturing device,
From each image included in the moving image data, a process of recognizing a region indicating the lips of a person, and extracting lip region image data indicating a continuous lip image of the person,
Based on the attribute information added to the lip area image data, a process of selecting a recognition model used for recognition of the utterance content of the person from a plurality of recognition models,
Recognizing the utterance content of the person using the selected recognition model,
An utterance recognition program that executes a process of outputting a recognition result of the utterance content.

Images