JP2022144417A - Hearing support device, hearing support method and hearing support program - Google Patents

Abstract

To provide a technique supporting hearing in a situation in which a plurality of people utters.SOLUTION: A hearing support device includes a mixed voice acquisition unit for acquiring mixed voice information including utterance of a plurality of speakers from the other device; an individual voice acquisition unit for acquiring individual voice information including utterance by any speaker from a prescribed voice input device; a speaker separation unit for acquiring a text of first utterance obtained by separating utterances from the mixed voice information, a text of second utterance which specifies one of speaker's utterances from the individual voice information; an utterance similarity determination unit for determining similarity of the text of the first utterance and the text of the second utterance; and a parallel display generation unit for generating a display screen which displays an utterance list in which the text of the first utterance and the text of the second utterance are arranged in a chronological order.SELECTED DRAWING: Figure 1

Description

TECHNICAL FIELD The present invention relates to a listening support device, a listening support method, and a listening support program.

Patent Literature 1 discloses that "correlation between each speaker's mouth shape included in video data of a plurality of speakers and each speech segment from the speaker included in audio data is calculated, and the calculated correlation A speaker diarization method that builds a speaker model for each speaker based on the relationship, and identifies the speaker uttering the speech included in the audio data based on the built speaker model. .

JP 2020-187346 A

The above technology can identify the speaker of the voice included in the audio data using a video containing the shape of the speaker's mouth. It is not possible to support listening in an environment where multiple people speak, such as an online teleconference that does not use data.

SUMMARY OF THE INVENTION An object of the present invention is to provide a technique for assisting listening in a situation where a plurality of people are speaking.

The present application includes a plurality of means for solving at least part of the above problems, and examples thereof are as follows. A listening support device according to an aspect of the present invention includes a mixed speech acquisition unit that acquires mixed speech information including utterances of a plurality of speakers from another device, and individual speech information including the utterances by any of the speakers. from a predetermined voice input device, a text of a first utterance obtained by separating the utterance from the mixed voice information, and any of the utterances of the speaker from the individual voice information a speaker separation unit for obtaining a text of the second utterance that specifies the text of the second utterance; a utterance similarity determination unit that determines similarity between the text of the first utterance and the text of the second utterance; A parallel display creation unit for creating a display screen for displaying together a message list in which the text of the first message and the text of the second message are arranged in chronological order.

Further, for example, in the above-described listening support device, the parallel display creation unit may add the text of the first utterance determined by the utterance similarity determination unit to be similar to any of the texts of the second utterance as the text of the first utterance. The text of the first utterance that is not determined to be similar to any of the texts of the two utterances may be displayed in a different manner.

Further, for example, in the above-described listening support device, the parallel display creation unit may add the text of the first utterance determined by the utterance similarity determination unit to be similar to any of the texts of the second utterance as the text of the first utterance. The visibility may be kept lower than the text of the first utterance that was not determined to be similar to any of the texts of the two utterances.

Further, for example, in the above-described listening support device, the parallel display creation unit may select the text of the first utterance that is not determined to be similar to any of the texts of the second utterance by the utterance similarity determination unit. The text of the first utterance determined to be similar to any of the text of the second utterance may be highlighted over the text.

Further, for example, in the above listening support device, the mixed speech acquisition unit acquires the mixed speech information in real time, the individual speech acquisition unit acquires the individual speech information in real time, and the parallel display creation unit , when displaying the text of the first utterance that is not determined to be similar to any of the texts of the second utterance by the utterance similarity determination unit, a speaker different from the speaker related to the individual voice information is displayed. may be displayed in the utterance list of the text of the second utterance.

Further, for example, in the listening support device described above, the individual voice acquisition unit may acquire a plurality of the individual voice information from a plurality of predetermined voice input devices.

Further, for example, in the above-described listening support device, the utterance similarity determination unit may determine similarity between the text of the first utterance and the text of the second utterance at approximately the same time.

Further, for example, in the above-described listening support device, the mixed speech information and the individual speech information may be speech of the same online conference, and the other device may be a device controlling the online conference.

A listening support method according to another aspect of the present invention is a listening support method using a listening support device, wherein the hearing support device includes a processing unit, and the processing unit includes a plurality of speakers' utterances. a mixed speech acquiring step of acquiring from another device mixed speech information containing the above; an individual speech acquiring step of acquiring individual speech information containing the utterance by any of the speakers from a predetermined speech input device; and the mixed speech a speaker separation step of obtaining a text of a first utterance obtained by separating the utterance from the information and a text of a second utterance specifying the utterance of one of the speakers from the individual voice information; , an utterance similarity determination step of determining similarity between the text of the first utterance and the text of the second utterance, the text of the first utterance, and the text of the second utterance, respectively; and a parallel display creating step of creating a display screen for displaying the utterance list arranged in chronological order.

A listening support program according to another aspect of the present invention is a listening support program that causes a computer to function as a listening support device, wherein the computer includes a processor, and the processor is configured to transmit utterances of a plurality of speakers. a mixed speech acquisition step of acquiring from another device mixed speech information including the above-mentioned speech information; an individual speech acquisition step of acquiring individual speech information including the utterance by one of the speakers from a predetermined speech input device; and the mixed speech information a speaker separation step of obtaining a text of a first utterance obtained by separating said utterance from said individual voice information and a text of a second utterance specifying said utterance of any of said speakers from said individual speech information; an utterance similarity determination step of determining similarity between the text of the first utterance and the text of the second utterance; the text of the first utterance and the text of the second utterance; and a parallel display creation step of creating a display screen for displaying the series of statement lists together.

According to the present invention, it is possible to provide a technique for assisting listening in a situation where a plurality of people are speaking without using video data.

Problems, configurations, and effects other than those described above will be clarified by the following description of the embodiments.

1 is a block diagram illustrating the configuration of a remote conference support system; FIG. It is a figure which shows the example of a data structure of a microphone user memory|storage part. It is a figure which shows the example of a data structure of a whole conversation record memory|storage part. It is a figure which shows the data structure example of the speech memory|storage part classified by speaker. It is a figure which shows the hardware structural example of a listening assistance apparatus. It is a figure which shows the example of a conference recording flow. It is a figure which shows the example of an utterance recognition processing flow. It is a figure which shows the example of a conversation confirmation screen. It is a figure which shows another example of a conversation confirmation screen.

A remote conference support system 1, which is a listening support system to which an embodiment according to one aspect of the present invention is applied, will be described below with reference to the drawings. In principle, the same members are denoted by the same reference numerals in all drawings for describing the embodiments, and repeated description thereof will be omitted. In addition, in the following embodiments, the constituent elements (including element steps, etc.) are not necessarily essential, unless otherwise specified or clearly considered essential in principle. Needless to say. In addition, when saying "consisting of A", "consisting of A", "having A", or "including A", other elements are excluded unless it is explicitly stated that only those elements are included. It goes without saying that it is not something to do. Similarly, in the following embodiments, when referring to the shape, positional relationship, etc. of components, etc., unless otherwise specified or in principle clearly considered otherwise, the shape is substantially the same. It shall include things that are similar or similar to, etc.

In the following description, "display unit 220" and "browser unit" may be one or more interface devices. The one or more interface devices may be at least one of the following:
- One or more I/O (Input/Output) interface devices. The I/O interface device is an interface device for at least one of the I/O device and the listening support device 100 . The I/O interface device for listening assistance apparatus 100 may be a communication interface device. The at least one I/O device may be any of a user interface device, eg, an input device such as a keyboard and pointing device, and an output device such as a display device.
- One or more communication interface devices. The one or more communication interface devices may be one or more of the same type of communication interface device (e.g., one or more NICs (Network Interface Cards)) or two or more different types of communication interface devices (e.g., NIC and It may be an HBA (Host Bus Adapter).

Also, in the following description, "memory" refers to one or more memory devices, which are examples of one or more storage devices, and may typically be a main memory device. At least one memory device in the memory may be a volatile memory device or a non-volatile memory device.

Also, in the following description, "storage unit" or "storage" may be memory or both of memory and permanent storage. In particular, the permanent storage device may be, for example, a HDD (Hard Disk Drive), an SSD (Solid State Drive), an NVME (Non-Volatile Memory Express) drive, or an SCM (Storage Class Memory).

Also, in the following description, a "processing unit" or "processor" may be one or more processor devices. The at least one processor device may typically be a microprocessor device such as a CPU (Central Processing Unit), but may be another type of processor device such as a GPU (Graphics Processing Unit). At least one processor device may be single-core or multi-core. At least one processor device may be a processor core. At least one processor device is a circuit (for example, FPGA (Field-Programmable Gate Array), CPLD (Complex Programmable Logic Device) or ASIC (Application A processor device in a broad sense such as Specific Integrated Circuit) may also be used.

In addition, in the following description, the function may be described using the expression “yyy part”, but the function may be realized by executing one or more computer programs by a processor, or may be realized by executing one or more computer programs. It may be realized by the above hardware circuits (for example, FPGA or ASIC), or may be realized by a combination thereof. When a function is realized by executing a program by a processor, the defined processing is performed using a storage device and/or an interface device as appropriate, so the function may be at least part of the processor. good. A process described with a function as the subject may be a process performed by a processor or a device having the processor. Programs may be installed from program sources. The program source may be, for example, a program distribution computer or a computer-readable recording medium (for example, a non-temporary recording medium). The description of each function is an example, and multiple functions may be combined into one function, or one function may be divided into multiple functions.

Further, in the following explanation, the processing may be explained with the subject of "program" or "processing unit", but the processing explained with the program as the subject may be the processing performed by the processor or the device having the processor. . Also, two or more programs may be implemented as one program, and one program may be implemented as two or more programs.

In addition, in the following description, expressions such as "xxx table" and "yyy part" may be used to describe information that can be obtained as an output in response to an input, but the information may be a table of any structure. However, it may be a learning model represented by a neural network, a genetic algorithm, or a random forest that generates an output in response to an input. Therefore, the "xxx table" or "yyy part" can also be called "xxx information". Also, in the following description, the configuration of each table is an example, and one table may be divided into two or more tables, or all or part of two or more tables may be one table. may

Also, in the following description, the "remote conference support system" may be a system configured with one or more physical computers, or a system implemented on a physical computing resource group (for example, cloud infrastructure). (eg, cloud computing system). "Displaying" the display information by the remote conference support system may be displaying the display information on the display device of the computer, or by the computer transmitting the display information to the display computer. (in the latter case the display information is displayed by a display computer).

It is important to recognize the speaker in conversation. Conventionally, in remote meetings, the voice input system is divided for each user, and there is a function such as highlighting the icon of the user who has voice input. Also, in real-time speech recognition processing in a remote conference, text can be displayed after tagging the speaker, assuming that the speech input system is completely separate.

In particular, when a hearing-impaired person participates in a remote conference, it is important for the speaker to be able to visually read the associated utterance. On the other hand, in real-time speech recognition processing, not only is the speech input system completely separate for each user, but there are also cases where multiple people from the same room participate using a shared microphone. Depending on the accuracy of identification, it may not be possible to know who is speaking. Even in such a case, the hearing-impaired person would like to obtain as much information as possible, such as estimating the participants who are not the speakers, as much as possible.

In other words, even in the case where only some speakers can be narrowed down from the voice input system, if it can be estimated that the utterance is not from a speaker whose speakers can be specified from the voice input system, It is important to visually show the estimation results or suggestions.

In the listening support device 100 according to the present embodiment, using the voice information output by the remote conference device 200 that can be mixed with the utterances of all the participants, any of the utterances of the participants using the voice input system that can identify the utterances can be identified. By visually showing the similarity/dissimilarity to Kato, it assists hearing-impaired people in listening.

FIG. 1 is a block diagram illustrating the configuration of a remote conference support system according to this embodiment. In the remote conference support system 1, conference participants typically use a terminal device (participant A terminal 400 such as a notebook computer, tablet, or smart phone) remotely via a network 40 such as the Internet or an intranet. Connect to the conference device 200 and hold a remote conference by conversation with other participants. Alternatively, another participant connects to the remote conference device 200 using the conference room terminal 500 arranged in a predetermined conference room, and conducts a remote conference by conversation with the other participant.

The participant A terminal 400 is provided with a microphone A410, collects the speech of the participant A, and transmits it to the remote conference device 200 in real time. Participant A and participant A terminal 400 are located in a different location (for example, remote location) from conference room terminal 500, and pick up only participant A's speech. Conference room terminal 500 is provided with a plurality of microphones (common microphone 510 or microphone D520), collects the utterances of participants B, C, and D who are present in the conference room, It transmits to the listening assistance device 100 in real time. It is assumed that the microphone D520 is a pin microphone or the like, and exclusively collects the speech of the participant D. The shared microphone 510 is shared among participants B, C, and D, and does not distinguish between sound collecting directions. However, the microphone is not limited to this, and may be a microphone capable of distinguishing the sound collection direction. That is, the voice of participant D is almost certainly input by microphone D520, but depending on the common microphone 510, voice input may not be performed depending on the sound pressure and direction of voice of participant D.

The remote conference device 200 is a so-called online conference system, and includes a conference control unit 210 that controls holding/management/finishing of a conference, a display unit 220 that controls display, a participant A terminal 400, a conference room terminal 500, and and a communication unit 230 that communicates with the listening support device 100 . The conference control unit 210 associates and stores the identification information (microphone ID) of the microphone A 410 and the participant A who is the user of the microphone A 410 each time a conference is held. The conference control unit 210 also associates and stores the identification information (microphone ID) of the microphone D520 and the participant D who is the user of the microphone D520 each time the conference is held. Note that the conference control unit 210 integrates the voices input from the microphone A410, the shared microphone 510, and the microphone D520, and eliminates duplication if there are duplicate voices. In other words, even if the speeches of participants B and C are input to both shared microphone 510 and microphone D520, conference control section 210 excludes the input from microphone D520. Similarly, even if participant D's speech is input to both shared microphone 510 and microphone D 520 , conference controller 210 excludes input from shared microphone 510 . The communication unit 230 controls communication with other devices via the networks 40 and 50 . Further, when the remote conference device 200 receives a request for audio information of the conference from the listening support device 100, the remote conference device 200 transmits audio information of the overall conversation, which is a mixture of the overall conversation of the conference, to the listening support device 100 by streaming or sending an audio file. hand over.

The remote conference support system 1 also includes a viewing device 300 that can be viewed by one or more participants. The browsing device 300 may be an independent terminal device, or may also be used as the conference room terminal 500 or the participant A terminal 400 . The participant requests the screen information from the listening support device 100 via the browser section of the viewing device 300, and when receiving from the listening support device 100 a screen displaying both the overall conversation record and the utterances by speaker, the screen is displayed. .

The network 40 is a network such as the Internet or an intranet. The network 40 is not limited to this, and may be a WAN (Wide Area Network), a mobile phone network, etc., or a communication network combining these. The network 40 may be a VPN (Virtual Private Network) on a wireless communication network such as a mobile phone communication network.

The network 50 is a network such as the Internet or an intranet. The network 50 is not limited to this, and may be a WAN, a mobile phone network, etc., or a communication network combining these. The network 50 may be a VPN or the like on a wireless communication network such as a mobile phone communication network. Also, network 50 may be the same as network 40 .

The listening support device 100 transmits a request for conference voice information to the remote conference device 200, and receives voice information of the overall conversation, which is a mixture of the overall conversation of the conference, from the remote conference device 200 by streaming or as an audio file. Upon receiving a request for screen information from viewing device 300 , listening support device 100 transmits to viewing device 300 a screen displaying both the overall conversation record and the speech by speaker.

Listening assistance device 100 includes storage unit 110 , processing unit 120 , and communication unit 130 . The storage unit 110 includes a microphone user storage unit 111 , an overall conversation record storage unit 112 , and a speaker-by-speaker utterance storage unit 113 .

FIG. 2 is a diagram showing an example data structure of a microphone user storage unit. The microphone user storage unit 111 stores a microphone ID 111A and a user name 111B in association with each other. The microphone ID 111A is information for identifying the microphones used by the participants in the conference. The user name 111B is information for identifying the participants (one or more; if there are two or more, a shared microphone) who uses the microphone identified by the microphone ID 111A in the conference. In other words, the microphone user storage unit 111 is information that associates and identifies the participants who use the microphone.

FIG. 3 is a diagram showing an example of the data structure of the general conversation record storage unit. The entire conversation record storage unit 112 stores an utterance identifier 112A, an utterance time 112B, and an utterance text 112C in association with each other. The utterance identifier 112A is information that identifies utterances in the conference. The utterance time 112B is information specifying the date and time when the conference participant uttered an utterance. The utterance text 112C is information obtained by converting the utterances of the conference participants into text.

FIG. 4 is a diagram showing an example data structure of a speaker-specific utterance storage unit. Speaker-specific utterance storage unit 113 stores user 113A, utterance identifier 113B, utterance time 113C, utterance text 113D, and microphone ID 113E in association with each other. The user 113A is information that identifies a participant. The utterance identifier 113B is information for identifying utterances in the conference. The utterance time 113C is information specifying the date and time when the conference participant uttered an utterance. The utterance text 113D is information obtained by converting the utterances of the conference participants into text. The microphone ID 113E is information for identifying the microphone used by the participant in the conference.

Returning to the description of FIG. The processing unit 120 includes a mixed speech acquisition unit 121, an individual speech acquisition unit 122, a speaker separation unit 123, an acoustic modeling unit 124, a language modeling unit 125, an utterance similarity determination unit 126, and a parallel display creation unit. 127 and .

The mixed speech acquisition unit 121 acquires mixed speech information including utterances of a plurality of speakers from another device. Specifically, the mixed voice acquisition unit 121 requests the remote conference device 200 for the mixed voice information of the conference, and acquires the mixed voice information of the entire conversation including all of the utterances of the participants.

The individual voice acquisition unit 122 acquires individual voice information including an utterance by one of the speakers from a predetermined voice input device. Specifically, individual voice acquisition section 122 acquires individual voice information from shared microphone 510 and microphone D 520 .

The speaker separation unit 123 extracts the text of the first utterance obtained by separating the utterance from the mixed speech information (hereinafter also referred to as the mixed speech text), and the first utterance specified by one of the speakers from the individual speech information. Obtain two utterance texts (hereafter referred to as individual voice texts). Specifically, the speaker separation unit 123 uses the phoneme model and the language model created by the acoustic modeling unit 124 and the language modeling unit 125 to obtain information in which the utterance is converted into text. Alternatively, the speaker separation unit 123 uses a predetermined algorithm to build a neural network that performs deep learning focusing on differences in pitch and sound pressure, intonation, etc., performs speaker diarization processing, performs environmental sound, Separation of utterances by speakers and text conversion thereof may be performed. Alternatively, the speaker separation unit 123 may obtain information in which speech is converted into text using a voice recognition service as an external cloud service (not shown).

Acoustic modeling unit 124 performs pattern matching processing with a pronunciation dictionary on the similarity of waveform feature amounts for utterances separated from mixed speech information and individual speech information, and creates a phoneme model for estimating a phoneme string.

The language modeling unit 125 creates a language model for text conversion by applying it to a phoneme string and performing inference by applying it to similar words and terms.

The utterance similarity determination unit 126 determines the first utterance text (mixed voice text) obtained by separating the utterance from the mixed voice information and the second utterance text (individual voice text) specified by the individual voice information at approximately the same time. text) and similarity. Specifically, the utterance similarity determination unit 126 calculates the cosine similarity or the like to calculate the distance between texts, and determines the similarity. Note that the statement similarity determination unit 126 is not limited to this, and may determine similarity by various known means for determining similarity between texts.

The parallel display creation unit 127 creates a display screen that displays together a statement list in which the mixed voice text and the individual voice text are arranged in chronological order. For example, the parallel display creation unit 127 displays the mixed speech text judged to be similar to any of the individual speech texts by the utterance similarity judgment unit 126 to display the mixed speech text that is not judged to be similar to any of the individual speech texts. Create screen information in different modes.

Alternatively, the parallel display creation unit 127 makes the mixed speech text judged to be similar to any of the individual speech texts by the utterance similarity judgment unit 126 more visible than the mixed speech texts not judged to be similar to any of the individual speech texts. Create screen information by keeping the quality low (reducing the font size, lowering the saturation of the character color, etc.).

In addition, the parallel display creation unit 127 emphasizes the mixed speech text that has not been determined to be similar to any of the individual voice texts by the utterance similarity determination unit 126 over the mixed voice text that has been determined to be similar to any of the individual voice texts. The screen information may be created so as to be displayed (bold type, large font size, underline display, blinking display, etc.).

Note that, when displaying the mixed voice text that was not determined to be similar to any of the individual voice texts by the utterance similarity determination unit 126, the parallel display creation unit 127 determines that a speaker different from the speaker related to the individual voice information is uttering an utterance. You may make it display to the effect that it is doing in the statement list of an individual voice text. In this way, at least participants who are not easily identifiable speakers can be identified as speakers.

The communication unit 130 communicates with the remote conference device 200 via the network 50 .

FIG. 5 is a diagram showing a hardware configuration example of a listening support device. The listening support device 100 has a hardware configuration realized by a housing of a so-called server device, workstation, personal computer, smartphone, or tablet terminal. The listening support device 100 includes a processor 101, a memory 102, a storage 103, a communication device 104, and a bus 107 connecting each device. The same applies to the remote conference device 200 as well. In addition, the remote conference device 200 includes input/output devices such as a touch panel, keyboard, and display.

The processor 101 is, for example, an arithmetic device such as a CPU (Central Processing Unit).

The memory 102 is, for example, a memory device such as a RAM (Random Access Memory).

The storage 103 is a non-volatile storage device such as a so-called hard disk (Hard Disk Drive), SSD (Solid State Drive), or flash memory that can store digital information.

The communication device 104 is a wired communication device such as a network card, or a wireless communication device.

Mixed speech acquisition unit 121, individual speech acquisition unit 122, speaker separation unit 123, acoustic modeling unit 124, language modeling unit 125, utterance similarity determination unit 126, and combined display creation of the above-described listening support device 100 The unit 127 is implemented by a program that causes the processor 101 to perform processing. This program is stored in the memory 102, the storage 103, or a ROM device (not shown), loaded onto the memory 102 for execution, and executed by the processor 101. FIG.

Also, the storage unit 110 of the listening support apparatus 100 is realized by the memory 102 and the storage 103 . Also, the communication unit 130 is implemented by the communication device 104 . The hardware configuration example of the listening support apparatus 100 has been described above.

The configuration of listening assistance apparatus 100 can also be classified into more components according to the processing content. Also, one component can be grouped to perform more processing.

In addition, each processing unit (mixed speech acquisition unit 121, individual speech acquisition unit 122, speaker separation unit 123, acoustic modeling unit 124, language modeling unit 125, utterance similarity determination unit 126, parallel display creation unit 127) may be built with dedicated hardware (ASIC, GPU, etc.) that implements each function. Moreover, the processing of each processing unit may be executed by one piece of hardware, or may be executed by a plurality of pieces of hardware.

Next, the operation of the remote conference support system 1 according to this embodiment will be described.

FIG. 6 is a diagram showing an example of a conference recording flow. A conference recording flow is started, for example, by an operation of one of the participants on the remote conference device 200 .

First, the remote conference device 200 starts a conference (step S001). Specifically, the conference control unit 210 opens a remote conference room, receives input from the participants regarding the correspondence between the participants and the microphone IDs to be used, and stores the input in the microphone user storage unit 111 of the listening support apparatus 100 . do.

After the conference starts, when an utterance is received by the shared microphone 510 , the shared microphone 510 transmits voice data of the received utterance P to the listening support device 100 and the conference room terminal 500 . Although not shown, the conference room terminal 500 transmits voice data of the utterance P to the remote conference device 200 (step S002). Then, when receiving the utterance P, the conference control unit 210 of the remote conference device 200 creates mixed voice information including the utterance P as the utterance P' that constitutes the general conversation, and transmits the mixed voice information to the listening support device 100 (step S003). ).

The mixed speech acquisition unit 121 of the listening support device 100 receives an utterance P′ from the remote conference device 200 as a general conversation, and the individual speech acquisition unit 122 receives an individual speech utterance P from the shared microphone 510 as an utterance P″.

Then, listening support device 100 performs speech recognition processing, which will be described later, on speech P′ and speech P″, creates screen information including a speaker distinction screen and a general conversation screen, and transmits the screen information to viewing apparatus 300 (step S004). ).

The browser unit of the browsing device 300 receives and displays textual information about the utterance P′ and the utterance P″ (step S005).

When the microphone A410 of the participant A terminal 400 receives an utterance by the speaker A, the participant A terminal 400 transmits voice data of the received utterance Q to the remote conference device 200 (step S006). Then, when receiving the utterance Q, the conference control unit 210 of the remote conference device 200 creates mixed voice information including the utterance Q as the utterance Q' forming the general conversation, and transmits it to the listening support device 100 (step S007). ).

The mixed speech acquisition unit 121 of the listening support device 100 receives the utterance Q' from the remote conference device 200 as a general conversation.

Then, listening support apparatus 100 performs an utterance recognition process, which will be described later, on utterance Q', creates screen information including a speaker distinction screen and a general conversation screen, and transmits the screen information to viewing apparatus 300 (step S008).

The browser unit of the browsing device 300 receives and displays the textual information about the utterance Q' (step S009).

When the microphone D520 receives an utterance, the microphone D520 transmits the audio data of the received utterance R to the listening support apparatus 100 and the conference room terminal 500 . Although not shown, the conference room terminal 500 transmits voice data of the utterance R to the remote conference device 200 (step S010). Then, when receiving the utterance R, the conference control unit 210 of the remote conference device 200 creates mixed speech information incorporating the utterance R as an utterance R' that constitutes the general conversation, and transmits it to the listening support device 100 (step S011). ).

The mixed speech acquisition unit 121 of the listening support device 100 receives an utterance R′ from the remote conference device 200 as a general conversation, and the individual speech acquisition unit 122 receives an individual speech utterance R from the microphone D520 as an utterance R″.

Then, listening support device 100 performs an utterance recognition process, which will be described later, on utterances R' and utterances R'', creates screen information including a speaker distinction screen and an overall conversation screen, and transmits the screen information to viewing apparatus 300 (step S012). ).

The browser unit of viewing device 300 receives and displays textual information about utterance R′ and utterance R″ (step S013).

Such conference recording is repeated until the conference ends, and the remote conference device 200 ends the conference (step S014).

The above is the conference recording flow. According to the conference recording flow, from the start to the end of the conference, the participants participating in the conference and the input from the microphone used are recorded in association with each other, and the entire conversation is displayed on the screen of the viewing device 300. They can be displayed in a comparable manner.

FIG. 7 is a diagram showing an example of a speech recognition processing flow. The speech recognition processing flow is started in listening support device 100 at steps S004, S008, and S012 of the conference recording flow.

First, the mixed speech acquisition unit 121 and the individual speech acquisition unit 122 receive speech data (step S101). Specifically, the mixed voice acquisition unit 121 and the individual voice acquisition unit 122 receive the mixed voice information from the remote conference device 200 and the individual voice information from the common microphone 510 and the microphone D520, respectively.

Then, the speaker separation unit 123 determines whether or not the received voice data is a general conversation (step S102). Specifically, if the speech data received in step S101 is acquired by the mixed speech acquisition unit 121, the speaker separation unit 123 determines that it is a general conversation, and the individual speech acquisition unit 122 acquires If it is, it is determined that it is not a general conversation.

If the received voice data is not a general conversation ("NO" in step S102), speaker separation unit 123 identifies the microphone ID of the utterance source (step S103). Then, the speaker separation unit 123 refers to the microphone user storage unit 111 and narrows down the speakers (step S104). Specifically, the speaker separation unit 123 searches the microphone user storage unit 111 to identify the user associated with the microphone ID identified in step S103.

Then, the speaker separating unit 123 separates the voice and the environmental sound from the received voice data (step S105). In this process, the speaker separation unit 123 causes the acoustic modeling unit 124 to construct a phoneme model for individual speech information, and uses the phoneme model to estimate a phoneme sequence.

Then, the speaker separation unit 123 performs speech recognition on the phoneme string and converts it into text (step S106). Specifically, the speaker separation unit 123 converts the phoneme strings estimated by the speaker separation unit 123 into text by applying a language model to apply inference to similar words or terms.

Then, the parallel display creation unit 127 creates a screen for displaying text for each speaker (step S107). Then, the created speaker distinction screen is transmitted to the viewing device 300 (step S108).

If the received voice data is a general conversation ("YES" in step S102), the speaker separation unit 123 separates voice and environmental sound from the received voice data (step S109). In this process, the speaker separation unit 123 causes the acoustic modeling unit 124 to perform Fourier transform on the mixed speech information to separate it into synthesized sine waves, and specifies the feature amount of each waveform.

Then, the speaker separating unit 123 separates the voice for each speaker (step S110). Specifically, the speaker separation unit 123 causes the acoustic modeling unit 124 to construct a phoneme model for each separated waveform, and estimates a phoneme string using the phoneme model.

Then, the speaker separation unit 123 determines whether or not the speaker separation is successful (step S111). Specifically, the speaker separation unit 123 determines that the speaker separation is successful when the generation of the phoneme model is successful.

If the speaker separation is not successful ("NO" in step S111), the language modeling unit 125 replaces the speech data with dummy text such as "unidentifiable" as the utterance is unrecognizable. (Step S112). Then, the parallel display creation unit 127 advances the control to step S115.

If the speaker separation is successful ("YES" in step S111), the speaker separation unit 123 speech-recognizes the phoneme string and converts it into text (step S113). Specifically, the speaker separation unit 123 estimates phonemes using the phoneme model constructed by the acoustic modeling unit 124, applies the language model to apply inference to similar words and words, and performs text conversion. conduct.

Then, the statement similarity determination unit 126 marks texts that are not similar to the individual conversation texts (step S114). Specifically, the utterance similarity determination unit 126 calculates the similarity distance between the text of each speaker obtained in step S113 and the text of the individual speech information at substantially the same time obtained in step S106, and determines the similarity. If not, mark the speaker's text.

Then, the parallel display creation unit 127 creates a screen by highlighting the marked text (step S115). More specifically, the parallel display creation unit 127 creates a display screen that displays together an utterance list in which the mixed voice text and the individual voice text are arranged in chronological order. For example, the parallel display creation unit 127 creates screen information by keeping the unmarked text lower in visibility than the marked text (reducing the font size, lowering the saturation of the character color, etc.). Alternatively, the screen information is created so that the marked text is highlighted (bold, larger font size, underlined, blinking, etc.) than the unmarked text.

Then, the parallel display creation unit 127 transmits the created general conversation screen to the viewing device 300 (step S116).

The above is an example of the speech recognition processing flow. According to the utterance recognition processing, the individual voice information is converted into text and displayed, the mixed voice information is converted into text and displayed, and the text similar to the individual voice information stands out more faintly than the dissimilar text. can be displayed side by side. By doing so, even when an input is made from a microphone whose speaker is unknown, it is possible to present information that serves as a clue for estimating the speaker.

Note that, in step S115, the parallel display creation unit 127 displays the mixed voice text that has not been determined to be similar to any of the individual voice texts by the utterance similarity determination unit 126. It is also possible to create a screen that displays the fact that the speaker is speaking in the speech list of the individual voice text. In this way, at least participants who are not easily identifiable speakers can be identified as speakers.

FIG. 8 is a diagram showing an example of a conversation confirmation screen. An example of conversation confirmation screen 600 is an example of an integrated screen in which the screens created in steps S108 and S116 of the speech recognition process are displayed in speaker-specific utterance area 602 and overall conversation recording area 601, respectively.

In the speaker-by-speaker speech area 602, the speaker, the text of the content of the speech, and the time of speech are displayed in chronological order for speech input from a microphone that can identify the speaker. Note that the speaker-by-speaker utterance area 602 is an input area for receiving an instruction to sort and redisplay the speakers in ascending/descending order, or to sort and redisplay in ascending/descending order of utterance time, and It has a scroll bar that accepts scroll operations.

In the whole conversation record area 601, the text of the utterance contents is shown in chronological order for all the utterances in the conference that are input from the speaker-identifiable microphone and the speaker-unidentifiable microphone. In addition, in the whole dialogue record area 601, text that is not similar to the text displayed in the speaker-specific utterance area 602 is displayed in bold, and if there is an utterance that is unclear or has an inseparable utterance whose utterance period overlaps, The text "Recognition error" is displayed. By displaying in this way, even in the case of an audio conference that does not use video data, it is possible to assist listening in a situation in which a plurality of people are speaking.

FIG. 9 is a diagram showing another example of the conversation confirmation screen. The conversation confirmation screen 600 ′ is basically the same screen as the conversation confirmation screen 600 . The difference is that when the text displayed in the speaker-specific utterance area 602 is not similar to the text displayed in the speaker-specific utterance area 602 in the conversation record shown on the conversation confirmation screen 600, the individual voice information is displayed in the speaker-specific utterance area 602 when the text is displayed in bold. The point is that it is displayed that a speaker different from the speaker concerned is speaking.

By showing the conversation confirmation screen 600′ in this way, even if the speaker cannot be specified, especially the hearing-impaired person can at least estimate the participants who are not the speaker. You can get the information that will be

The above is an example of the remote conference support system according to the embodiment. According to the example of the remote conference support system 1 according to the embodiment, even in the case of an audio conference that does not use video data, it is possible to support listening in situations where multiple people are speaking.

In addition, although the example of the above embodiment has been described for a general remote conference, it can also be applied to a remote conference for obtaining video data of the speaker's face and mouth. In this case, more information for identifying the speaker can be obtained, so it can be said that accuracy in identifying the speaker can be further improved.

Also, the technical elements of the above-described embodiments may be applied singly, or may be applied after being divided into a plurality of parts such as program parts and hardware parts.

The present invention has been described above with a focus on the embodiments.

REFERENCE SIGNS LIST 1 remote conference support system 40, 50 network 100 listening support device 110 storage unit 111 microphone user storage unit 112 entire conversation record storage Section 113 Speaker-specific utterance storage section 120 Processing section 121 Mixed speech acquisition section 122 Individual speech acquisition section 123 Speaker separation section 124 Acoustic modeling unit 125 Language modeling unit 126 Utterance similarity determination unit 127 Parallel display creation unit 130 Communication unit 200 Remote conference device 210 Conference control unit 220 Display unit 230 Communication unit 300 Browsing device 400
Participant A terminal, 410... microphone, 500... conference room terminal, 510... shared microphone, 520... microphone D.

Claims (10)

a mixed speech acquisition unit that acquires mixed speech information including utterances of a plurality of speakers from another device;
an individual voice acquisition unit that acquires individual voice information including the utterance by any of the speakers from a predetermined voice input device;
A speaker who obtains a text of a first utterance obtained by separating said utterance from said mixed voice information and a text of a second utterance specifying any of said utterances of said speaker from said individual voice information. a separation unit;
an utterance similarity determination unit that determines similarity between the text of the first utterance and the text of the second utterance;
a parallel display creation unit for creating a display screen for displaying together an utterance list in which the text of the first utterance and the text of the second utterance are arranged in chronological order;
A listening support device comprising: The listening support device according to claim 1,
The parallel display creation unit determines that the text of the first utterance determined to be similar to any of the texts of the second utterance by the utterance similarity determination unit is similar to any of the texts of the second utterance. making the display mode different from the text of the first utterance that was not determined;
A listening support device characterized by: The listening support device according to claim 1 or 2,
The parallel display creation unit determines that the text of the first utterance determined to be similar to any of the texts of the second utterance by the utterance similarity determination unit is similar to any of the texts of the second utterance. keeping the visibility lower than the text of the first utterance that was not judged;
A listening support device characterized by: The listening support device according to any one of claims 1 to 3,
The side-by-side display creation unit determines that the text of the first utterance that is not determined to be similar to any of the texts of the second utterance by the utterance similarity determination unit is similar to any of the texts of the second utterance. highlighting over the text of the first utterance determined to be
A listening support device characterized by: The listening support device according to any one of claims 1 to 4,
The mixed speech acquisition unit acquires the mixed speech information in real time,
The individual voice acquisition unit acquires the individual voice information in real time,
The parallel display creation unit, when displaying the text of the first utterance that has not been determined to be similar to any of the texts of the second utterance by the utterance similarity determination unit, displays the text of the first utterance related to the individual voice information. displaying in the utterance list of the text of the second utterance that a speaker different from the speaker is uttering.
A listening support device characterized by: The listening support device according to any one of claims 1 to 5,
The individual voice acquisition unit acquires the plurality of individual voice information from a plurality of predetermined voice input devices,
A listening support device characterized by: The listening support device according to any one of claims 1 to 6,
The utterance similarity determination unit determines similarity between the text of the first utterance and the text of the second utterance at substantially the same time.
A listening support device characterized by: The listening support device according to any one of claims 1 to 7,
The mixed audio information and the individual audio information are audio of the same online conference, and the other device is a device that controls the online conference.
A listening support device characterized by: A listening support method using a hearing support device,
The listening support device includes a processing unit,
The processing unit is
a mixed speech obtaining step of obtaining mixed speech information including utterances of multiple speakers from another device;
an individual voice acquisition step of acquiring individual voice information including the utterance by any of the speakers from a predetermined voice input device;
A speaker who obtains a text of a first utterance obtained by separating said utterance from said mixed voice information and a text of a second utterance specifying any of said utterances of said speaker from said individual voice information. a separation step;
an utterance similarity determination step of determining similarity between the text of the first utterance and the text of the second utterance;
a parallel display creating step of creating a display screen for displaying together an utterance list in which the text of the first utterance and the text of the second utterance are arranged in chronological order;
A hearing assistance method characterized by carrying out. A listening support program that causes a computer to function as a listening support device,
The computer comprises a processor,
to the processor;
a mixed speech obtaining step of obtaining mixed speech information including utterances of multiple speakers from another device;
an individual voice acquisition step of acquiring individual voice information including the utterance by any of the speakers from a predetermined voice input device;
A speaker who obtains a text of a first utterance obtained by separating said utterance from said mixed voice information and a text of a second utterance specifying any of said utterances of said speaker from said individual voice information. a separation step;
an utterance similarity determination step of determining similarity between the text of the first utterance and the text of the second utterance;
a parallel display creating step of creating a display screen for displaying together an utterance list in which the text of the first utterance and the text of the second utterance are arranged in chronological order;
A listening support program characterized by implementing

Images