KR20250085270A - Appratus and method for providing meeting information - Google Patents

Abstract

According to one embodiment of the present invention, a conference management device may include a first extraction unit that extracts audio features related to the conference using audio of conference participants; a second extraction unit that extracts visual features related to the conference using captured images of the participants; a content recognition unit that recognizes conference content in relation to the audio features and the visual features of each of the participants; a situation analysis unit that analyzes a conference situation based on the audio features and the visual features; and a summary unit that generates data related to at least one of a conference summary or a conference situation based on recognition/analysis results of the content recognition unit and the situation analysis unit.

Description

{APPRATUS AND METHOD FOR PROVIDING MEETING INFORMATION}

Various embodiments disclosed in this document relate to meeting minutes writing techniques.

In general, formal meetings (e.g., corporate meetings or government agencies) require minutes after the meeting ends. During the meeting, the meeting content is recorded in video or audio, and the recorder checks the recorded content and organizes it to write the minutes. Since the records are recorded or recorded even when there is no speech at the meeting, organizing them takes a lot of time and effort. To solve this inconvenience, an automated minutes system has been launched.

Conventional meeting minutes automation systems can recognize the speaker's voice using a voice recognition device and then create a summary of the meeting contents based on the recognition results.

The automated minutes system simply summarized the voice recognition results, making it difficult for absentees to accurately infer the meeting situation.

Various embodiments disclosed in this document can provide a conference management device and a conference management method capable of automatically generating conference minutes based on audio and image data.

According to one embodiment of the present invention, a conference management device may include a first extraction unit that extracts audio features related to the conference using audio of conference participants; a second extraction unit that extracts visual features related to the conference using captured images of the participants; a content recognition unit that recognizes conference content in relation to the audio features and the visual features of each of the participants; a situation analysis unit that analyzes a conference situation based on the audio features and the visual features; and a summary unit that generates data related to at least one of a conference summary or a conference situation based on recognition/analysis results of the content recognition unit and the situation analysis unit.

In addition, a conference management method by a conference management device according to an embodiment disclosed in this document may include an operation of acquiring audio and images of conference participants; an operation of extracting audio features related to the conference using the acquired audio, and an operation of extracting visual features related to the conference using the captured image; an operation of recognizing/analyzing conference content and a conference situation in relation to the audio features and the visual features of each of the participants; and an operation of generating summary data related to at least one of the conference content or the conference situation based on a result of the recognition/analysis.

In addition, a user terminal according to an embodiment disclosed in the present document includes a communication module providing a designated communication channel; a multi-channel microphone, an omnidirectional camera, and an output module provided at a conference venue; and a processor, wherein the processor generates audio data of conference participants through the multi-channel microphone, generates image data of the participants through the camera, synchronizes the audio data and the image data and transmits them to an external conference management device through the communication module, and receives at least one of conference situation summary data or conference mediation data generated as the external conference management device recognizes/analyzes the conference content and the conference situation in relation to each other through the communication circuit, and provides the at least one received data to the conference participants through the output module.

According to various embodiments disclosed in this document, meeting minutes can be automatically generated based on audio and image data. In addition, various effects that can be directly or indirectly identified through this document can be provided.

Figure 1 illustrates a configuration diagram of a multi-modal conference management system according to one embodiment.
Figure 2 shows a configuration diagram of a user terminal according to one embodiment.
FIG. 3 shows a configuration diagram of a conference management device according to one embodiment, and FIG. 4 shows an example diagram of conference situation summary data according to one embodiment.
Figure 5 illustrates a schematic flowchart of a meeting management method according to one embodiment.
Figure 6 illustrates a flow chart of a method for generating meeting minutes data according to one embodiment.
Figure 7 illustrates a flow chart of a meeting situation summary method according to one embodiment.
In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components.

Figure 1 illustrates a configuration diagram of a multi-modal conference management system according to one embodiment.

Referring to FIG. 1, a multimodal conference management system (1) according to one embodiment may include a user terminal (20) and a conference management device (30). The user terminal (20) and the conference management device (30) may be configured as a single device.

According to one embodiment, the user terminal (20) may be a computing device provided at a conference location or adjacent thereto, and including or linked with a microphone, a camera, and an output module. The user terminal (20) may include a PC, a laptop, a tablet, and a portable terminal. The user terminal (20) may generate audio data and image data of conference participants, and transmit the generated audio data and image data to a conference management device (30) through a designated communication channel.

According to one embodiment, the conference management device (30) can obtain audio data and image data through a designated communication channel, and extract audio features and visual features based on the audio data and the image data. According to one embodiment, the conference management device (30) can generate conference summary data related to at least one of information among conference speech and conference situation by using the audio features and the visual features. The conference summary data can be generated at at least one of the midpoint of the conference or the end point of the conference.

According to one embodiment, the conference management device (30) can check the conference situation using audio features and visual features, and determine whether conference mediation is necessary based on the check result. If the conference management device (30) determines that conference mediation is necessary, it can generate conference mediation data. For example, the conference management device (30) can determine whether conference mediation is necessary based on at least one of the following situational information: the speaking time of each participant, the relevance of the topic of the speech, or the surrounding noise.

According to one embodiment, the conference management device (30) can transmit at least one of conference summary data or conference mediation data to the user terminal (20) through a designated communication channel.

In addition, the user terminal (20) can obtain at least one of conference mediation data or conference summary data from the conference management device (30) through a designated communication channel, and provide at least one of the data to conference participants (e.g., conference host) through an output module.

In this way, the multimodal conference management system (1) according to one embodiment can comprehensively analyze and organize the conference situation by analyzing not only the voice recognition of the conference participants but also the visual information of the participants.

In addition, the multimodal conference management system (1) according to one embodiment can organize and provide conference content or conference situations during the conference based on audio and image data of conference participants, and can also act as a conference mediator by making appropriate comments when the conference does not proceed smoothly.

Figure 2 shows a configuration diagram of a user terminal according to one embodiment.

Referring to FIG. 2, a user terminal (20) according to one embodiment may include a microphone (210), a camera (220), a communication module (230), an output module (240), a memory (250), and a processor (230). In one embodiment, the user terminal (20) may omit some components or may further include additional components. In addition, some of the components of the user terminal (20) may be combined to form a single entity, but may perform the functions of the corresponding components prior to combination in the same manner.

A microphone (210) is installed in a conference room and can acquire or detect audio (e.g., voice) of conference participants. The microphone (210) can include a multi-channel array microphone capable of acquiring sound from multiple sound sources at one time. The multi-channel array microphone can acquire the voice of the current speaker through beam forming.

A camera (220) may be installed in a conference venue to capture images of the conference venue (or of conference attendees). The camera (220) may include at least one image sensor among a CCD (charge coupled device) and a MOS (metal oxide semi-conductor). The camera (220) may be a 360-degree camera installed at a midpoint of the conference venue to capture images of conference attendees in all directions. Alternatively, the camera (220) may include a plurality of image sensors capable of capturing images in all directions.

The output module (240) can output at least one data of symbols, numbers, or characters visually or audibly under the control of the processor (230). The output module (240) can include, for example, at least one output module of a liquid crystal display, an OLED, a touch screen display, and a speaker.

The communication module (230) can support establishment of a communication channel or a wireless communication channel between a user terminal (20) and another device (e.g., a conference management device (30)), and performance of communication through the established communication channel. The communication channel can include, for example, at least one of LAN, FTTH, xDSL, Wibro, Wireless LAN, Wi-Fi, Bluetooth, Zigbee, WFD (Wi-Fi Direct), UWB (Ultrawideband), Infrared Data Association (IrDA), Bluetooth Low Energy (BLE), Near Field Communication (NFC), 3G, 4G, or 5G.

The memory (250) may include various forms of volatile memory or nonvolatile memory. For example, the memory (250) may include ROM (read only memory) and RAM (random access memory). In one embodiment, the memory (250) may be located inside or outside the processor (230), and the memory (250) may be connected to the processor (230) via various means already known. The memory (250) may store various data used by at least one component of the user terminal (20) (e.g., the processor (230)). The data may include, for example, input data or output data for software and commands related thereto. For example, the memory (250) may store at least one instruction for providing a conference management service.

The processor (230) may control at least one other component (e.g., hardware or software component) of the user terminal (20) by executing at least one instruction, and may perform various data processing or calculations. The processor (230) may include, for example, at least one of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor, an application processor, an application specific integrated circuit (ASIC), and a field programmable gate array (FPGA), and may have multiple cores.

- According to one embodiment, the processor (230) may acquire audio (e.g., voice) of a meeting place using a microphone (210) and generate audio data based on the acquired audio. The processor (230) may acquire audio signals for each sound source through a multi-channel microphone and generate audio data by temporally synchronizing the acquired audio signals for each sound source.

According to one embodiment, the processor (230) may capture an image using the camera (220) and generate image data based on the captured image. The processor (230) may generate an omnidirectional image through the camera (220). Alternatively, the processor (230) may divide the generated omnidirectional image into speaker-specific (or section-specific) image data and generate image data per speaker/or per image sensor (e.g., when the camera (220) includes a plurality of image sensors). The image data may include temporally continuous frames (video).

According to one embodiment, the processor (230) may transmit audio data and image data to the conference management device (30) through the communication module (230). In this case, the conference management device (30) may generate conference summary data by summarizing the conference content as a whole, by speaker, or by topic using the audio data and the image data. In addition, the conference management device (30) may determine whether a situation requires conference mediation (e.g., inducing speech, requesting focus) using the audio data and the image data, and if a situation requires conference mediation, may generate conference mediation data and provide it to conference participants through the output module ().

According to one embodiment, the processor (230) may obtain meeting summary data (e.g., meeting minutes, meeting situation summary) or meeting mediation data from the meeting management device (30) through the communication module (230) and provide it to meeting participants through the output module (240).

According to the above-described embodiment, at least some of the operations performed by the user terminal (20) can be performed by the conference management device (30).

In this way, the user terminal (20) according to one embodiment can obtain images and audio related to the meeting at the meeting location and transmit them to the meeting management device (30), thereby assisting the meeting management device (30) in summarizing the meeting (content/situation) and mediating the meeting, and can interact with the meeting management device (30) to perform the role of a meeting mediator.

FIG. 3 shows a configuration diagram of a conference management device according to one embodiment, and FIG. 4 shows an example diagram of conference situation summary data according to one embodiment.

Referring to FIG. 3, a conference management device (30) according to one embodiment may include a communication module (310), a memory (320), and a processor (330). In one embodiment, the conference management device (30) may omit some components or may further include additional components. In addition, some of the components of the conference management device (30) may be combined to form a single entity, but may perform the functions of the corresponding components prior to combination in the same manner.

The communication module (310) can support establishment of a communication channel or a wireless communication channel between the conference management device (30) and another device (e.g., user terminal (20)), and performance of communication through the established communication channel. The communication channel can include, for example, at least one of LAN, FTTH, xDSL, Wibro, Wireless LAN, Wi-Fi, Bluetooth, Zigbee, WFD (Wi-Fi Direct), UWB (Ultrawideband), Infrared Data Association (IrDA), Bluetooth Low Energy (BLE), Near Field Communication (NFC), 3G, 4G, or 5G.

The memory (320) may include various forms of volatile memory or nonvolatile memory. For example, the memory (320) may include a ROM (read only memory) and a RAM (random access memory). In one embodiment, the memory (320) may be located inside or outside the processor (330), and the memory (320) may be connected to the processor (330) via various means known in the art. The memory (320) may store various data used by at least one component of the conference management device (30) (e.g., the processor (330)). The data may include, for example, input data or output data for software and commands related thereto. For example, the memory (320) may store at least one instruction for providing a meeting minutes writing service.

The processor (330) may control at least one other component (e.g., hardware or software component) of the user terminal (20) by executing at least one instruction, and may perform various data processing or calculations. The processor (330) may include, for example, at least one of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor, an application processor, an application specific integrated circuit (ASIC), and a field programmable gate array (FPGA), and may have a plurality of cores. The processor (330) may obtain audio data and image data from the user terminal (20) through the communication module (310).

According to one embodiment, the processor (330) may include a first extraction unit (331), a second extraction unit (332), a content recognition unit (333), a situation analysis unit (334), and a summary unit (335). The first extraction unit (331), the second extraction unit (332), the summary unit (335), the content recognition unit (333), and the situation analysis unit (334) may be software modules (e.g., an artificial intelligence model) executed by the processor (330) or may be configured as hardware modules (e.g., an IC). At least some of the components (331 to 335) of the processor (330) may be omitted or integrated with other components. For example, the first extraction unit (331) and the second extraction unit (332) may be configured as one module. For another example, the summary unit (335) may be configured as a first summary unit related to a meeting content summary and a second summary unit related to a meeting situation summary, respectively.

According to one embodiment, the first extraction unit (331) can extract audio features related to the meeting from the acquired audio data. For example, the first extraction unit (331) can extract audio features of at least one of spectral features, frequency domain features, time domain features, time series features, and time-frequency domain features from the audio data.

The first extraction unit (331) can distinguish between speech, noise, and sound, and store audio features in the memory (320). The first extraction unit (331) can separate speech, noise, and sound from the acquired audio data, and extract audio features including speech features, noise features, and sound features from the separated signals. For example, the first extraction unit (331) can separate speech and non-speech (sound) through frequency band distinction. In this regard, the first extraction unit (331) can separate speech and sound/noise based on spectrum features and dynamic range expansion. The first extraction unit (331) can distinguish between speech features, noise features related to noise types and levels, and sound features related to sound types, and store them in the memory (320).

According to one embodiment, the second extraction unit (332) can extract visual features related to the meeting participants from the acquired image data. For example, the second extraction unit (332) can extract at least one of the lip shape, facial expression, gaze, head direction, gesture, or pose of each of the participants from the acquired image.

According to one embodiment, the content recognition unit (333) can obtain audio features and visual features, and recognize the audio features and visual features in relation to each other. For example, the content recognition unit (333) can convert voice data into text data, and generate a recognition result text by considering the grammar, vocabulary, and structure of the language in the text. The content recognition unit (333) can correct the voice recognition result and map it to each speaker based on the recognition result text, voice features, and visual features of each participant. The content recognition unit (333) can correct the voice recognition result by more accurately recognizing the speaker and his/her speech based on at least one visual feature of, for example, lip shape, facial expression, gaze, head direction, gesture, or pose. Accordingly, the content recognition unit (333) according to one embodiment can accurately perform voice recognition even in a noisy environment.

According to one embodiment, the content recognition unit (333) can organize the utterances of each speaker in chronological order and perform post-processing such as correcting interjections, repetitions, and recognition errors.

According to one embodiment, the situation analysis unit (334) can analyze the meeting situation based on at least one of the audio features and the visual features. For example, the situation analysis unit (334) can identify at least one of the meeting situations among the emotional state of the speaker, the concentration of the participants in the meeting, the relevance of the topic of the speech, the meeting atmosphere, or the meeting participation level based on the audio features and the visual features.

According to one embodiment, the situation analysis unit (334) may analyze the emotional state of a speaker among the participants based on audio features and at least one of voice features, lip shape features, and facial expression features among the visual features. For example, the voice feature may include at least one of voice features such as voice waveform (e.g., amplitude, periodicity, vibration period), voice tone (e.g., tone pitch), or intonation (e.g., length, intensity, beat, pitch, speed). The lip shape feature may include at least one of lip shape features such as mouth corner direction, lip size change, or mouth opening degree. The facial expression feature may include at least one of eye corner direction, eye opening degree, facial wrinkle change, and eyebrow shape.

According to one embodiment, the situation analysis unit (334) can determine the meeting concentration of the participants based on at least one of the visual features of the gaze, head direction, gesture, and pose. For example, if the gaze or head direction is not directed toward another participant, or if the gaze, head direction, gesture, or pose detects at least one behavior of drowsiness, inattention, or intention not to attend the meeting, the situation analysis unit (334) can determine that the meeting concentration of the participant is low.

According to one embodiment, the situation analysis unit (334) can detect keywords of speech based on voice features among audio features, and confirm topical relevance of speech based on the detected keywords. The situation analysis unit (334) can detect keywords based on voice recognition results (texts), and confirm topical relevance (e.g., similarity) with the detected keywords. In this regard, the situation analysis unit (334) can determine topics from the detected keywords, or determine topics based on the speech of the moderator. Alternatively, the situation analysis unit (334) can confirm topical relevance based on commonality between keywords.

According to one embodiment, the situation analysis unit (334) can identify the final ending of a utterance based on audio features, and analyze the meeting atmosphere based on the final ending. The meeting atmosphere can include at least one of coercion, obligation, urgency, encouragement, and possibility. For example, if the situation analysis unit (334) identifies 'must~' in the final ending, it can determine the meeting atmosphere as at least one of 'coercion', 'obligation', and 'urgency'. For example, if the situation analysis unit (334) identifies at least one of '~ 면 되다', '~ 냠 낳으면', '~ 냠 낳으면', or '~ 냠 구나' in the final ending, it can determine the meeting atmosphere as 'encouragement'. For example, if the situation analysis unit (334) identifies at least one of '~ 할 수 있습니다' and '~ 할 수 있습니다' in the final ending, it can determine the meeting atmosphere as at least one of 'possibility' or 'encouragement'.

According to one embodiment, the situation analysis unit (334) can calculate the speaking time of each participant and analyze the meeting participation of each participant based on the calculated speaking time. For example, the situation analysis unit (334) can calculate the meeting participation as a ratio of the speaking time to the total meeting time.

According to one embodiment, the situation analysis unit (334) may determine whether meeting mediation is necessary based on at least one of situation information from among emotional state, meeting concentration, meeting participation, topic relevance of speech, meeting atmosphere, or surrounding noise. In one embodiment, if the processor (330) (e.g., the situation analysis unit (334)) determines that meeting mediation is necessary, it may generate meeting mediation data. The meeting mediation data may include, for example, data related to at least one of a request for attention, an encouragement of meeting participation, or a request for calming down.

In one embodiment, the situation analysis unit (334) may determine that meeting mediation is necessary if the topic relevance of the identified keyword is less than a specified value. In this case, the situation analysis unit (334) may generate an attention request including a message such as, for example, "The conversation is continuing off-topic, so please focus on the meeting topic."

According to one embodiment, the situation analysis unit (334) may determine that meeting mediation is necessary if the meeting participation rate of the attendees is below a specified percentage. In this case, the situation analysis unit (334) may generate a meeting participation encouragement message for all attendees or at least one attendee with a relatively low participation rate. The meeting participation encouragement data may include, for example, a message such as "This time, I would like Mr. A to give his opinion on the topic."

According to one embodiment, the situation analysis unit (334) may determine that the meeting situation is excessively agitated based on at least one of the meeting situation information such as the emotional state or the meeting atmosphere (e.g., if the meeting atmosphere is noisy based on the noise level and the type of sound), in this case, the situation analysis unit (334) may generate calming request data including a message such as “Everyone, please calm down” or “Everyone except the speaker, please be quiet.”

According to one embodiment, the summary unit (335) may generate meeting summary data (e.g., meeting content summary data, meeting situation summary data) based on at least one result of audio-visual feature recognition and situation analysis.

According to one embodiment, the summary unit (335) may generate meeting content summary data (hereinafter, referred to as minutes data) by summarizing the meeting content based on at least one criterion among topic-wise, speaker-wise, or overall content-wise based on the results of audio-visual feature recognition. The minutes data may be generated at the end of the meeting, and may be generated during the middle of the meeting (e.g., during a break) according to settings or a request from a user terminal (20).

According to one embodiment, the summary unit (335) may generate meeting situation summary data during or after the meeting based on the analyzed meeting situation. For example, the summary unit (335) may generate meeting situation summary information of at least one of the emotional state of the speaker, the meeting concentration of the participants (e.g., the speaking time of each participant), the topic relevance of the speech, the meeting atmosphere, or the meeting participation based on the situation analysis result. Referring to FIG. 4, the meeting situation summary data may include the meeting atmosphere, the meeting concentration, the topic relevance of the speeches, and the meeting participation.

According to one embodiment, the processor (330) may provide the generated meeting mediation data or meeting situation summary data to the user terminal (20) in real time through the communication module (310). In this case, the user terminal (20) may output the meeting mediation data or the meeting situation summary data to the meeting participants visually or audibly. In this way, the meeting situation summary data may be transmitted to the user terminal (20) in real time during the meeting to support efficient meeting progress by the meeting host or smooth discussion/debate among the meeting participants. Alternatively, the meeting situation summary data may be recorded together with the meeting content after the meeting ends to support the meeting participants to vividly understand the meeting situation.

In this way, the conference management device (30) according to one embodiment can perform more accurate voice recognition using multi-modal information including audio and visual data about conference participants, thereby automatically generating highly accurate conference content summary data.

In addition, the conference management device (30) according to one embodiment can analyze the conference situation from various angles and summarize and record the conference situation, so that even those who are absent from the meeting can accurately understand the conference situation.

Furthermore, the conference management device (30) according to one embodiment can provide comments for conference mediation appropriately according to the conference situation detected from various angles based on audio-visual feature information, and thus can act as a conference mediator or substitute for a conference host to help the conference proceed smoothly.

Fig. 5 is a schematic flowchart of a meeting management method according to one embodiment. The detailed configuration of operations 520, 530, and 540 of Fig. 5 will be described later with reference to Figs. 6 and 7.

Referring to FIG. 5, in operation 510, the conference management device (30) (e.g., processor (330)) can obtain audio and images related to the conference. For example, the conference management device (30) can obtain audio and images through a camera installed in a conference venue and shooting around conference participants and a microphone that obtains audio (e.g., voice) of the conference venue.

In operation 520, the conference management device (30) can extract audio features related to the conference using the acquired audio, and can extract visual features related to the conference using the captured image.

In operation 540, the conference management device (30) can recognize/analyze the conference content and conference situation in relation to the audio characteristics and visual characteristics of each participant.

In operation 540, the conference management device (30) may generate summary data related to at least one of the conference content or the conference situation based on the result of recognition/analysis. The summary data may include at least one of the conference minutes data, the conference situation summary data, and the conference mediation data.

Figure 6 illustrates a flow chart of a method for generating meeting minutes data according to one embodiment.

Referring to FIG. 6, in operation 610, the conference management device (30) (e.g., the processor (330)) may extract audio features including voice, noise, and sound features from audio data related to the conference, and store the extracted audio features. For example, the conference management device (30) may extract audio features including at least one of spectral features, frequency domain features, time domain features, time series features, and time-frequency domain features. For example, the conference management device (30) may separate voice and non-voice (sound) through frequency band distinction, separate voice and sound/noise based on spectral features and dynamic range expansion, and store voice features, noise features related to noise types and levels, and sound features related to sound types in the memory (320).

In operation 620, the conference management device (30) can extract visual features related to conference participants from the acquired image data. For example, the second extraction unit (332) can extract at least one of the lip shape, facial expression, gaze, head direction, gesture, or pose of each of the participants from the acquired image data.

In operation 630, the conference management device (30) can obtain audio features and visual features, and recognize the audio features and visual features in relation to each other. For example, the conference management device (30) can convert voice data into text data, generate a recognition result text by considering the grammar, vocabulary, and structure of the language in the text, and map and improve the accuracy of the speech content of each speaker based on the voice features and visual features. For another example, the conference management device (30) can analyze at least one conference situation among the emotional state of the speaker, the concentration of the participants in the meeting, the relevance of the topic of the speech, the meeting atmosphere, or the meeting participation.

In operation 640, the conference management device (30) may perform post-processing on the speech recognition results for each speaker for the recognized results and then store the recognition results. For example, the conference management device (30) may perform post-processing such as sequentially arranging the recognized results for each speaker and correcting interjections, repetitions, or recognition errors.

In operation 650, the conference management device (30) may generate conference content summary data (e.g., minutes) based on the audio-visual feature-based voice recognition result, and store the generated data in the memory (320). For example, the conference management device (30) may generate conference content data by summarizing the conference content based on at least one criterion among subject-wise, speaker-wise, or overall content based on the result of audio-visual feature recognition. The conference data may be generated when the conference ends, and may be generated in the middle of the conference (e.g., during a break) according to a setting or a request of a user terminal (20).

In this way, the conference management device (30) according to one embodiment can perform more accurate voice recognition using multi-modal information including audio and visual data about conference participants, thereby quickly generating more accurate conference minutes data.

Figure 7 illustrates a flow chart of a meeting situation summary method according to one embodiment.

Referring to FIG. 7, in operation 710, the conference management device (30) (e.g., processor (330)) may analyze the emotional state of a speaker among the participants based on at least one of voice features (e.g., tone), lip shape features (e.g., corners of the mouth), and facial expression features (e.g., changes in wrinkles, corners of the eyes).

In operation 720, the conference management device (30) can determine the conference concentration of each participant based on at least one of visual features of gaze, head direction, gesture, and pose. For example, the situation analysis unit (334) can determine that the conference concentration of a conference participant who has confirmed at least one behavior of drowsiness, inattention, or intention not to attend the meeting in the gaze, head direction, gesture, or pose is low.

In operation 730, the conference management device (30) can detect keywords of speech based on the results of voice-visual recognition and confirm the relevance of the detected keywords to the topic. The topic may be pre-specified or estimated from the conference speech (keyword).

In operation 740, the conference management device (30) can identify the final ending of the speech based on voice characteristics and analyze the meeting atmosphere based on the final ending. The meeting atmosphere can include at least one atmosphere of coercion, obligation, urgency, encouragement, and possibility.

In operation 750, the conference management device (30) can calculate the speaking time of each participant and analyze the meeting participation of each participant based on the calculated speaking time. For example, the situation analysis unit (334) can check the meeting participation as a ratio of the speaking time to the total meeting time.

In operation 760, the conference management device (30) can identify noise level and sound type based on audio characteristics and identify other conference situations. For example, the conference management device (30) can identify noisy situations such as background music (e.g., music related to conference agenda) or ambient noise based on noise level or sound type and identify other conference situations.

In operation 770, the conference management device (30) may generate conference situation summary data based on at least one contextual information and store the generated data. For example, the conference management device (30) may generate conference situation summary information related to at least one of the emotional state of the speaker, the conference concentration of the participants (e.g., the speaking time of each participant), the topic relevance of the speech, the conference atmosphere, or the conference participation level based on the contextual analysis result.

In this way, the conference management device (30) according to one embodiment can analyze the conference situation from various angles and record the conference situation in summary, so that it can help even those who are absent from the meeting to accurately understand the conference situation. Furthermore, the conference management device (30) according to one embodiment can provide comments for appropriate conference mediation according to the conference situation analyzed from various angles, so that it can act as a conference moderator or help the conference proceed smoothly.

The various embodiments of this document and the terminology used herein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to encompass various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" may include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first", "second", or "first" or "second" may be used simply to distinguish the corresponding component from other corresponding components, and do not limit the corresponding components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as being “coupled” or “connected” to another component (e.g., a second component), with or without the terms “functionally” or “communicatively,” it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

The term "module" as used in this document may include a unit implemented in hardware, software or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be an integrally configured component or a minimum unit of the component or a portion thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of this document may be implemented as software (e.g., a program) including one or more commands stored in a storage medium (e.g., memory (320)) (e.g., built-in memory or external memory) that can be read by a machine (e.g., a user terminal). For example, a processor (e.g., a processor (e.g., a processor (330 of FIG. 3)) of a device (e.g., a user terminal (20)) can call at least one command among one or more commands stored from a storage medium and execute it. This enables the device to operate to perform at least one function according to the called at least one command. The one or more commands may include code generated by a compiler or code executable by an interpreter. The storage medium readable by the device may be provided in the form of a non-transitory storage medium. Here, ‘non-transitory’ only means that the storage medium is a tangible device and does not include a signal (e.g., electromagnetic wave), and this term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily in the storage medium.

According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded between a seller and a buyer as a commodity. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store ^TM ) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a part of the computer program product may be at least temporarily stored or temporarily generated in a machine-readable storage medium, such as a memory of a manufacturer's server, a server of an application store, or an intermediary server.

Components according to various embodiments of this document may be implemented in the form of software or hardware such as a digital signal processor (DSP), a field programmable gate array (FPGA), or an application specific integrated circuit (ASIC), and may perform predetermined roles. The 'components' are not limited to software or hardware, and each component may be configured to be in an addressable storage medium and may be configured to play one or more processors. As an example, the components may include components such as software components, object-oriented software components, class components, and task components, and processes, functions, properties, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables.

According to various embodiments, each component (e.g., a module or a program) of the above-described components may include a single or multiple entities. According to various embodiments, one or more of the components or operations of the above-described components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the components of the plurality of components identically or similarly to those performed by the corresponding component of the plurality of components prior to the integration. According to various embodiments, the operations performed by the module, program or other component may be executed sequentially, in parallel, repeatedly, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims (20)

In the conference management device,
A first extraction unit for extracting audio features related to a meeting using audio acquired at a meeting location;
A second extraction unit that extracts visual features related to conference attendees using a photographed image of the conference venue;
A content recognition unit that recognizes the meeting content based on the audio features and the visual features that are interrelated with each other;
A situation analysis unit that analyzes a meeting situation based on at least one of the above audio features and the above visual features; and
A summary unit that generates meeting summary data related to at least one of the meeting content or meeting situation based on the recognition/analysis results of the above content recognition unit and the above situation analysis unit.
A conference management device comprising: In claim 1,
Further comprising a communication module, wherein the first and second extraction units,
A conference management device that acquires the audio and the image respectively from a microphone and a camera installed in a conference venue through the communication module. In claim 1, the first extraction unit,
A conference management device that separates speech, noise and sound from the acquired audio, and extracts audio features including speech features, noise features and sound features from the separated signals. In claim 1, the second extraction unit,
A conference management device that extracts at least one visual feature of each of the attendees from the captured image, including lip shape, facial expression, gaze, head direction, gesture, or pose. In claim 1, the summary section comprises:
A conference management device that generates minutes by summarizing the conference contents based on at least one criterion among speaker-specific, topic-specific, or entire conference-specific based on the recognition result of the audio-visual feature by the content recognition unit. In claim 1, the situation analysis unit,
A conference management device that analyzes the emotional state of a speaker among participants based on at least one of the voice, lip shape, and facial expression features among the above audio features and the above visual features. In claim 1, the situation analysis unit,
A conference management device that checks the conference concentration of each attendee based on at least one of the above visual features: gaze, head direction, gesture, and pose. In claim 1, the situation analysis unit,
A conference management device that detects keywords of a speech based on the above audio features and confirms relevance to a topic based on the detected keywords. In claim 8, the situation analysis unit,
A conference management device that generates data related to a request for attention if the relevance to the above topic is less than a specified value. In claim 1, the situation analysis unit,
Based on the above audio features, the final suffix of the speech is identified, and the meeting atmosphere is identified from the final suffix.
A meeting management device wherein the above meeting atmosphere includes at least one of the following atmospheres: coercion, obligation, urgency, encouragement, and possibility. In claim 1, the situation analysis unit,
A conference management device that calculates the speaking time of each of the above attendees and analyzes the meeting participation of each attendee based on the calculated speaking time. In claim 11, the situation analysis unit,
A conference management device that generates data related to encouraging conference participation when the conference participation rate of the above attendees is below a specified percentage. In a method for managing a conference using a conference management device,
Actions to acquire audio and images related to meeting attendees;
An operation of extracting audio features related to the meeting using the acquired audio, and extracting visual features related to the meeting using the captured image;
An action of recognizing/analyzing the meeting content and meeting situation in relation to the audio features and visual features of each of the above attendees; and
An action to generate summary data related to at least one of the meeting content or meeting situation based on the results of the above recognition/analysis.
A method of managing a meeting, including: In claim 13, the recognizing/analyzing operation comprises:
A conference management method comprising an action of analyzing the emotional state of a speaker among participants based on at least one of the features of voice, lip shape, and facial expression among the above audio features and the above visual features. In claim 13, the recognizing/analyzing operation comprises:
A meeting management method including an action of checking the meeting concentration of each attendee based on at least one of the above visual features including gaze, head direction, gesture, and pose. In claim 13, the recognizing/analyzing operation comprises:
A conference management method for detecting keywords of a speech based on the above audio features and confirming relevance to a topic based on the detected keywords. In claim 13, the recognizing/analyzing operation comprises:
It includes an action of identifying the final suffix of a utterance based on the above audio features and identifying the meeting atmosphere from the final suffix.
A meeting management method wherein the above meeting atmosphere includes at least one of the following atmospheres: coercion, obligation, urgency, encouragement, and possibility. In claim 13, the recognizing/analyzing operation comprises:
A meeting management method for calculating the speaking time of each of the above attendees and analyzing the meeting participation of each attendee based on the calculated speaking time. In claim 11,
An operation of generating meeting situation summary data related to at least one of the participants' meeting participation, topic relevance of speech, meeting concentration, and meeting atmosphere based on the above audio features and the above visual features; and
A conference management method further comprising the action of providing the conference status summary data to the conference attendees. In the user terminal,
A communication module providing a designated communication channel;
Multi-channel microphones, omnidirectional cameras and output modules provided in the conference room; and
comprising a processor, said processor comprising:
Generate audio data of conference participants through the above multi-channel microphone, and generate image data of the participants through the above camera,
Synchronizing the above audio data and the above image data and transmitting them to an external conference management device through the communication module,
Through the above communication circuit, the external conference management device receives at least one of conference situation summary data or conference mediation data generated by recognizing/analyzing the conference content and conference situation in relation to the audio data and the image data,
A user terminal that provides at least one of the received data to the conference participants through the output module.

Images