JP2020113197A - Information processing apparatus, information processing method, and information processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To evaluate a relationship between participants. SOLUTION: An information processing device 10 obtains facial expression relationship information from facial expression information by a facial expression information acquisition unit 13 that acquires facial expression information of a plurality of participants, a voice information acquisition unit 14 that acquires utterance information of participants. The facial expression relationship information generation unit 15 to be generated, the utterance relationship information generation unit 16 to generate the utterance relationship information from the utterance information, and the relationship information to generate the relationship information from the facial expression relationship information and the utterance relationship information. It includes a generation unit 17. [Selection diagram] Fig. 1

Description

The present invention relates to an information processing device, an information processing method, and an information processing program.

A service that supports smooth communication between users is known as a conventional technique. Patent Literature 1 describes a telephone voice monitoring evaluation system that evaluates the performance of an operator by recognizing emotions based on voices in conversation between a call center operator and a customer and performing voice analysis by combining voices and emotions. Has been done. Patent Literature 2 describes an emotion matching device that recognizes emotions indicated by a chat sentence input by a user and obtains the similarity of emotions between users.

JP-A-2017-135642 (Published August 3, 2017) JP-A-2005-284822 (Published October 13, 2005)

However, the above-described conventional techniques have a problem that multi-faceted emotion recognition cannot be performed because the emotion of the speaker is recognized based on only the voice in the conversation or only the input chat sentence. ..

An aspect of the present invention is to provide a communication support technique for recognizing emotions of each user during a conversation from multiple sides and notifying a conversation evaluation based on the recognized emotions.

In order to solve the above problems, an information processing apparatus according to an aspect of the present invention provides first facial expression information regarding a facial expression of a first participant among a plurality of participants, and first facial expression information among the plurality of participants. Facial expression information acquisition unit for acquiring second facial expression information regarding facial expressions of the second participant, first utterance information regarding utterances of the first participant, and second participant of the plurality of participants. The first participant and the second participant with reference to the voice information acquisition unit that acquires the second utterance information related to the utterance of the user and the first facial expression information and the second facial expression information. The facial expression relationship information generating unit that generates facial expression relationship information indicating a relationship related to facial expressions, and the first participant and the first utterance information with reference to the first utterance information and the second utterance information. The first participant with reference to the utterance relationship information generating unit that generates utterance relationship information indicating a relationship regarding utterance with the second participant, and the facial expression relationship information and the utterance relationship information. And a relationship information generating unit that generates relationship information that is information indicating the relationship between the second participant and the second participant.

According to this, the relationship between the participants in the conference can be evaluated based on both the voice information and the facial expression information of each participant.

In the information processing device according to the aspect, the relationship information is real-time or time-dependent information indicating a relationship between the first participant and the second participant.

According to this, the relationship between the participants in the conference can be evaluated in real time based on both the voice information and the facial expression information of each participant.

In the information processing apparatus according to the one aspect, the first facial expression information includes a plurality of first indexes expressing the facial expression of the first participant, and the second facial expression information includes , A plurality of second indexes expressing the facial expressions of the second participant, and the facial expression relationship information generating unit is configured to detect a facial expression difference regarding a difference between the first index and the second index. Information is generated, and the generated facial expression difference information is included in the facial expression relationship information.

According to this, since the plurality of indexes are used to express the facial expression of the participant referred to in order to generate the facial expression relationship information, the facial expression of the participant can be expressed more accurately.

In the information processing device according to the aspect, the first facial expression information includes first visual line information regarding a visual line direction of the first participant, and the second facial expression information includes the first visual line information. The second line-of-sight information regarding the line-of-sight of the second participant is included, and the facial expression relationship information generation unit refers to the first line-of-sight information and the second line-of-sight information to determine the line-of-sight relationship. Information is generated, and the generated line-of-sight relationship information is included in the facial expression relationship information.

According to this, since the participant's facial expression information referred to in order to generate the facial expression relationship information also includes the participant's line-of-sight information, the participant's facial expression can be expressed more accurately.

In the information processing device according to the aspect, the utterance relationship information generating unit may include the utterance time of the first participant indicated by the first utterance information and the second utterance time indicated by the second utterance information. The utterance time relationship information indicating the relationship with the utterance time of the participant is generated, and the generated utterance time relationship information is included in the utterance relationship information.

According to this, since the utterance time relationship information is included in the utterance information of the participant referred to for generating the utterance relationship information, the utterance relationship information of the participant can be generated more accurately.

In the information processing device according to the aspect, the utterance relationship information generating unit may include utterance content included in a specific category in at least one of the first utterance information and the second utterance information. It is determined whether or not there is information, and information according to the determination result is included in the utterance relationship information.

According to this, since the utterance relationship information also includes information according to the determination result of whether or not the utterance content included in a specific category is included, it is possible to more accurately generate the relationship information between the participants. You can

In the information processing device according to the one aspect, the utterance relationship information generating unit is a word having a relatively high appearance frequency within a predetermined time from at least one of the first utterance information and the second utterance information. And the extracted word is included in the utterance relationship information.

According to this, since the utterance relationship information includes the upper part of a word having a high frequency, it is possible to more accurately generate the relationship information between the participants.

In the information processing device according to the one aspect, the relationship information generation unit refers to the relationship information and presents presentation information to be presented to at least one of the first participant and the second participant. To generate.

According to this, the participant can be made to recognize the relationship information by presenting the presentation information to the participant.

The presentation information includes information indicating a ratio between the utterance time of the first participant and the utterance time of the second participant, the line-of-sight direction of the first participant, and the second It contains information about changes over time in the match rate with the gaze direction of the participants.

According to this, by presenting the presentation information to the participants, the participants can be made aware of the ratio of the utterance time of each participant and the temporal change in the matching rate of the gaze direction of each participant.

In the information processing device according to the aspect, the facial expression relationship information generating unit and the utterance relationship information generating unit further refer to participant information indicating attributes of the first and second participants, The facial expression relationship information and the utterance relationship information are generated.

According to this, since the participant's attribute is also referred to in the generation of the facial expression relationship information and the utterance relationship information, more accurate facial expression relationship information and utterance relationship information can be generated.

Further, in order to solve the above problems, an information processing method according to an aspect of the present invention provides first facial expression information regarding a facial expression of a first participant among a plurality of participants, and among the plurality of participants. A facial expression information acquisition step of acquiring second facial expression information regarding the facial expression of the second participant, first speech information regarding the speech of the first participant, and second participation among the plurality of participants. The first participant and the second participant with reference to the voice information acquisition step of acquiring second utterance information related to a person's utterance, and the first facial expression information and the second facial expression information. A facial expression relationship information generating step of generating facial expression relationship information indicating a relationship related to a facial expression with a person, and the first participant with reference to the first utterance information and the second utterance information. The first participation with reference to the utterance relationship information generating step of generating utterance relationship information indicating a relationship regarding utterance with the second participant, and the facial expression relationship information and the utterance relationship information. Relationship information generating step of generating relationship information which is real-time or time-dependent information indicating a relationship between a person and the second participant.

According to this, the relationship between the participants in the conference can be evaluated based on the voice information and the facial expression information of each participant.

Further, in order to solve the above problems, an information processing program according to an aspect of the present invention is an information processing program for causing a computer to function as the information processing device according to any one of the above, and the facial expression information acquisition unit. A computer is caused to function as the voice information acquisition unit, the facial expression relationship information generation unit, the utterance relationship information generation unit, and the relationship information generation unit.

According to this, the relationship between the participants in the conference can be evaluated based on the voice information and the facial expression information of each participant.

According to one aspect of the present invention, it is possible to evaluate the relationship between participants in a conference based on the voice information and facial expression information of each participant.

It is a block diagram which shows an example of the component of the information processing system containing the information processing apparatus which concerns on one Embodiment of this invention. It is a figure showing an outline of an information processing system containing an information processor concerning one embodiment of the present invention. It is a figure which shows the outline of the flow of the data in the information processing system containing the information processing apparatus which concerns on one Embodiment of this invention. It is a figure which shows an example of the information which the information processing system containing the information processing apparatus which concerns on one Embodiment of this invention presents. It is a figure which shows the other example of the information which the information processing system containing the information processing apparatus which concerns on one Embodiment of this invention presents.

[Embodiment 1]
Hereinafter, an embodiment of the present invention will be described in detail. FIG. 1 is a diagram showing an outline of an information processing system 100 including the information processing apparatus 10 of this embodiment. As shown in FIG. 1, the information processing system 100 includes an information processing device 10, a first terminal device 20, and a second terminal device 30. The number of terminal devices is not limited to this embodiment and may be three or more.

FIG. 2 is a diagram showing an outline of an information processing system 100 including the information processing device 10 according to the embodiment of the present invention. In the information processing system 100, as shown in FIG. 2, between the first participant 200 who uses the first terminal device 20 and the second participant 201 who uses the second terminal device 30. Evaluate the relationship.

In the information processing system 100, information is obtained based on facial expression information and speech information of the first participant 200 and the second participant 201 during the meeting, which are obtained from the first terminal device 20 and the second terminal device 30. The processing device 10 evaluates the relationship between the first participant 200 and the second participant 201 during the conference. The information processing system 100 displays the result of evaluation of the relationship on at least one of the first terminal device 20 and the second terminal device 30 so that the first participant 200 and the second participant 201 can receive the result. The evaluation results are fed back to at least one side in real time to encourage improvement of the communication status during the meeting.

In the present embodiment, the “meeting” is not limited to a meeting in a narrow sense, but includes an interview, an interview, a counseling, an inquiry, a customer service, an interview, a consultation, and the like. As an example,
・Interview with supervisor and subordinates ・Interview with patient by doctor ・Counseling of target person by counselor ・Customer service and customer consultation at customer ・Remote communication such as Web Meeting ・For video image such as e-learning Communication etc. are included.

Further, FIG. 3 is a diagram showing an outline of a data flow in the information processing system 100.

[First Terminal Device 20]
As shown in FIG. 1, the first terminal device 20 includes a camera 21, a microphone 22, a display unit 23, a control unit 24, a speaker 25, and a communication unit 26.

<Video acquisition process>
The camera 21 images the first participant and supplies the captured image to the control unit 24. Here, the image captured by the camera 21 is preferably a moving image, and in the case of this configuration, as shown in FIG. 3, the camera 21 is included in the moving image file and the moving image file for the control unit 24. At least one of the moving image file lists, which is a list of each image, is supplied. The camera 21 supplies the control unit 24 with a time stamp indicating the imaging time of each image included in the moving image file.

The first terminal device 20 may have a configuration including a plurality of cameras, and in this configuration, the control unit 24 can identify the camera 21 by referring to the camera device identification information.

<Image recognition processing>
The control unit 24 refers to the moving image file, the moving image file list, and the time stamp supplied from the camera 21, and performs the image recognition process.

As an example, as shown in FIG. 3, the control unit 24 performs the image recognition process with reference to the moving image file, the moving image file list, and the time stamp, thereby performing the time-series facial expression value, the time-series face part coordinate, and the time-series line-of-sight. Calculate the coordinates. The time-series facial expression value, the time-series facial part coordinate, and the time-series gaze coordinate are examples of first facial expression information regarding the facial expression of the first participant.

<Voice acquisition processing>
The microphone 22 mainly collects the voices uttered by the first participant, provides the control unit 24 with a voice file indicating the collected voices, and a time stamp for specifying the point of utterance in the voice file. Supply.

The first terminal device 20 may be configured to include a plurality of microphones. In this configuration, the control unit 24 can identify the microphone 22 by referring to the audio device identification information.

<Utterance recognition processing>
The control unit 24 refers to the audio file and the time stamp supplied from the microphone 22 and performs the speech recognition process.

As an example, as illustrated in FIG. 3, the control unit 24 generates the section time-series text data by performing the speech recognition process with reference to the audio file and the time stamp. Here, the section time-series text data is information that mainly shows the contents uttered by the first participant in time series as text data. The section time-series text data is an example of first utterance information indicating the utterance of the first participant.

[Second terminal device 30]
The second terminal device 30 also includes a camera 31, a microphone 32, a display unit 33, a control unit 34, a speaker 35, and a communication unit 36.

<Video acquisition process>
The camera 31 images the second participant and supplies the captured image to the control unit 34. Here, the image captured by the camera 31 is preferably a moving image, and in the case of this configuration, the camera 31 is included in the moving image file and the moving image file for the control unit 34, as shown in FIG. At least one of the moving image file lists, which is a list of each image, is supplied. The camera 31 supplies the control unit 34 with a time stamp indicating the image capturing time of each image included in the moving image file.

Note that the second terminal device 30 may have a configuration including a plurality of cameras, and in this configuration, the control unit 34 can identify the camera 31 by referring to the camera device identification information.

<Image recognition processing>
The control unit 34 refers to the moving image file, the moving image file list, and the time stamp supplied from the camera 31, and performs image recognition processing.

As an example, as shown in FIG. 3, the control unit 34 performs the image recognition process with reference to the moving image file, the moving image file list, and the time stamp, thereby performing the time-series facial expression value, the time-series face part coordinate, and the time-series line-of-sight. Calculate the coordinates. The time-series facial expression value, the time-series facial part coordinate, and the time-series gaze coordinate are examples of second facial expression information regarding the facial expression of the second participant.

<Voice acquisition processing>
The microphone 32 mainly collects the voices uttered by the second participant, provides the control unit 34 with a voice file indicating the collected voices, and a time stamp for specifying the point of utterance in the voice file. Supply.

The second terminal device 30 may be configured to include a plurality of microphones, and in this configuration, the control unit 34 can identify the microphone 32 by referring to the audio device identification information.

<Utterance recognition processing>
The control unit 34 refers to the audio file and the time stamp supplied from the microphone 32 and performs the speech recognition process.

As an example, as illustrated in FIG. 3, the control unit 34 generates the section time-series text data by performing the speech recognition process with reference to the audio file and the time stamp. Here, the section time-series text data is information that mainly shows the content uttered by the second participant in time series as text data. The section time-series text data is an example of second utterance information indicating the utterance of the second participant.

[Information processing device 10]
The information processing device 10 includes a facial expression information acquisition unit 13, a voice information acquisition unit 14, a facial expression relationship information generation unit 15, an utterance relationship information generation unit 16, and a relationship information generation unit 17. The information processing device 10 further includes a communication unit 11. The facial expression information acquisition unit 13, the voice information acquisition unit 14, the facial expression relationship information generation unit 15, the utterance relationship information generation unit 16, and the relationship information generation unit 17 are included in the calculation unit 12.

(Facial expression information acquisition unit 13)
The facial expression information acquisition unit 13 relates to first facial expression information regarding the facial expression of the first participant among the plurality of conference participants and the facial expression of the second participant among the plurality of conference participants via the communication unit 11. And the second facial expression information.

<Numerical data cleaning process>
The facial expression information acquisition unit 13 may, for example, via the communication unit 11, time-series facial expression values, which are time-series numerical data included in the facial expression information regarding the first participant, time-series face part coordinates, and time-series gaze coordinates. The numerical data cleaning process is performed on the time-series numerical data by performing the following process as an example.
・Delete the invalid data section ・Average the data in the valid data section ・Convert to variance and number of terms The facial expression information acquisition unit 13 performs the above-mentioned numerical data washing processing to determine the time of the section regarding the first participant. Generate series numerical data. The section time-series numerical data includes time-series facial expression values, time-series face part coordinates, and time-series line-of-sight coordinates in the effective section.

The facial expression information acquisition unit 13 performs the same process on the facial expression information regarding the second participant, and generates the section time series numerical data regarding the second participant.

<Detection of facial expression>
The facial expression information acquisition unit 13 refers to the section time-series numerical data regarding the first participant to calculate a plurality of first indexes expressing the facial expression of the first participant. In addition, the facial expression information acquisition unit 13 refers to the section time-series numerical data regarding the second participant, and calculates a plurality of second indexes expressing the facial expression of the second participant.

Here, examples of the index expressing the facial expression include the following indices.
・Anger
・Insult (contempt)
・Disgust
・Fear
・Happiness
・Neutral
・Sadness
・Surprise
Therefore, the index expressing the facial expression can also be referred to as an index expressing the emotion indicated by the facial expression.

Note that the facial expression information acquisition unit 13 may use the time-series facial expression values included in the section time-series numerical data regarding the first participant, as they are, as a plurality of first indexes expressing the facial expression of the first participant. Good. Similarly, the facial expression information acquisition unit 13 uses the time-series facial expression values included in the section time-series numerical data regarding the second participant, as they are, as a plurality of second indexes expressing the facial expression of the second participant. Good.

The facial expression of the first participant and the facial expression of the second participant can also be expressed as a vector having the above-mentioned index as each component. These vectors are sometimes called facial expression vectors.

It should be noted that the technique of detecting the facial expression of each participant and the technique of indexing and expressing the emotion indicated by the detected facial expression do not limit the present embodiment, and a known technique can be used, for example.

<Detection of line of sight>
In addition, the facial expression information acquisition unit 13 acquires information regarding the line-of-sight directions of the first participant and the second participant from the first terminal device 20 and the second terminal device 30 via the communication unit 11. .. Specifically, as an example, the facial expression information acquisition unit 13 acquires the time-series line-of-sight coordinates included in the section time-series numerical data regarding the first participant described above as the information regarding the line-of-sight direction of the first participant. .. Similarly, the facial expression information acquisition unit 13 acquires the time-series line-of-sight coordinates included in the above-described section time-series numerical data regarding the second participant, as information regarding the line-of-sight direction of the second participant.

The method of acquiring the line-of-sight coordinates is not particularly limited, but a point light source (not shown) is provided in the first terminal device 20 and the second terminal device 30, and a corneal reflection image of light from the point light source is captured by the camera. For example, a method of acquiring the line-of-sight coordinates of the user by photographing with the camera 21 and the camera 31 for a predetermined time. The type of point light source is not particularly limited, and examples thereof include visible light and infrared light. By using an infrared LED, for example, the line-of-sight coordinates can be acquired without causing discomfort to the user.

<Detection of distance>
The facial expression information acquisition unit 13 also acquires the time-series face part coordinates of the first participant included in the section time-series numerical data, and calculates the distance between the first participant and the imaging unit (camera 21). May be calculated. The facial expression information acquisition unit 13 also acquires the time-series face part coordinates of the second participant included in the section time-series numerical data, and calculates the distance between the second participant and the image capturing unit (camera 31). May be calculated. The distance between the participant and the imaging means can be calculated, for example, as the reciprocal of the outer corner of the eye, which is obtained by correcting the outer corner of the eye in the captured image obtained from the face part coordinates as the outer corner of the eye. ..

(Voice information acquisition unit 14)
The voice information acquisition unit 14 acquires the first utterance information regarding the utterance of the first participant and the second utterance information regarding the utterance of the second participant among the plurality of conference participants. That is, the voice information acquisition unit 14 acquires information about the utterances of the first participant and the second participant from the first terminal device 20 and the second terminal device 30 via the communication unit 11.

As an example, the voice information acquisition unit 14 acquires the time-series utterance text included in the section time-series text data regarding the above-mentioned first participant. Similarly, the voice information acquisition unit 14 acquires, as an example, the time-series utterance text included in the section time-series text data regarding the second participant described above.

Further, as an example, the voice information acquisition unit 14 acquires the time-series utterance text relating to the first participant and the time-series face part coordinates at the time of utterance of the time-series utterance text. The voice information acquisition unit 14 refers to the time-series face part coordinates and, if the first participant's mouth is open at the time of utterance of the time-series utterance text, sets the time-series utterance text to the first participant. Included in the first utterance information related to the utterance. Similarly, the voice information acquisition unit 14 acquires the time-series utterance text regarding the second participant and the time-series face part coordinates at the time of utterance of the time-series utterance text. The voice information acquisition unit 14 refers to the time-series face part coordinates and, if the mouth of the second participant is open at the time of utterance of the time-series utterance text, sets the time-series utterance text to the second participant. Included in the second utterance information regarding the utterance of. Accordingly, even if a simple microphone having no directivity is used as the microphone 22 or the microphone 32, the person who speaks can be specified.

(Facial expression relationship information generation unit 15)
The facial expression relationship information generation unit 15 refers to the first facial expression information and the second facial expression information to generate facial expression relationship information indicating a facial expression relationship between the first participant and the second participant. To generate.

The satisfaction of a conference participant with respect to the conference depends not only on the content and conclusion of the conference, but also on the presence or absence of good communication between the participants. The communication state between the participants is represented by the relationship between the participants during the conference, and the relationship between the participants can be evaluated by the degree of agreement of emotions. The facial expression relationship information generation unit 15 acquires the first facial expression information and the second facial expression information from the facial expression information acquisition unit 13 and, based on these facial expression information, determines the degree of coincidence of emotions among the participants in the conference. By evaluating the facial expressions of both participants, the communication status between the participants is evaluated in real time.

The facial expression information acquired by the facial expression relationship information generation unit 15 from the facial expression information acquisition unit 13 is calculated based on the section time-series numerical data, that is, the facial expression of each participant in real time or over time. It is calculated based on information. The facial expression relationship information generation unit 15 generates facial expression relationship information based on the facial expression information of each participant in real time or over time. It expresses the relationship regarding various facial expressions.

<Facial expression matching rate judgment>
As described above, the first facial expression information includes a plurality of first indexes expressing the facial expression of the first participant, and the second facial expression information includes the facial expression of the second participant. A plurality of second indexes expressing the above are included.

The facial expression relationship information generation unit 15 may generate facial expression difference information regarding the difference between the first index and the second index, and may include the generated facial expression difference information in the facial expression relationship information.

As an example, the facial expression relationship information generating unit 15 displays a first facial expression vector having a plurality of indices expressing the facial expression of the first participant and a plurality of indices expressing the facial expression of the second participant. The expression disagreement amount is calculated using the absolute value of the difference from the second expression vector as an element. It can be said that the calculated facial expression dissimilarity amount is also an index representing the harmony state of emotions during the conference between the participants. In addition, the facial expression relationship information generating unit 15 may calculate the facial expression matching rate as an index indicating the rate of matching facial expressions from the start of the conference to the present time. The facial expression matching rate is obtained, for example, by subtracting the time when the facial expressions do not match from the time from the start of the conference to the present time and dividing by the time from the start of the conference to the present time.

<Gaze matching rate determination>
In addition, the first facial expression information is configured to include first line-of-sight information regarding the first participant's line-of-sight direction, and the second facial expression information includes second line-of-sight information regarding the second participant's line-of-sight direction. It may be configured to include. The facial expression relationship information generating unit 15 may generate the visual line relationship information by referring to the first visual line information and the second visual line information, and may include the generated visual line relationship information in the facial expression relationship information. ..

As an example, the facial expression relationship information generating unit 15 calculates the line-of-sight matching rate between the first participant and the second participant as the line-of-sight relationship information. It can be said that the calculated line-of-sight match rate is an index indicating a state of being aware of the states of other participants during the conference. More specifically, first, the control unit 24 or the facial expression relationship information generation unit 15 identifies the eye position of the first participant in the conference room by analyzing the image captured by the camera 21, and the control unit 34 or the facial expression relationship information generation unit 15 specifies the eye position of the second participant in the conference room by analyzing the image captured by the camera 31.

Then, the facial expression relationship information generation unit 15 determines at each time point whether or not the line-of-sight direction of the first participant indicated by the first line-of-sight information is toward the eye of the second participant, By determining whether or not the line-of-sight direction of the second participant indicated by the line-of-sight information of No. 2 is toward the first participant's eyes, the line-of-sight of the first participant and the second line-of-sight Determine whether the line of sight of the participant matches.

As an example, the facial expression relationship information generation unit 15 sets the line-of-sight flag of the first participant to 1 when it is determined that the line-of-sight of the first participant is facing the eyes of the second participant. .. In addition, the facial expression relationship information generation unit 15 sets the line-of-sight flag of the second participant to 1 when it is determined that the line-of-sight of the second participant is facing the eyes of the first participant. Then, the facial expression relationship information generating unit 15 determines that the line of sight matches when both line of sight flags are 1.

Then, the facial expression relationship information generation unit 15 calculates the line-of-sight matching rate as an index indicating the rate of matching the line-of-sight from the start of the conference to the present time. The line-of-sight matching rate is obtained, for example, by dividing the time at which the line of sight matches the time from the start of the conference to the current time.

Note that the determination as to whether or not the lines of sight are toward each other's eyes may be configured to further refer to position information indicating a relative positional relationship between the first terminal device 20 and the second terminal device 30. ..

Further, when the lines of sight of each other are not necessarily the eyes of the other party but face the direction of the other party's face or the body of the other party, it may be determined that the lines of sight match.

Further, when the participants have a meeting via the Internet or the like, the line-of-sight matching rate between the participants on the screen of the terminal device is calculated. More specifically, as an example, the position of the face of the second participant displayed on the display screen of the first terminal device 20 is specified as the coordinates on the display screen, and the position of the second participant is compared with the specified coordinates. When the line of sight of one participant is directed, the line-of-sight flag of the first participant is set to 1. Similarly, the position of the face of the first participant displayed on the display screen of the second terminal device 30 is specified as the coordinates on the display screen, and the line of sight of the second participant with respect to the specified coordinates. If is directed, the line-of-sight flag of the second participant is set to 1.

<Judgment rate before>
In addition, the facial expression relationship information generation unit 15 may calculate the previous turnover rate of the first participant and the second participant, and may include the calculated previous turnover rate in the facial expression relationship information. It can be said that the calculated turn-over rate is an index representing a state in which the participant is interested in the speech of another participant during the conference. As an example, when the distance between the first participant and the second participant from the respective image capturing means is less than a threshold value within a preset fixed time, the facial expression relationship information generating unit 15 First, it is determined that the first participant and the second participant are in the forward leaning state.

Then, the facial expression relationship information generating unit 15 identifies the ratio of the time when the first participant is leaning forward in the time from the start of the conference to the present time, as the previous leaning rate regarding the first participant, and the second The percentage of time that the participants in the previous turn are in the previous turn is identified as the previous turn rate for the second participant.

In addition, the facial expression relationship information generation unit 15 determines the face image size obtained based on the distances from the image capturing means of the first participant and the second participant from the change within a preset fixed time to the conference. The posture of the inside participant may be calculated and included in the facial expression relationship information. It can be said that the calculated attitude of the participant is an index representing an attitude suitable for listening to the utterances of other participants during the conference.

Further, the facial expression relationship information generating unit 15 calculates a correlation between the change in the posture of the first participant and the change in the second facial expression vector of the second participant, and includes the correlation in the facial expression relationship information. May be. The correlation between the change in posture and the change in facial expression vector can be said to be an index representing the influence of the posture of one participant on the facial expressions of other participants. Similarly, the facial expression relationship information generating unit 15 calculates the correlation between the change in the posture of the second participant and the change in the first facial expression vector of the first participant, and uses the correlation as the facial expression relationship information. May be included.

In addition, the facial expression relationship information generation unit 15 refers to the postures of the first participant and the postures of the second participant, and the degree of similarity between the postures of the first participant and the second participant. May be calculated and the calculated similarity may be included in the facial expression relationship information. The similarity of the posture states represents the mirroring state, and can be said to be an index indicating a state in which the participant is interested in the speech of another participant during the conference.

The facial expression relationship information generating unit 15 may generate facial expression relationship information by further referring to the participant information indicating the attributes of the first and second participants. Participant information indicating the attributes of the participant includes at least one of age, sex, blood type, personality, place of origin, family relationship, job title, years of service, number of job changes, job history, etc. of the participant. The participant information also includes the usage history of the system.

As an example, the facial expression relationship information generation unit 15 refers to the participant information, and when the participant determines that the particular facial expression is likely to appear, the weighting coefficient smaller than 1 is set to the index corresponding to the specific facial expression. The facial expression vector of the participant may be corrected by performing a correction by multiplying by, and the facial expression relationship information may be generated using the corrected facial expression vector.
For example, when the participant information indicating the attribute of the first participant indicates that the first participant is shy, the facial expression relationship information generation unit 15 determines that the index is “neutral”. Is multiplied by a weight of 0.8, and the remaining 0.2 weight is proportionally distributed to another index, thereby correcting the facial expression vector of the first participant and the corrected facial expression. The facial expression relationship information may be generated using a vector.

The information processing device 10 is configured to further acquire biometric information such as a participant's pulse wave and brain wave, and environmental information such as temperature, humidity, carbon dioxide concentration, and illuminance around the participant, and a facial expression relationship information generation unit. The reference numeral 15 may generate facial expression relationship information by further referring to biometric information and environment information.

As an example, the facial expression relationship information generating unit 15 expresses the stress state of the participant determined from the pulse wave or respiration of the first participant, and the facial expression of the second participant immediately before or at that time. The facial expression of the second participant that stresses the first participant is estimated with reference to the index of. Then, the facial expression relationship information generating unit 15 may recognize the estimated facial expression of the second participant as an NG facial expression for the first participant and include the information in the facial expression relationship information. The NG facial expression of one participant with respect to another participant can be said to be an index representing the effect of the facial expression of one participant on the stress state of another participant. Similarly, the facial expression relationship information generation unit 15 may estimate the facial expression of the first participant that gives stress to the second participant, and recognize the NG facial expression for the second participant.

In addition, the facial expression relationship information generating unit 15 changes the environmental information around the participant, the first facial expression vector of the first participant, and the second participant's first facial expression vector within a predetermined period. The correlation with the change in the average value of the facial expression vector 2 may be calculated and the correlation may be included in the facial expression relationship information. The correlation between the change in environmental information and the change in the average value of facial expression vectors can be said to be an index representing the influence of the surrounding environment of the participants on the communication state between the participants.

<Dialogue management processing>
As an example, the facial expression relationship information generation unit 15 uses, via the communication unit 11, the user IDs of the first participant and the second participant, the time at which the participant represented by the user ID starts and the end of the meeting. The dialogue management processing is performed by referring to the time stamp indicating the time when the dialogue was performed. The facial expression relationship information generation unit 15 extracts the user IDs of the other participants who are interacting at that time point with respect to the data at a certain time point of the section time-series numerical data regarding one participant, and in which participant It is also possible to determine whether or not the data is obtained from the above and include the result in the facial expression relationship information.

(Utterance relationship information generation unit 16)
The utterance relationship information generation unit 16 refers to the first utterance information and the second utterance information to generate utterance relationship information indicating a relationship regarding utterance between the first participant and the second participant. To generate. The utterance relationship information generation unit 16 acquires the first utterance information and the second utterance information from the voice information acquisition unit 14, and based on these utterance information, determines the degree of coincidence of emotions among the participants in the conference. The communication state between participants is evaluated by evaluating the utterances of both participants.

Each utterance information acquired by the utterance relationship information generating unit 16 from the voice information acquiring unit 14 is calculated based on the section time-series text data, that is, relating to the utterance of each participant in real time or over time. It is calculated based on information. The utterance relationship information generation unit 16 generates the utterance relationship information based on the utterance information of each participant in real time or over time. It represents a relationship related to different utterances.

<Speech ratio judgment>
The utterance relationship information generating unit 16 is an utterance time relationship indicating a relationship between the utterance time of the first participant indicated by the first utterance information and the utterance time of the second participant indicated by the second utterance information. Information may be generated, and the generated utterance time relationship information may be included in the utterance relationship information.

As an example, the utterance relationship information generating unit 16 calculates the utterance ratio between the utterance time of the first participant and the utterance time of the second participant within a predetermined fixed time, and calculates the utterance relationship information. include. It can be said that the calculated utterance ratio is also an index showing the equality of the relationship between the participants.

<Speech frequency judgment>
Further, the utterance relationship information generating unit 16 determines whether or not at least one of the first utterance information and the second utterance information includes utterance content included in a specific category, and the determination result May be included in the utterance relationship information.

Examples of specific categories included in utterances include open questions, action words (so?, I see, sure), parrots, utterance interruptions, utterance omissions, negative words (but but). .. As an example, the utterance relationship information generating unit 16 calculates the frequency of utterance of the utterance content included in such a specific category within a predetermined fixed time period, and outputs information regarding the calculated frequency as the utterance relationship. Include in information.

Specifically, as an example, the specific category included in the utterance content is set as an open question, and the utterance relationship information generation unit 16 includes the categorized time category text data of the first participant within a certain period of time. Extract text data that represents an open question. Then, the utterance relationship information generating unit 16 calculates the frequency of utterances of the open question as an open question rate by dividing the number of words of the extracted text data by the number of words of all the text data within the fixed time. To do. Similarly, the utterance relationship information generating unit 16 calculates the open question rate from the section time series text data of the second participant. Then, the utterance relationship information generating unit 16 compares the open question rate of the first participant with the open question rate of the second participant, calculates the open question rate, and includes it in the utterance relationship information. It can be said that the calculated open question ratio is an index showing the equality of the relationship between the participants. Further, the utterance relationship information generation unit 16 may include the open question rate of the first participant and the open question rate of the second participant in the utterance relationship information.

Similarly, the specific category included in the utterance content is set as the action prompting word, and the utterance relationship information generating unit 16 sets the action prompting word included within the fixed time from the section time series text data of the first participant. Extract the text data to represent. Then, the utterance relationship information generating unit 16 divides the number of words of the extracted text data by the number of words of all the text data within the above-mentioned fixed time to prompt the action and the frequency of the utterance of the words to obtain the question rate. calculate.
Similarly, the utterance relationship information generating unit 16 calculates the prompting question rate from the section time-series text data of the second participant. Then, the utterance relationship information generation unit 16 compares the prompted question rate of the first participant with the prompted question rate of the second participant, calculates the prompted question rate, and includes it in the spoken relationship information. The calculated prompting question ratio can be said to be an index showing the equality of the relationship between the participants. Further, the utterance relationship information generating unit 16 may include the urging question rate of the first participant and the urging question rate of the second participant in the utterance relationship information.

<Word-based evaluation>
Further, the utterance relation information generating unit 16 extracts a word having a relatively high appearance frequency within a predetermined time from at least one of the first utterance information and the second utterance information, and the extracted word is the utterance relation. May be included in sex information.

As an example, the utterance relationship information generating unit 16 participates in the number of appearances of each word included within a predetermined fixed time from the section time series text data of each of the first participant and the second participant. The words are calculated and ranked for each person, and a plurality of words having a relatively high appearance frequency are extracted from the top. Then, the utterance relationship information generation unit 16 includes the extracted words with the highest appearance frequency in the utterance relationship information as frequent words. In addition, the utterance relationship information generating unit 16 determines whether or not the frequent words of the first participant and the second participant and their ranks match, and includes the determination result in the utterance relationship information. Good.

Further, the utterance relationship information generating unit 16 extracts, for each participant, a word included within a predetermined fixed time from the section time series text data of each of the first participant and the second participant. The matching rate of the extracted words may be calculated and included in the utterance relationship information. The word matching rate can be calculated as a ratio of words matched by the first participant and the second participant in all the words included in the section time-series text data within a predetermined fixed time. it can. The calculated word concordance rate can be said to be an index of whether parrots have been successfully returned.

In addition, the utterance relationship information generating unit 16 extracts the utterance timing from the section time series text data of each of the first participant and the second participant, and calculates the temporal overlap of the utterance timing. Good. Then, the utterance relationship information generating unit 16 may calculate the number of overlaps within a predetermined fixed time as the overlap frequency and include it in the utterance relationship information. The calculated overlap frequency can be said to be an index indicating the frequency of interrupting the utterances of other participants.

Further, the utterance relationship information generating unit 16 may generate the utterance relationship information by further referring to the participant information indicating the attributes of the first and second participants. Participant information indicating the attributes of the participant includes at least one of age, sex, blood type, personality, place of origin, family relationship, job title, years of service, number of job changes, job history, etc. of the participant. The participant information also includes the usage history of the system.

The information processing apparatus 10 is configured to further acquire biological information such as the participant's pulse wave and brain wave, and environmental information such as the temperature, humidity, carbon dioxide concentration, and illuminance around the participant, and the utterance relationship information generating unit. 16 may further generate the utterance relationship information by further referring to the biometric information and the environment information.

Further, as an example, the utterance relationship information generating unit 16 determines the stress state of the participant determined from the pulse wave or respiration of the first participant, and the section time series text of the second participant immediately before or at that time. And the text data of the second participant that stresses the first participant is estimated. Then, the utterance relationship information generating unit 16 may recognize the estimated text data of the second participant as an NG word for the first participant and include the information in the utterance relationship information. The NG word of one participant with respect to another participant can also be said to be an index representing the influence of the speech of one participant on the stress state of another participant. Similarly, the utterance relationship information generating unit 16 may estimate the text data of the first participant who gives stress to the second participant, and may recognize the NG word for the second participant.

Further, as an example, the utterance relationship information generating unit 16 determines the activity level of the participant's thought determined from the electroencephalogram of the first participant and the section time series text data of the second participant immediately before or at that time. With reference to, the text data of the second participant that activates the thought of the first participant is estimated. Then, the utterance relationship information generating unit 16 may recognize the estimated text data of the second participant as an important word for the first participant and include the information in the utterance relationship information. It can be said that the important word of one participant with respect to another participant is an index showing the influence of the speech of one participant on the activation of thinking of other participants. Similarly, the utterance relationship information generating unit 16 may estimate the text data of the first participant that activates the thinking of the second participant, and may recognize the important word for the second participant.

Further, the utterance relationship information generating unit 16 may calculate the degree of coincidence of voice tones among the participants, the degree of coincidence of speech speed, the degree of coincidence of volume, and the like.

Further, the utterance relationship information generating unit 16 extracts logs representing utterances of the first participant and the second participant from the accumulated section time-series text data, acquires the morphological analysis data thereof. , Frequent words in the past utterance may be listed and included in the utterance relationship information. The frequently-used words in the past utterances can be used to assist the decision of the theme of the conference by presenting them to both participants in the conference.

<Dialogue management processing>
As an example, the utterance relationship information generating unit 16 uses, via the communication unit 11, the user IDs of the first participant and the second participant, the time when the participant represented by the user ID starts, and the end time of the meeting. The dialogue management processing is performed by referring to the time stamp indicating the time when the dialogue was performed. The utterance relationship information generating unit 16 extracts the user IDs of other participants who are interacting at that time with respect to the data at a certain time of the section time-series text data regarding one participant, and which participant is talking It is also possible to determine whether or not the data is obtained from the above and include the result in the utterance relationship information.

(Relationship information generating unit 17)
The relationship information generation unit 17 refers to the facial expression relationship information and the utterance relationship information, and is real-time or time-dependent information indicating a relationship between the first participant and the second participant. Generate sex information. By evaluating both facial expressions and utterances of both participants during the conference, the communication state between the participants can be evaluated in more detail. Further, since the relationship information generation unit 17 generates the relationship information based on the facial expression information and the utterance information between the participants in real time or over time, it is necessary to evaluate the communication state between the participants in real time or over time. You can

The relationship information generation unit 17 may generate presentation information to be presented to at least one of the first participant and the second participant. The presentation information may include the degree of coincidence between the emotions of both participants, which is comprehensively evaluated based on the facial expression relationship information and the utterance relationship information (for example, the line-of-sight matching rate is high, and the utterance ratio is high). If they are equal, increase the degree of agreement of emotions).

By presenting the presentation information generated by the relationship information generation unit 17 to the participants, the relationships between the participants can be fed back to the participants. By presenting the presentation information to the participants in real time, it is possible to confirm the relationship in real time during the conversation, and thus it is possible to prompt the improvement of communication in real time.

The presentation information may be presented to both the first participant and the second participant, or may be presented to either one. Further, the relationship information may present the same content to the first participant and the second participant, or may present different content. By generating the relationship information that presents the same content to the first participant and the second participant, it can be expected to build a flat relationship between the participants. The participants may be allowed to select the presentation information, or the presentation information presented may be changed according to a rule or agreement between the participants.

The relationship information includes information indicating the ratio of the utterance time of the first participant and the utterance time of the second participant, the line-of-sight direction of the first participant, and the line-of-sight of the second participant. Information about the change over time in the matching rate with the direction may be included. In addition, the relationship information may include information regarding changes in the facial expression matching rate or the facial expression mismatch rate over time, the previous turnover rate, the text of the utterance content, the frequent words, and the like. Further, the relationship information includes the participant's ID, the participant's own facial image, the avatar image showing the facial expressions of other participants, and the display of the recommended agenda or recommended words extracted from the accumulated data based on the utterance content. It may be.

Further, the presentation information may include an evaluation result for improving the communication skills of the conference participants. As an example, along with the facial expression matching rate, information that encourages improvement of reflexive listening skills by increasing the facial expression matching rate is presented, and a posture and attitude suitable for communication by increasing the gaze matching rate together with the gaze matching rate. You may provide the information that encourages you to take it. In addition, recommended words and question contents may be presented to encourage deduction of the dialogue level and question level.

Specifically, as a method of presenting the presentation information, for example, a method of displaying the information on the display unit (each of the display unit 23 and the display unit 33) of the conference participants and a common display unit that can be visually recognized by all the conference participants. Display method, method of presenting other than conference participants by network distribution, method of presenting by physical action (vibration, electrical stimulation, etc.) from a wearable device such as a wristwatch type device, environmental equipment (certification, air conditioning) , Music, etc.) to present by a physical action (such as illuminating the room red when the discussion is heated), a method of presenting by an image image corresponding to an index indicating emotion (volcanic eruption indicating anger, etc.) , A method of presenting with an avatar's facial expression corresponding to an index indicating emotions, and the like.

An example of a screen in which presentation information is presented on at least one of the display unit 23 and the display unit 33 will be described with reference to FIGS. 4 and 5. FIG. 4 is a diagram showing an example of information presented by an information processing system including an information processing apparatus according to an embodiment of the present invention, and FIG. 5 is information including an information processing apparatus according to an embodiment of the present invention. It is a figure which shows the other example of the information which a processing system presents.

As shown in FIG. 4, in the screen 400, the user IDs of the conference participants are displayed in the area 401, the face images of the conference participants are displayed in the area 402, and the target to present the presentation information is specified. The utterance ratio is displayed as a talk ratio in the area 403, for example, a pie chart is displayed, the facial expression of the avatar corresponding to the index indicating the emotion of the participant in the dialogue is displayed in the area 404, and the facial expression matching rate over time is displayed in the area 405. By displaying changes in a graph, it is possible to instantly check the current state of communication during a meeting. Further, in the area 406, the text representing the utterance content is displayed as Talk Stream, and the conversation theme and words recommended during the conference are displayed. The facial expression matching rate displayed in the area 405 is an example of presentation information generated based on temporal information from the past to the present regarding facial expression relationship information. The facial expression of the avatar displayed in the area 404 is an example of presentation information generated based on real-time information regarding facial expression relationship information.

Further, as shown in FIG. 5, in the screen 500, similarly to the screen 400, the user ID is displayed in the region 501, the face image is displayed in the region 502, the talk ratio is displayed in the region 503, and the avatar is displayed in the region 504. The facial expression may be displayed, the utterance content, the recommended theme, etc. may be displayed in the area 505, and the line-of-sight matching rate may be displayed in the area 505 instead of the facial expression matching rate.

<Appendix 1>
A part or all of the processing in the control unit of the terminal device may be configured to be performed in the calculation unit 12 included in the information processing device 10. For example, the calculation unit 12 acquires the captured image of the camera 21 via the communication unit 11, and the facial expression information acquisition unit 13 acquires the first facial expression information regarding the facial expression of the first participant and the second participation. The second facial expression information regarding the facial expression of the person may be generated.

<Appendix 2>
Further, in the above example, the conference by two persons, the first participant and the second participant, has been described as an example, but the present embodiment is not limited to this. Of course, the invention described in the present specification can be applied to a conference with N names (N is 3 or more). In that case, the configurations described in the present specification can be individually applied to any two pairs of N persons. For example, for a conference of three persons (A, B, C), the invention described in this specification is applied individually to three sets of (A, B) (A, C) (B, C). can do.
As described above, the invention described in the present embodiment uses the data representing the state of N persons, the history data of the state of N persons, and the environmental information of N persons to participate in a part or all of the N persons. It can also be said to generate "relationship information" indicating a relationship between persons.
<Appendix 3>
Further, in the above example, the case where both the first participant and the second participant are humans has been described as an example, but this does not limit the present embodiment.
For example, the second participant may not be a human but a pseudo human represented by a computer such as a preset avatar or BOT. In the case of such a configuration, the second terminal device is not essential, and the facial expression information acquisition unit 13 and the voice information acquisition unit 14 provide the facial expression and utterance content represented by the BOT created in advance to the second participant. The facial expression information and the voice information may be acquired.
The facial expression and utterance content represented by the BOT may be data created in advance before the conference, or may be adaptively changed according to the facial expression and utterance of the first participant in the conference. Good.

[Example of software implementation]
The control block (particularly the arithmetic unit 12) of the information processing device 1 may be realized by a logic circuit (hardware) formed in an integrated circuit (IC chip) or the like, or may be realized by software.

In the latter case, the information processing apparatus 1 includes a computer that executes the instructions of a program that is software that realizes each function. The computer includes, for example, one or more processors and a computer-readable recording medium that stores the program. Then, in the computer, the processor reads the program from the recording medium and executes the program to achieve the object of the present invention. As the processor, for example, a CPU (Central Processing Unit) can be used. As the recording medium, a "non-transitory tangible medium" such as a ROM (Read Only Memory), a tape, a disk, a card, a semiconductor memory, a programmable logic circuit, or the like can be used. Further, a RAM (Random Access Memory) for expanding the program may be further provided. The program may be supplied to the computer via any transmission medium (communication network, broadcast wave, etc.) capable of transmitting the program. Note that one aspect of the present invention can also be realized in the form of a data signal embedded in a carrier wave, in which the program is embodied by electronic transmission.

The present invention is not limited to the above-described embodiments, but various modifications can be made within the scope of the claims, and embodiments obtained by appropriately combining the technical means disclosed in the different embodiments Is also included in the technical scope of the present invention.

100 Information Processing System 10 Information Processing Device 20 First Terminal Device 30 Second Terminal Device 13 Facial Expression Information Acquisition Unit 14 Voice Information Acquisition Unit 15 Facial Expression Relation Information Generation Unit 16 Utterance Relation Information Generation Unit 17 Relation Information Generation Unit

Claims (12)

A facial expression information acquisition unit that acquires first facial expression information regarding a facial expression of a first participant among the plurality of participants and second facial expression information regarding a facial expression of a second participant among the plurality of participants. ,
A voice information acquisition unit that acquires first utterance information about the utterance of the first participant and second utterance information about the utterance of a second participant of the plurality of participants.
A facial expression relationship that refers to the first facial expression information and the second facial expression information to generate facial expression relationship information that indicates a facial expression relationship between the first participant and the second participant. An information generator,
Utterance relationship for generating utterance relationship information indicating a utterance relationship between the first participant and the second participant with reference to the first utterance information and the second utterance information. An information generator,
A relationship information generation unit that generates relationship information that is information indicating a relationship between the first participant and the second participant with reference to the facial expression relationship information and the utterance relationship information,
An information processing apparatus comprising: The information processing apparatus according to claim 1, wherein the relationship information is real-time or temporal information indicating a relationship between the first participant and the second participant. The first facial expression information includes a plurality of first indexes expressing the facial expression of the first participant,
The second facial expression information includes a plurality of second indexes expressing the facial expression of the second participant,
The facial expression relationship information generating unit may generate facial expression difference information regarding a difference between the first index and the second index, and include the generated facial expression difference information in the facial expression relationship information. The information processing apparatus according to claim 1. The first facial expression information includes first visual line information regarding a visual line direction of the first participant,
The second facial expression information includes second visual line information regarding a visual line direction of the second participant,
The facial expression relationship information generating unit generates visual line relationship information by referring to the first visual line information and the second visual line information, and includes the generated visual line relationship information in the facial expression relationship information. The information processing apparatus according to any one of claims 1 to 3, characterized in that. The utterance relationship information generation unit,
Generating utterance time relationship information indicating a relationship between the utterance time of the first participant indicated by the first utterance information and the utterance time of the second participant indicated by the second utterance information, The information processing apparatus according to claim 1, wherein the generated utterance time relationship information is included in the utterance relationship information. The utterance relationship information generation unit,
It is determined whether or not at least one of the first utterance information and the second utterance information includes utterance content included in a specific category, and information corresponding to the determined result is used as the utterance relationship. The information processing device according to claim 1, wherein the information processing device is included in information. The utterance relationship information generation unit,
A feature that a word having a relatively high appearance frequency within a predetermined time is extracted from at least one of the first utterance information and the second utterance information, and the extracted word is included in the utterance relationship information. The information processing apparatus according to any one of claims 1 to 6. The relation information generating unit refers to the relation information to generate presentation information to be presented to at least one of the first participant and the second participant. The information processing apparatus according to any one of 1 to 7. The presentation information includes
Information indicating the ratio of the utterance time of the first participant to the utterance time of the second participant, and
The information processing apparatus according to claim 8, wherein the information processing apparatus includes information about a change over time in the matching rate between the line-of-sight direction of the first participant and the line-of-sight direction of the second participant. The facial expression relationship information generation unit and the utterance relationship information generation unit further refer to the participant information indicating the attributes of the first and second participants, and the facial expression relationship information and the utterance relationship information. The information processing apparatus according to claim 1, wherein the information processing apparatus generates information. A facial expression information acquisition step of acquiring first facial expression information regarding a facial expression of a first participant among the plurality of participants and second facial expression information regarding a facial expression of a second participant among the plurality of participants; ,
A voice information acquisition step of acquiring first utterance information related to the utterance of the first participant and second utterance information related to the utterance of a second participant of the plurality of participants;
A facial expression relationship that refers to the first facial expression information and the second facial expression information to generate facial expression relationship information that indicates a facial expression relationship between the first participant and the second participant. An information generation step,
Utterance relationship for generating utterance relationship information indicating a utterance relationship between the first participant and the second participant with reference to the first utterance information and the second utterance information. An information generation step,
A relationship that refers to the facial expression relationship information and the utterance relationship information to generate relationship information that is real-time or temporal information indicating a relationship between the first participant and the second participant. An information generation step,
An information processing method comprising: An information processing program for causing a computer to function as the information processing apparatus according to claim 1, wherein the facial expression information acquisition unit, the voice information acquisition unit, the facial expression relationship information generation unit, An utterance relationship information generating unit, and an information processing program for causing a computer to function as the relationship information generating unit.

Images