CN112653902B - Speaker recognition method and device and electronic equipment - Google Patents

Abstract

The embodiment of the present application discloses a speaker recognition method, device and electronic equipment. The method includes: separating and obtaining an audio file and an image file from a video file to be recognized; performing speech segmentation on the audio file, obtaining start and end time information and speaker identification information corresponding to at least one speech segment, and Face recognition is performed on the image file to obtain face recognition results corresponding to different times; the speaker identification information and the face recognition results are aligned in time to determine the speaker corresponding to the at least one speech segment . Such a solution helps to improve the accuracy of speaker recognition.

Description

Speaker recognition method, device and electronic equipment

technical field

The present application relates to the technical field of speech recognition, in particular to a speaker recognition method, device and electronic equipment, a video processing method, device and electronic equipment.

Background technique

Voiceprint recognition, also known as speaker recognition, can analyze and process voice signals through computer voice processing technology to determine the identity of the speaker.

For scenarios that may involve multiple speakers, such as live broadcasts and conferences, when performing speaker recognition, it is not only necessary to identify who is included in the voice signal, but also to determine the specific voice segments corresponding to each person. Taking the dialogue voice between user A and user B as an example, voice segmentation can be performed first to find the voice transition points between different speakers, divide the entire dialogue voice into several voice segments, and mark the corresponding voice segment of each voice segment. The start and end time and speaker id, and then perform speaker clustering, and merge the speech segments corresponding to the same speaker id.

Ideally, after processing the above dialogue speech, two categories can be obtained: one category corresponds to user A’s id1, including all speech segments spoken by user A during the dialogue; the other category corresponds to user B’s id2, including user B’s All speech segments spoken during the conversation.

However, in the actual application process, due to the influence of ambient noise, segmentation errors may occur during speech segmentation. For example, a speech segment corresponding to user A is split into two sub-speech segments, and the corresponding speaker ids are marked as id1 and id2 respectively, which leads to the sub-speech segment marked as id2 when speakers are clustered To be merged into the category corresponding to user B. Although the speaker identification was correct, the merged speech segments were incorrect.

Alternatively, the speaker ids of the two sub-segments that were split above may be identified as id1 and id3. After such identification, not only the merged speech segment is wrong, but also the speaker’s identity is incorrectly identified, and the original only Speech recognition for two user conversations as three users.

How to improve the accuracy of speaker recognition, especially in the vocal language communication activities where many people participate, such as in the scene of multi-person speech, where there is strong noise interference such as background music and applause, how to improve the accuracy of speaker recognition has become a key issue for technical personnel. Solved technical problems.

Contents of the invention

The present application provides a speaker recognition method, device and electronic equipment, which can perform speaker recognition based on voiceprint features and face features, and help to improve recognition accuracy.

This application provides the following solutions:

A speaker recognition method, comprising:

Separate and obtain audio files and image files from video files to be identified;

Carrying out voice segmentation to the audio file, obtaining start and end time information and speaker identification information corresponding to at least one voice segment, and performing face recognition on the image file to obtain face recognition results corresponding to different times;

The speaker identification information and the face recognition result are aligned in time to determine the speaker corresponding to the at least one speech segment.

A video file processing method, comprising:

The server obtains the video files generated by the audio language communication activities;

Perform speaker identification according to the audio files and image files separated from the video file to obtain a collection of speech segments associated with different speakers;

performing speech recognition on a set of speech segments associated with the speaker to obtain the speech content of the speaker;

An association relationship between the speaker and the corresponding speech content is established.

A video file processing method, comprising:

The client receives the point-of-view file sent by the server and obtained by processing the video of the vocal language communication activity. The separated audio files and image files are subjected to speaker identification, and the determined sets of speech segments associated with different speakers are extracted;

Push the highlight file to the user associated with the client.

A speaker recognition device, comprising:

The file separation unit is used to separate and obtain audio files and image files from video files to be identified;

A voice segmentation unit, configured to perform voice segmentation on the audio file, and obtain start and end time information and speaker identification information corresponding to at least one voice segment;

A face recognition unit, configured to perform face recognition on the image file to obtain face recognition results corresponding to different times;

The alignment processing unit is configured to perform temporal alignment processing on the speaker identification information and the face recognition result to determine the speaker corresponding to the at least one speech segment.

A video file processing device applied to a server, comprising:

The video file obtaining unit is used to obtain the video files produced by the vocal language communication activities;

A speaker recognition unit, configured to perform speaker recognition based on the audio files and image files separated from the video file, to obtain a collection of speech segments associated with different speakers;

A utterance content obtaining unit, configured to perform speech recognition on a set of speech segments associated with the speaker, and obtain the utterance content of the speaker;

An association relationship establishing unit, configured to establish an association relationship between the speaker and the corresponding utterance content.

A video file processing device, applied to a client, comprising:

The highlight file receiving unit is used to receive the highlight file sent by the server and obtained by processing the video of the audio language exchange activity. The highlight file is added with content highlights of different speakers, and the content highlights are Carrying out speaker identification according to the separated audio files and image files in the video, and extracting from the set of speech segments associated with different speakers determined;

A point-of-view file pushing unit, configured to push the point-of-view file to users associated with the client.

An electronic device comprising:

one or more processors; and

A memory associated with the one or more processors, the memory is used to store program instructions, and when the program instructions are read and executed by the one or more processors, perform the following operations:

Separate and obtain audio files and image files from video files to be identified;

Carrying out voice segmentation to the audio file, obtaining start and end time information and speaker identification information corresponding to at least one voice segment, and performing face recognition on the image file to obtain face recognition results corresponding to different times;

The speaker identification information and the face recognition result are aligned in time to determine the speaker corresponding to the at least one speech segment.

An electronic device comprising:

one or more processors; and

A memory associated with the one or more processors, the memory is used to store program instructions, and when the program instructions are read and executed by the one or more processors, perform the following operations:

Access to video files produced by spoken language exchange activities;

Perform speaker identification according to the audio files and image files separated from the video file to obtain a collection of speech segments associated with different speakers;

performing speech recognition on a set of speech segments associated with the speaker to obtain the speech content of the speaker;

An association relationship between the speaker and the corresponding speech content is established.

An electronic device comprising:

one or more processors; and

A memory associated with the one or more processors, the memory is used to store program instructions, and when the program instructions are read and executed by the one or more processors, perform the following operations:

Receiving the highlights file sent by the server and obtained by processing the video of the audio language communication activity, the highlights file is added with the highlights of the contents of different speakers, and the contents of the highlights are based on the separation of the video from the video. audio files and image files for speaker recognition, and extract from the set of determined speech segments associated with different speakers;

Push the highlight file to the user associated with the client.

According to the specific embodiments provided by the application, the application discloses the following technical effects:

In the embodiment of the present application, the video file can be separated from the sound image, and the separated audio file can be divided into voices to obtain at least one voice segment, and the start and end time information and speaker identification information corresponding to each voice segment; at the same time, Face recognition can also be performed on the separated image files to obtain face recognition results corresponding to different times. Then, the voiceprint features and face features are aligned to obtain speaker clustering results, that is, to determine the speaker corresponding to at least one speech segment. Among them, the alignment process can be based on the start and end time corresponding to the speech segment, determine whether the face recognition result corresponding to the start and end time can correspond to the same speaker; or, the alignment process can be based on the face recognition result, determine the adjacent speech segment Whether it can correspond to the same speaker. Such speaker identification based on the user's voiceprint features and face features helps to solve the problem of low recognition accuracy in the prior art.

Of course, implementing any product of the present application does not necessarily need to achieve all the above-mentioned advantages at the same time.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the accompanying drawings required in the embodiments. Obviously, the accompanying drawings in the following description are only some of the present application. Embodiments, for those of ordinary skill in the art, other drawings can also be obtained based on these drawings without any creative effort.

Fig. 1 is a schematic diagram of the system provided by the embodiment of the present application;

Fig. 2 is a flow chart of the first method provided by the embodiment of the present application;

Fig. 3 is a schematic diagram of the first communication text provided by the embodiment of the present application;

Fig. 4 is a schematic diagram of the second communication text provided by the embodiment of the present application;

Fig. 5 is a flow chart of the second method provided by the embodiment of the present application;

Fig. 6 is a flow chart of the third method provided by the embodiment of the present application;

Fig. 7 is a flow chart of the fourth method provided by the embodiment of the present application;

Fig. 8 is a schematic diagram of the first device provided by the embodiment of the present application;

Fig. 9 is a schematic diagram of a second device provided by an embodiment of the present application;

Fig. 10 is a schematic diagram of a third device provided by an embodiment of the present application;

Fig. 11 is a schematic diagram of the architecture of the computer system provided by the embodiment of the present application;

FIG. 12 is a schematic diagram of an architecture of an electronic device provided by an embodiment of the present application.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the application with reference to the drawings in the embodiments of the application. Apparently, the described embodiments are only some of the embodiments of the application, not all of them. All other embodiments obtained by persons of ordinary skill in the art based on the embodiments in this application belong to the protection scope of this application.

Example 1

In order to solve the problem of low speaker recognition accuracy due to the influence of environmental noise and other factors in a multi-person speaking scene, the embodiment of the present application provides a tool for speaker recognition based on voiceprint features and face features. The identification process will be explained below in combination with specific examples.

The identification objects involved in this embodiment of the present application are mainly video files including audio and images. After obtaining the video file to be recognized, the sound and image can be separated first, and the video file can be split into an audio file that can extract voiceprint features, and an image file that can extract facial features. For example, an audio file and an image file can be separated from a video file by a multimedia video processing tool FFmpeg (FastForward Mpeg).

For audio files, voice segmentation can be performed through voiceprint tracking technology to find the voice transition points between different speakers, divide the audio file into multiple voice segments, and mark the corresponding start and end time of each voice segment and Speaker identification information. For example, the time length of video file 1 is 10 minutes. After voice segmentation is performed on the audio file separated from video file 1, the relevant information of the voice segment shown in Table 1 can be obtained.

Table 1

Voice segment identification information Start and end time information (unit: s) speaker identification information speech segment 1 0～11 id1 speech segment 2 12～550 id2 Speech segment 3 551～600 id1

For image files, face recognition can be performed on the included image frames, and the speakers included in the image files can be determined through face features. As an example, image frames may be extracted from image files at preset time intervals, and identity information of speakers corresponding to each image frame may be determined through face recognition technology. For example, for the image file separated from the video file 1, the face recognition result information shown in the following Table 2 can be obtained by using frame-by-second method.

Table 2

Time information (unit: s) speaker identity information 1 user 1 2 user 1 ... ... 12 user 2 13 user 2 14 user 1 ... ... 600 user 2

Wherein, the speaker's identity information may be a number customized during the face recognition process. For example, face recognition is performed based on the image frame corresponding to the first second, and the determined user can be defined as user 1. If the face recognition process is performed on the image frame corresponding to the second to eleventh seconds, the recognition result indicates that it corresponds to the same user 1, continue to use the same number to define the user; if the face recognition is performed on the image frame corresponding to the 12th second, it is determined that the user corresponding to the face feature is a new user, then the new user can be defined for user2. By analogy, define the corresponding number for the user that appears in the image file.

Alternatively, a speaker information database may be created in advance, and the association relationship between at least one speaker's identity information (for example, the identity information may be embodied as a speaker's name, etc.) and facial feature information is stored in the information database. In this way, after face recognition is performed on the image frame and the corresponding face feature information is extracted, the information database can be queried to determine the identity information of the speaker corresponding to the face feature information.

According to the introduction above, after obtaining the recognition results in Table 1 and Table 2, the voiceprint features and face features can be aligned to obtain the speaker clustering results.

The feature alignment process is illustrated below with a specific example.

In one manner, the target face recognition result corresponding to the start and end time can be obtained according to the start and end time information corresponding to the speech segment, and it can be determined whether the target face recognition result corresponds to the same speaker.

Specifically, if the target face recognition result corresponds to a target user, that is, the voiceprint features and face features are aligned within the start and end time, then the association between the voice segment and the target user can be directly established.

Taking speech segment 1 shown in Table 1 as an example, the corresponding start and end times are 0-11s, and it can be seen from Table 2 that the speaker identity information of the face recognition corresponding to the start and end times is user 1, that is, the target face recognition Results correspond to only one target user. At this point, it can be determined that the speech segment 1 is the audio generated during the speaking process of the user 1, and an association relationship between the speech segment 1 and the user 1 can be established.

Specifically, if the target face recognition results correspond to at least two users, a target user can be determined from them, that is, the target face recognition results are merged, and at least two users are merged into one target user to achieve voice recognition. Align the fingerprint features and face features within the start and end time, and then associate the voice segment and the target user.

Taking speech segment 2 shown in Table 1 as an example, the corresponding start and end times are 12 to 550s, and it can be seen from Table 2 that the speaker identity information for face recognition corresponding to the start and end times includes user 1 and user 2, that is, the target person The face recognition results correspond to two target users. The reason for this situation may be that while a certain user is speaking, the camera switches to capture another user, that is, the speaker is usually one of the two users, so a target user can be determined from it to establish a target user Relationships with speech segments. For example, the user with the most occurrences within the start and end time periods may be determined as the target user.

In the above example, if among the image frames corresponding to 12-550s, the proportion of image frames identified as user 1 is significantly less than that of image frames identified as user 2, user 2 may be determined as the target user. That is, it can be determined that the speech segment 2 is the audio generated during the speaking process of the user 2, and an association relationship between the speech segment 2 and the user 2 can be established.

In another manner, it may be determined whether adjacent speech segments correspond to the same speaker according to the face recognition result.

Specifically, the target start and end time for alignment processing can be determined according to the respective start and end time information of adjacent speech segments; the target face recognition result corresponding to the target start and end time can be obtained; if the target face recognition result corresponds to a target user, then It can be determined that the adjacent speech segment corresponds to the same speaker, and the adjacent speech segment and the target user can be associated.

Taking speech segment 3 shown in Table 1 as an example, the corresponding start and end time is 551 to 600s. If the speaker identity information corresponding to the start and end time is determined as user 2 through Table 2, (maybe the face recognition result corresponding to the start and end time only includes User 2; or, may also include multiple users, but user 2 appears the most), that is, for adjacent speech segment 2 and speech segment 3, although they correspond to different user ids during audio recognition, However, they correspond to the same user identity during image recognition. The reason for this situation may be that it is affected by environmental noise. During voice segmentation, the voice of the same user is divided into multiple ids. For this purpose, adjacent voice segments can be mapped to the speakers identified by the image to achieve Alignment of voiceprint features and facial features within the target start and end time. That is to say, it can be determined that both the voice segment 2 and the voice segment 3 are audio generated during the speaking of the user 2, and an association relationship between the voice segment 2 and the voice segment 3 and the user 2 can be established.

Through the above alignment process, the speaker clustering results shown in Table 3 below can be obtained.

table 3

It should be noted that, for the case where the same speaker is associated with at least two speech segments, as an example, as shown in Table 3, an association between the speaker and the corresponding at least two speech segments can be established; Alternatively, at least two speech segments may be merged, and then an association between the speaker and the merged speech segment is established.

Such speaker identification based on the user's voiceprint features and face features helps to solve the problem of low recognition accuracy in the prior art.

It can be understood that the embodiment of the present application can perform speaker recognition based on complete video files. As the example above, audio files and image files of 0 to 600s can be analyzed and processed; or, it can be based on part of the video that the user is interested in Files for speaker recognition, for example, intercept the video files corresponding to the 12th to 600s, perform sound and image separation, and perform speaker recognition according to the processing process introduced above. No matter which method is used for speaker recognition, as long as the audio files and image files are aligned in time.

In addition, it should be noted that the above-mentioned speaker recognition tool can be deployed on the terminal device associated with the user to perform local analysis and processing on the video files obtained by the terminal device; The submitted video files are analyzed and processed, and several speakers are identified in the video file, and which speech segments each speaker corresponds to.

It can be understood that the speaker recognition tool provided by the embodiment of the present application can be applied to various scenarios. As an example, it can be applied to a verbal language communication activity in which multiple people participate, for example, a multi-person meeting, a multi-person speech, and other activities.

Taking a multi-person conference scenario as an example, identity information of a speaker (for example, a participant) may be identified, and a set of speech segments generated by each speaker when speaking at the conference may be determined. As an example, based on the set of speech segments corresponding to the speaker, the viewpoint of the meeting expounded by the speaker may be extracted. Taking a multi-person speech scene as an example, identity information of a speaker (for example, a speaker) may be identified, and speech segments produced by each speaker during a speech may be determined. As an example, based on the set of speech segments corresponding to the speaker, the highlights of the speaker's speech may be extracted.

Alternatively, it can be applied to the scene of a film and television drama to identify the identity information of different characters and the set of speech segments corresponding to each character. As an example, plot highlights may be extracted based on a set of speech segments corresponding to characters.

The following takes the video taken in the live lecture scene as an example to explain the processing process of the video file.

Example 2

For the live broadcast of speeches, the existing technology can only extract the highlights of different speakers’ speeches through manual analysis after the live broadcast is over, and then manually add them to the video file, resulting in a large amount of labor and time cost in the video processing process. .

Correspondingly, this embodiment provides a tool that can automatically process video files. As shown in FIG. 1 , it may include a client and a server. Among them, the client can be deployed on the terminal device associated with the user, and the server can be deployed on the cloud server, for example, on the live broadcast server, and the speaker recognition tool provided in Embodiment 1 can be used as a function of the server to identify Speaker identities in lecture videos and determine the set of speech segments associated with different speakers.

In the following, the video file processing process will be explained in combination with the flow chart shown in FIG. 2 .

S101: The server obtains the video file generated by the audio communication activity.

In this example, the video files generated by the spoken language communication activities, that is, the video files generated by the speech, can be obtained by the live broadcast server for the live speech broadcast, and submitted to the server. If the live broadcast server submits the video file generated by the speech to the server after the live broadcast of the speech, the server can perform post-processing of the live broadcast and add the corresponding content highlights to the file (in this example, the content highlights can be reflected as the speaker highlights of the speech), for users to review. If the live broadcast server submits the video file generated by the speech to the server during the live broadcast, the server can process it in real time and add the corresponding speech highlights to the video file that has been broadcast live. In this way, when watching the live broadcast, the user can You can choose to follow the live speech and watch it online synchronously; or you can choose to watch back part of the live broadcast.

For example, for a 2-hour live speech broadcast, if the live broadcast has been live for 30 minutes, the server can analyze and process the video files generated by the 30-minute live broadcast, add highlights of the speech, and allow users to review the live broadcast content of the previous 30 minutes , and can directly locate the speech clips corresponding to the highlights of the speech.

S102: Perform speaker recognition according to the audio file and the image file separated from the video file, and obtain a set of speech segments associated with different speakers.

For the specific implementation process, refer to the introduction in Embodiment 1 above, and details will not be repeated here.

If the video file 1 is the video file generated by the speech, the recognition results shown in Table 3 can be obtained after speaker recognition. Wherein, the speech segment 1 is the audio generated during the speech of the speaker 1, and the speech segment 2 and the speech segment 3 are the audio generated during the speech of the speaker 2.

As an example, before the speech begins, the speaker information database can be created in advance, and the facial feature information of different speakers can be saved through the speaker information database. In this way, when the server performs face recognition on the image file, it can The speaker's face features are used to determine the speaker's identity information from the speaker information database. Alternatively, the voiceprint characteristics of different speakers can be stored in the speaker information database. In this way, when the server tracks the voiceprint characteristics of the audio file, it can determine the speaker's identity from the information database according to the voiceprint characteristics of the speaker. Identity Information. Wherein, the speaker information database may be created by the server of the embodiment of the present invention, or may be created by other servers and then provided to the server of the embodiment of the present invention, which is not specifically limited in this embodiment of the present application.

Preferably, the noise threshold can also be set in advance, and when it is determined that the noise may affect the speech segmentation, speaker recognition is performed by performing feature alignment on voiceprint features and face features.

In one way, if the lecture venue is equipped with a noise collection device, after the server obtains the noise information collected by the noise collection device, it can determine whether the noise value indicated by the noise information exceeds a preset threshold (for example, the preset threshold can be embodied as a preset decibel value): if it exceeds the preset threshold, it can be determined that the noise may affect the accuracy of speech segmentation, and the audio file and image file can be separated from the video file according to the solution provided in embodiment 1 for speaker recognition. If the preset threshold is not exceeded, it can be determined that noise has little influence on the speech segmentation, and the audio file can be extracted from the video file, and speaker recognition can be performed only based on the audio file.

Or, in another way, the server obtains the video file, and after separating the sound and image of the video file, it can determine whether there is noise interference according to the volume change in the audio file. If the amplitude of the volume changes within a certain range, it may be caused by the speaker speaking. At this time, the audio file can be extracted from the video file, and the speaker can be identified based on the audio file. If the amplitude of the volume changes greatly, it may be due to noise interference such as applause or background music. At this time, according to the solution provided in Embodiment 1, the audio file and the image file can be separated from the video file for speaker recognition.

S103: Perform speech recognition on the set of speech segments associated with the speaker to obtain the speech content of the speaker.

S104: Establish an association between the speaker and the corresponding speech content.

Specifically, the speech segment can be converted into a recognized text through speech recognition technology to obtain the speech content of the speaker.

In the above example, speech recognition is performed on speech segment 1, and speech content 1 can be obtained (in this example, the speech content can be embodied as the speech content of the speaker), and an association relationship between speaker 1 and speech content 1 can be established ; Speech recognition is performed on the speech segment 2 and the speech segment 3, and the speech content 2 and the speech content 3 can be obtained respectively, and the association relationship between the speaker 2 and the speech content 2 and the speech content 3 is established.

Through the video processing process of the above example, the association relationship shown in Table 4 below can be obtained.

Table 4

As an example, after the speech recognition of the speech segment 2 and the speech segment 3, the speech content 2 and the speech content 3 can also be merged, and after the speech content 4 is obtained, the association between the speaker 2 and the speech content 4 can be established relation.

S105: Extract content highlights of the speaker from the utterance content associated with the speaker.

In the above example, after obtaining the lecture content related to the lecturer, the lecturer's speech point of view can be extracted from it, that is, the main point of view expressed by the lecturer during the speech. For example, the key points of a speech may be extracted through abstractive or extractive methods. The specific implementation process may refer to related technologies, and will not be described in detail here.

For the speaker 1, the key points of the speaker 1's speech may be extracted based on the speech content 1 . For the speaker 2, the highlights of the speaker 2's speech can be extracted based on the speech content 2 and the speech content 3 respectively; or, the highlights of the speaker 2's speech can be extracted based on the combined speech content 4 .

In addition, considering that the speaker's point of view may take a long time to elaborate, that is, it is more likely to extract the speaker's speech highlights from the speech segment with a longer start and end time, so the start and end time of the speech collection can be Longer partial speech segments for text recognition. For example, text recognition may be performed on the speech segment 2, and the highlights of the speaker 2's speech may be extracted according to the speech content 2.

To sum up, the speaker’s speech highlights can be automatically extracted from the speech video. Compared with the existing technology, which can only manually extract the speech highlights after the live broadcast is over, the embodiment of this application can not only improve the processing speed of video files , to reduce labor costs and time costs; in addition, during the live broadcast process, the content of the completed live broadcast can be processed in real time, allowing users to choose between watching back and watching the live broadcast online, providing users with more usage experience.

S106: Perform post-processing on the content highlight, generate a highlight file and send it to the client.

In the embodiment of the present application, the content highlights can be sent to the client in different forms, and the post-processing process will be explained in conjunction with specific examples below.

1. If the smart device associated with the client is a video playback device, the content highlights can be delivered in the form of video. For example, in a teaching scenario, a video with added content highlights can be sent to the client for users to watch and learn.

In one way, the server can add content highlights to the video file, and send the video file containing the content highlights to the client for video playback. That is, the highlight file generated by the post-processing is a video file containing content highlights.

In other words, post-processing can be performed on the basis of the original video to add content highlights. Specifically, the addition node of the content highlight may be determined according to the start and end time of the speech segment corresponding to the content highlight. For example, add the content highlight to the start time node of the corresponding speech segment, so that after the user clicks the content highlight displayed on the progress bar of the video file, he can quickly locate the playback segment corresponding to the content highlight for the user to watch .

Or, in another manner, a demo video of a simulated communication activity may be generated according to the content highlights, and the demo video including the content highlights may be sent to the client for video playback. That is, the highlights file generated by the post-processing is a demo video containing the highlights of the content.

That is to say, in order to facilitate users to quickly understand the main highlights involved in communication activities, a demo video can be generated based on the content highlights. This demo video can be used as a summary video of the original video, and the content highlights of different speakers should be as concise as possible. displayed to the user.

Specifically, the characters in the demonstration video can be determined based on the image of the speaker or other virtual images, and then the contents of the corresponding speaker can be expressed in the form of communication and dialogue between different characters.

2. If the smart device associated with the client is an audio playback device, the content highlights can be delivered in the form of audio. For example, after a client is deployed on a smart speaker, audio files with added content highlights can be sent to the client for users to listen to.

Specifically, the content highlight can be added to the audio file (for example, the addition node of the content highlight in the audio file can be determined according to the start and end time of the speech segment corresponding to the content highlight), and the audio file containing the content highlight The file is sent to the client for audio playback. That is, the point-of-view file generated by the post-processing is an audio file containing the point-of-view of the content.

As an example, after the client obtains the highlight file, it can directly play the audio. Alternatively, the content highlights added in the highlight file may be played first, for the user to select the target content highlights of interest. In this way, after the user enters the target content point of view (taking a smart speaker as an example, the user can speak out the target content point of view through voice control), the client can submit the target content point of view to the server, and the server sends the target content point of view The audio clip corresponding to the point of view is sent to the client for audio playback. Alternatively, the speaker's identity information can be played first for the user to select a target speaker of interest. In this way, the client and the server cooperate to push audio clips related to the target speaker to the user.

3. The content highlights can be delivered in text form.

Specifically, according to the display type of the client, a corresponding type of communication text may be generated from the content point of view, and the communication text may be sent to the client for display. That is, the point-of-view file generated by the post-processing is the communication text containing the point-of-view of the content.

As an example, the display type of the client can be text display, for example, to display content highlights in the form of emails, posts, etc. Correspondingly, the server can generate communication texts of the text type shown in Figure 3 and send them to the client for further processing. exhibit. Alternatively, the display type of the client may be poster display, and correspondingly, the server may generate a communication text of the poster type shown in FIG. 4 and send it to the client for display.

S107: The client receives the point-of-view file obtained by processing the video of the spoken language communication activity sent by the server, and pushes the point-of-view file to the user associated with the client.

In the embodiment of the present application, the client can push the interesting files to the user in the form of video, audio, or text. For details, please refer to the introduction above, and will not repeat them here.

S108: The client submits a service request to the server, triggering the server to provide related services for the users associated with the client.

It can be seen from the above introduction that the embodiment of the present application can implement data processing of video files. As a preferred solution, the server can also define an array structure according to requirements, which is used to store the relationship between information such as speakers, voice segments, and content highlights.

For the video file 1, the association relationship shown in Table 5 below can be obtained after video processing.

table 5

That is to say, in addition to pushing the highlight files delivered by the server to the user, the client can also request the server to provide users with various services. The following takes the speech scene as an example to illustrate the implementation process of different services provided by the server.

1. It can generate a personalized live broadcast cover for the forwarding request submitted by the user during the live broadcast.

Specifically, if the user wants to share the live broadcast while watching the live speech online, he can submit a request for forwarding the live broadcast to the server through the client associated with the user. Correspondingly, the server can obtain the speech highlight corresponding to the live broadcast content, for example, the speech highlight extracted based on the 30-minute live speech content, and send it to the client.

As an example, the client can display the speech highlights delivered by the server for the user to select the target speech highlights, and submit the target speech highlights to the server, so that the server can generate a live broadcast according to the target speech highlights cover. For example, the server can add the target speech highlights to the cover of the live broadcast; or, it can intercept a target image from the speech segment corresponding to the target speech highlights, and use the target image and the target speech highlights to generate the live cover, specifically , according to Table 3 and Table 5, the start and end time corresponding to the target speech point of view can be determined, and then the target image is intercepted from the speech segment corresponding to the start and end time.

After the server generates the cover of the live broadcast according to the highlights of the target speech, it can obtain a forwarding link containing the cover of the live broadcast. The forwarding link can be used as the user's exclusive link and sent to the client for the user to forward the live broadcast. In this way, when the user forwards the live broadcast, the user's exclusive personalized live broadcast cover can be displayed. Compared with the existing technology that can only forward the live broadcast link with a unified cover, the forwarding service provided by the embodiment of the application not only helps to improve the user experience, but also attracts more users to watch the live lecture through personalized sharing, and improves the communication guidance. efficiency.

2. When the user watches back, it can directly locate the playback segment that the user is interested in.

Specifically, whether it is to watch back after the live broadcast, or to review the content of the live broadcast during the live broadcast, the user can submit a play request to the server through the client, triggering the server to send the speech highlights to the client Display on the terminal for the user to select the target speech highlights, and then submit the target speech highlights selected by the user to the server. Correspondingly, the server can determine the target speech segment corresponding to the target speech point of view according to the association relationship shown in Table 3 and Table 5, and determine the playback segment corresponding to the target speech point of view from the video file according to the start and end time of the target speech segment , and send the playback clip to the client for video playback. That is to say, the client can only receive and play the segments that the user is interested in.

Or, in another way, if the client can display the speech highlights added in the video file to the user for viewing, for example, display the speech highlights on the progress bar of the video file, the client can obtain the target selected by the user Speech highlights, generate a playback request containing the target speech highlights, and submit it to the server. For example, after the user clicks on the speech highlight 2 displayed on the progress bar, a playback request for the speech highlight 2 may be generated. Correspondingly, after the server obtains the above playback request, it can determine that the user wants to watch the 12-550s speech segment according to the association relationship shown in Table 3 and Table 5, so it can control the client to load the 12-550s video resource and play this partial fragment. That is to say, the client can not only play the complete speech video, but also jump to play the clips that the user is interested in according to the user's operation.

For users who listen to lectures in the form of audio, after submitting a playback request through the client, the server can determine the playback clip corresponding to the target speech viewpoint from the audio file according to the target speech viewpoint determined by the user, and play the clip Send it to the client for audio playback.

3. According to the search information input by the user, the playback segment that the user is interested in can be directly located.

Specifically, the client can provide an operation option for submitting the retrieval information, and after obtaining the retrieval information through the operation option, a retrieval request can be generated and submitted to the server.

As an example, the search information submitted by the user may be identification information of a speaker, that is, the user wants to watch a certain (certain) speaker's speech clips. Correspondingly, after the server obtains the retrieval request, it can determine the target speech segment associated with the speaker according to the association relationship shown in Table 3 and Table 5, and determine the corresponding speech segment of the speaker from the video file according to the start and end time of the target speech segment. Play the segment, and send the playback segment to the client for video playback.

Or, as another example, the search information submitted by the user may be a content keyword, that is, the user wants to watch a speech segment related to certain (certain) content. Correspondingly, after the server obtains the retrieval request, it can determine the target speech highlights that match the content keywords from the speech highlights, and then determine the corresponding target speech highlights according to the associations shown in Table 3 and Table 5. The target speech segment, and according to the start and end time of the target speech segment, determine the playback segment corresponding to the target speech point from the video file, and send the playback segment to the client for video playback.

Preferably, in order to improve the accuracy of the user's playback clip retrieval based on the content keywords, the content keywords can be extracted from the speech highlights in advance, and the content keywords are sent to the client for display, so that the user can choose the ones he is interested in. Content keywords, submitted to the server.

Similarly, for users who listen to lectures in audio form, after submitting a retrieval request through the client, the server can determine the corresponding playback segment from the audio file according to the speaker's identification information or content keywords determined by the user, and Send the playback clip to the client for audio playback.

It is understandable that the target speech points of view determined in the above example may be the same point of view, or may be different points of view, which are mainly determined by the actual needs of users.

Example 3

This embodiment 3 is corresponding to embodiment 1, provides a kind of speaker identification method, referring to Fig. 5, this method may specifically include:

S201: Separate and obtain an audio file and an image file from the video file to be identified;

S202: Perform voice segmentation on the audio file, obtain start and end time information and speaker identification information corresponding to at least one voice segment, and perform face recognition on the image file to obtain face recognition results corresponding to different times;

S203: Perform temporal alignment processing on the speaker identification information and the face recognition result to determine a speaker corresponding to the at least one speech segment.

Example 4

Embodiment 4 corresponds to Embodiment 2. From the perspective of the server, a video file processing method is provided. Referring to FIG. 6, the method may specifically include:

S301: The server obtains the video file generated by the audio communication activity;

S302: Perform speaker identification according to the audio file and the image file separated from the video file, and obtain a set of speech segments associated with different speakers;

S303: Perform speech recognition on the set of speech segments associated with the speaker to obtain the speech content of the speaker;

S304: Establish an association relationship between the speaker and the corresponding utterance content.

Example 5

Embodiment 5 corresponds to Embodiment 2. From the perspective of the client, a video file processing method is provided. Referring to FIG. 7, the method may specifically include:

S401: The client receives the point-of-view file sent by the server and obtained by processing the video of the spoken language communication activity, the point-of-view file is added with point-of-view content of different speakers, and the point-of-view of the content is based on the The audio files and image files separated from the video are used for speaker identification, and the determined set of speech segments associated with different speakers is extracted;

S402: Push the highlight file to a user associated with the client.

Regarding the parts not described in detail in the foregoing embodiment 3 to embodiment 5, reference may be made to the records in the foregoing embodiments, and details are not repeated here.

Corresponding to Embodiment 1, this embodiment of the present application also provides a speaker recognition device, see Figure 8, the device can be applied to the server, including:

A file separation unit 501, configured to separate and obtain an audio file and an image file from the video file to be identified;

A voice segmentation unit 502, configured to perform voice segmentation on the audio file, and obtain start and end time information and speaker identification information corresponding to at least one voice segment;

A face recognition unit 503, configured to perform face recognition on the image file to obtain face recognition results corresponding to different times;

The alignment processing unit 504 is configured to perform temporal alignment processing on the speaker identification information and the face recognition result to determine the speaker corresponding to the at least one speech segment.

Wherein, the device also includes:

The merging processing unit is configured to perform merging processing on at least one speech segment corresponding to the speaker.

Wherein, the alignment processing unit is specifically used for:

According to the corresponding start and end time information of adjacent speech segments, determine the target start and end time for alignment processing; obtain the target face recognition result corresponding to the target start and end time; if the target face recognition result corresponds to a target user, then Perform merging processing on the adjacent speech segments, and associate the merged speech segments with the target user.

Wherein, the alignment processing unit is specifically used for:

According to the start and end time information corresponding to the voice segment, the target face recognition result corresponding to the start and end time is obtained; if the target face recognition result corresponds to at least two users, then determine a target user from them, and perform an analysis of the voice segment and the target user for association.

Wherein, the device also includes:

An information base obtaining unit, configured to obtain a speaker information base, wherein at least one face feature information associated with the identity information of the speaker is stored in the speaker information base;

The identity information determining unit is configured to determine the identity information of the target user from the speaker information database according to the facial feature information of the target user.

Corresponding to Embodiment 2, the embodiment of the present application also provides a video file processing device, see Figure 9, the device is applied to the server, including:

A video file obtaining unit 601, configured to obtain a video file generated by a vocal language communication activity;

The speaker identification unit 602 is configured to perform speaker identification based on the audio files and image files separated from the video file, and obtain a collection of speech segments associated with different speakers;

Speech content obtaining unit 603, configured to perform speech recognition on the set of speech segments associated with the speaker, and obtain the speech content of the speaker;

An association relationship establishing unit 604, configured to establish an association relationship between the speaker and the corresponding utterance content.

Wherein, the device also includes:

A content highlight extraction unit, configured to extract the speaker's content highlight from the speech content associated with the speaker;

The content highlight processing unit is configured to post-process the content highlight, generate a highlight file and send it to the client.

Wherein, the content highlight processing unit is specifically used for:

If the smart device associated with the client is a video playback device, adding the content highlights to the video file, and sending the video file containing the content highlights to the client for video playback.

Wherein, the content highlight processing unit is specifically used for:

If the smart device associated with the client is a video playback device, a demonstration video of a simulated communication activity is generated according to the content highlights, and the demonstration video including the content highlights is sent to the client for video playback.

Wherein, the content highlight processing unit is specifically used for:

If the smart device associated with the client is an audio playback device, adding the content highlights to the audio file, and sending the audio file containing the content highlights to the client for audio playback.

Wherein, the content highlight processing unit is specifically used for:

According to the display type of the client, generate a corresponding type of communication text from the content highlights, and send the communication text to the client for display.

Among them, if the communication activity venue is equipped with noise collection equipment, the device also includes:

a noise information obtaining unit, configured to obtain the noise information collected by the noise collection device;

The speaker identification unit is configured to separate the audio file and the image file from the video file to perform speaker identification when the noise information indicates that a noise value exceeds a preset threshold.

Wherein, the device also includes:

An audio file extracting unit, configured to extract an audio file from the video file when the noise information indicates that the noise value does not exceed the preset threshold, and perform speaker recognition based on the audio file.

Wherein, the video file is a video file obtained during the live broadcast of the exchange activity and corresponding to the content of the live broadcast.

Wherein, the device also includes:

The content highlight sending unit is configured to send the content highlight corresponding to the completed live broadcast content to the client when obtaining the live broadcast forwarding request submitted by the client, so that the user associated with the client can select a target from it content point of view;

A live cover generating unit, configured to receive the target content highlights submitted by the client, and generate a live cover according to the target content highlights;

A forwarding link sending unit, configured to obtain a forwarding link including the live cover, and send it to the client for live forwarding.

Wherein, the association establishing unit is further configured to establish an association among the speaker, the set of speech segments, and the content highlights.

Wherein, the device also includes:

A content point-of-view sending unit, configured to send the content point-of-view to the client when obtaining a playback request submitted by the client, for users associated with the client to select target content point-of-view;

Play segment determination unit, used to receive the target content point of view submitted by the client; determine the target voice segment corresponding to the target content point of view, and according to the start and end time of the target voice segment, from the video file Determine the playback segment corresponding to the target content point of view;

A playback segment delivery unit, configured to deliver the playback segment to the client for playback.

Wherein, the device also includes:

A retrieval request obtaining unit, configured to obtain a retrieval request submitted by a client, where the retrieval request includes speaker identification information;

A playback segment determination unit, configured to determine the target speech segment associated with the speaker, and determine the playback segment corresponding to the speaker from the video file according to the start and end time of the target speech segment;

A playback segment delivery unit, configured to deliver the playback segment to the client for playback.

Wherein, the device also includes:

A retrieval request obtaining unit, configured to obtain a retrieval request submitted by a client, wherein the retrieval request includes content keywords;

The playback segment determination unit is used to determine the target content viewpoint matching the content keyword from the content viewpoint; obtain the target speech segment corresponding to the target content viewpoint, and according to the target speech segment The start and end times of the video file are used to determine the playback segment corresponding to the target content point of view;

A playback segment delivery unit, configured to deliver the playback segment to the client for playback.

Corresponding to Embodiment 2, this embodiment of the present application also provides a video file processing device, see Figure 10, the device is applied to the client, including:

The point-of-view file receiving unit 701 is used to receive the point-of-view file sent by the server and obtained by processing the video of the spoken language communication activity, the point-of-view file is added with points of content of different speakers, and the point of view of the content is For performing speaker identification on the audio files and image files separated from the video, extracting from a set of speech segments associated with different speakers determined;

The point-of-view file pushing unit 702 is configured to push the point-of-view file to users associated with the client.

In addition, the embodiment of the present application also provides an electronic device, including:

one or more processors; and

A memory associated with the one or more processors, the memory is used to store program instructions, and when the program instructions are read and executed by the one or more processors, perform the following operations:

Separate and obtain audio files and image files from video files to be identified;

Carrying out voice segmentation to the audio file, obtaining start and end time information and speaker identification information corresponding to at least one voice segment, and performing face recognition on the image file to obtain face recognition results corresponding to different times;

The speaker identification information and the face recognition result are aligned in time to determine the speaker corresponding to the at least one speech segment.

and an electronic device comprising:

one or more processors; and

A memory associated with the one or more processors, the memory is used to store program instructions, and when the program instructions are read and executed by the one or more processors, perform the following operations:

Access to video files produced by spoken language exchange activities;

Perform speaker identification according to the audio files and image files separated from the video file to obtain a collection of speech segments associated with different speakers;

performing speech recognition on a set of speech segments associated with the speaker to obtain the speech content of the speaker;

An association relationship between the speaker and the corresponding speech content is established.

and an electronic device comprising:

one or more processors; and

A memory associated with the one or more processors, the memory is used to store program instructions, and when the program instructions are read and executed by the one or more processors, perform the following operations:

Receiving the highlights file sent by the server and obtained by processing the video of the audio language communication activity, the highlights file is added with the highlights of the contents of different speakers, and the contents of the highlights are based on the separation of the video from the video. audio files and image files for speaker recognition, and extract from the set of determined speech segments associated with different speakers;

Push the highlight file to the user associated with the client.

Wherein, FIG. 11 exemplarily shows the architecture of a computer system, which may specifically include a processor 810 , a video display adapter 811 , a disk drive 812 , an input/output interface 813 , a network interface 814 , and a memory 820 . The processor 810 , video display adapter 811 , disk drive 812 , input/output interface 813 , network interface 814 , and the memory 820 can be connected by communication bus 830 .

Wherein, the processor 810 may be implemented by a general-purpose CPU (Central Processing Unit, central processing unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, etc., for Relevant programs are executed to realize the technical solutions provided by this application.

The memory 820 can be implemented in the form of ROM (Read Only Memory, read-only memory), RAM (Random Access Memory, random access memory), static storage device, dynamic storage device, and the like. The memory 820 may store an operating system 821 for controlling the operation of the computer system 800 and a basic input output system (BIOS) for controlling low-level operations of the computer system 800 . In addition, a web browser 823, a data storage management system 824, and a video file processing system 825 can also be stored. The above-mentioned video file processing system 825 can be the server in the embodiment of the present application that specifically implements the operations of the aforementioned steps. In a word, when implementing the technical solution provided by the present application through software or firmware, related program codes are stored in the memory 820 and invoked by the processor 810 for execution.

The input/output interface 813 is used to connect the input/output module to realize information input and output. The input/output/module can be configured in the device as a component (not shown in the figure), or can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, mouse, touch screen, microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, and the like.

The network interface 814 is used to connect the communication module (not shown in the figure), so as to realize the communication interaction between the device and other devices. The communication module can realize communication through wired means (such as USB, network cable, etc.), and can also realize communication through wireless means (such as mobile network, WIFI, Bluetooth, etc.).

Bus 830 includes a path for carrying information between the various components of the device (eg, processor 810, video display adapter 811, disk drive 812, input/output interface 813, network interface 814, and memory 820).

It should be noted that although the above-mentioned devices only show the processor 810, the video display adapter 811, the disk drive 812, the input/output interface 813, the network interface 814, the memory 820, the bus 830, etc., in the specific implementation process, the A device may also include other components necessary for proper operation. In addition, those skilled in the art can understand that the above-mentioned device may only include components necessary to realize the solution of the present application, and does not necessarily include all the components shown in the figure.

Wherein, FIG. 12 exemplarily shows the architecture of an electronic device, for example, the device 900 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant , aircraft, etc.

12, device 900 may include one or more of the following components: processing component 902, memory 904, power supply component 906, multimedia component 908, audio component 910, input/output (I/O) interface 912, sensor component 914, and communication component 916 .

The processing component 902 generally controls the overall operations of the device 900, such as those associated with display, telephone calls, data communications, camera operations, and recording operations. The processing element 902 may include one or more processors 920 to execute instructions to complete all or part of the steps of the method provided by the technical solution of the present disclosure. Additionally, processing component 902 may include one or more modules that facilitate interaction between processing component 902 and other components. For example, the processing component 902 may include a multimedia module to facilitate interaction between the multimedia component 908 and the processing component 902 .

The memory 904 is configured to store various types of data to support operations at the device 900 . Examples of such data include instructions for any application or method operating on device 900, contact data, phonebook data, messages, pictures, videos, and the like. The memory 904 can be implemented by any type of volatile or non-volatile memory device or their combination, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable Programmable Read Only Memory (EPROM), Programmable Read Only Memory (PROM), Read Only Memory (ROM), Magnetic Memory, Flash Memory, Magnetic or Optical Disk.

The power supply component 906 provides power to various components of the device 900 . Power components 906 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for device 900 .

The multimedia component 908 includes a screen that provides an output interface between the device 900 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensor may not only sense a boundary of a touch or a swipe action, but also detect duration and pressure associated with the touch or swipe operation. In some embodiments, the multimedia component 908 includes a front camera and/or a rear camera. When the device 900 is in an operation mode, such as a shooting mode or a video mode, the front camera and/or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have focal length and optical zoom capability.

The audio component 910 is configured to output and/or input audio signals. For example, the audio component 910 includes a microphone (MIC) configured to receive external audio signals when the device 900 is in operation modes, such as call mode, recording mode and voice recognition mode. Received audio signals may be further stored in memory 904 or sent via communication component 916 . In some embodiments, the audio component 910 also includes a speaker for outputting audio signals.

The I/O interface 912 provides an interface between the processing component 902 and a peripheral interface module. The peripheral interface module may be a keyboard, a click wheel, a button, and the like. These buttons may include, but are not limited to: a home button, volume buttons, start button, and lock button.

Sensor assembly 914 includes one or more sensors for providing status assessments of various aspects of device 900 . For example, the sensor component 914 can detect the open/closed state of the device 900, the relative positioning of components, such as the display and keypad of the device 900, and the sensor component 914 can also detect a change in the position of the device 900 or a component of the device 900 , the presence or absence of user contact with the device 900 , the orientation or acceleration/deceleration of the device 900 and the temperature change of the device 900 . Sensor assembly 914 may include a proximity sensor configured to detect the presence of nearby objects in the absence of any physical contact. Sensor assembly 914 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor component 914 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor or a temperature sensor.

Communication component 916 is configured to facilitate wired or wireless communications between device 900 and other devices. The device 900 can access wireless networks based on communication standards, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 916 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 916 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BT) technology and other technologies.

In an exemplary embodiment, device 900 may be programmed by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable A gate array (FPGA), controller, microcontroller, microprocessor or other electronic component implementation for performing the methods described above.

In an exemplary embodiment, there is also provided a non-transitory computer-readable storage medium including instructions, such as a memory 904 including instructions, the above instructions can be executed by the processor 920 of the device 900 to complete the method provided by the technical solution of the present disclosure . For example, the non-transitory computer readable storage medium may be ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like.

It can be known from the above description of the implementation manners that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general-purpose hardware platform. Based on this understanding, the essence of the technical solution of this application or the part that contributes to the prior art can be embodied in the form of software products, and the computer software products can be stored in storage media, such as ROM/RAM, disk , CD, etc., including several instructions to make a computer device (which may be a personal computer, server, or network device, etc.) execute the methods described in various embodiments or some parts of the embodiments of the present application.

Each embodiment in this specification is described in a progressive manner, the same and similar parts of each embodiment can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and for related parts, please refer to the part of the description of the method embodiment. The systems and system embodiments described above are only illustrative, and the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is It can be located in one place, or it can be distributed to multiple network elements. Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. It can be understood and implemented by those skilled in the art without creative effort.

Above, the speaker recognition method, device and electronic equipment provided by this application, the video file processing method, device and electronic equipment have been introduced in detail. In this paper, specific examples have been used to illustrate the principle and implementation of the application. Above The description of the embodiments is only used to help understand the method of the present application and its core idea; at the same time, for those of ordinary skill in the art, according to the idea of the present application, there will be changes in the specific implementation and application scope. To sum up, the contents of this specification should not be understood as limiting the application.

Claims (27)

1. A speaker recognition method, characterized in that, comprising: Separate and obtain audio files and image files from video files to be identified; performing voice segmentation on the audio file, obtaining start and end time information and speaker identification information corresponding to at least one voice segment, and performing face recognition on the image file to obtain face recognition corresponding to image frames at different time points Result; wherein, at least one speech segment obtained by the speech segmentation includes: under the influence of the noise information contained in the audio file, the start-stop time splitting error caused, and/or the speech segment whose speaker identification information is wrong; The start and end time information of the speech segment obtained by the speech segmentation, the speaker identification information and the face recognition result are aligned in time, and the speaker corresponding to the at least one speech segment is determined, so that according to the person The face recognition result corrects the start and end time information and speaker identification information of the speech segment obtained from the speech segmentation. 2. The method according to claim 1, further comprising: Merging processing is performed on at least one speech segment corresponding to the speaker. 3. The method of claim 1, wherein, The time-aligned processing of the speaker identification information and the face recognition results includes: According to the corresponding start and end time information of adjacent speech segments, determine the target start and end time for alignment processing; Obtain the target face recognition result corresponding to the target start and end time; If the target face recognition result corresponds to a target user, the adjacent speech segments are merged, and the merged speech segment is associated with the target user. 4. The method of claim 1, wherein, The time-aligned processing of the speaker identification information and the face recognition results includes: According to the start and end time information corresponding to the speech segment, obtain the target face recognition result corresponding to the start and end time; If the target face recognition result corresponds to at least two users, then determine a target user, and associate the speech segment with the target user. 5. The method of claim 4, wherein, Determining a target user from said at least two users, comprising: The user with the largest number of appearances within the start and end time period is determined as the target user. 6. The method according to any one of claims 3 to 5, further comprising: Obtaining a speaker information base, which stores facial feature information associated with at least one speaker's identity information; The identity information of the target user is determined from the speaker information database according to the facial feature information of the target user. 7. A video file processing method, characterized in that, comprising: The server obtains the video files generated by the audio language communication activities; According to the audio file and the image file separated from the video file, perform speaker identification, and obtain a collection of speech segments associated with different speakers; wherein, the collection of speech segments is obtained in the following manner: for the audio Carrying out speech segmentation on the file to obtain at least one speech segment corresponding to start and end time information and speaker identification information, wherein the at least one speech segment obtained by the speech segmentation includes: under the influence of noise information contained in the audio file, The resulting start and end time split errors, and/or speech segments with incorrect speaker identification information; face recognition is performed on the image file, and face recognition results corresponding to image frames at different time points are obtained, and according to the person The face recognition result corrects the start and end time information and speaker identification information of the speech segment obtained by the speech segmentation; performing speech recognition on a set of speech segments associated with the speaker to obtain the speech content of the speaker; An association relationship between the speaker and the corresponding speech content is established. 8. The method according to claim 7, further comprising: Extracting the content highlights of the speaker from the speech content associated with the speaker; Perform post-processing on the content highlight, generate a highlight file and send it to the client. 9. The method of claim 8, wherein Perform post-processing on the highlights of the content, generate highlight files and send them to the client, including: If the smart device associated with the client is a video playback device, adding the content highlights to the video file, and sending the video file containing the content highlights to the client for video playback. 10. The method of claim 8, wherein, Perform post-processing on the highlights of the content, generate highlight files and send them to the client, including: If the smart device associated with the client is a video playback device, a demonstration video of a simulated communication activity is generated according to the content highlights, and the demonstration video including the content highlights is sent to the client for video playback. 11. The method of claim 8, wherein, Perform post-processing on the highlights of the content, generate highlight files and send them to the client, including: If the smart device associated with the client is an audio playback device, adding the content highlights to the audio file, and sending the audio file containing the content highlights to the client for audio playback. 12. The method of claim 8, wherein, Perform post-processing on the highlights of the content, generate highlight files and send them to the client, including: According to the display type of the client, generate a corresponding type of communication text from the content highlights, and send the communication text to the client for display. 13. The method according to claim 7, characterized in that, if the communication activity venue is equipped with noise collection equipment, the method further comprises: obtaining noise information collected by the noise collection device; If the noise information indicates that the noise value exceeds a preset threshold, then separate the audio file and the image file from the video file. 14. The method of claim 13, further comprising: If the noise information indicates that the noise value does not exceed the preset threshold, an audio file is extracted from the video file, and speaker recognition is performed based on the audio file. 15. The method of claim 7, wherein, The video file is obtained during the live broadcast of the exchange event and corresponds to the content of the live broadcast. 16. The method of claim 15, further comprising: If the live broadcast forwarding request submitted by the client is obtained, the content highlights corresponding to the completed live broadcast content are sent to the client, for users associated with the client to select target content highlights; receiving the target content highlights submitted by the client, and generating a live broadcast cover according to the target content highlights; A forwarding link including the cover of the live broadcast is obtained, and delivered to the client for forwarding the live broadcast. 17. The method of claim 8, further comprising: An association between the speaker, the set of speech segments, and the content point of view is established. 18. The method of claim 17, further comprising: If the playback request submitted by the client is obtained, the content highlights are sent to the client, for users associated with the client to select target content highlights; receiving the target content highlights submitted by the client; Determine the target speech segment corresponding to the target content point of view, and determine the corresponding playback segment of the target content point of view according to the start and end time of the target speech segment; Send the playback segment to the client for playback. 19. The method of claim 17, further comprising: Obtain a retrieval request submitted by the client, where the retrieval request includes speaker identification information; Determine the target speech segment associated with the speaker, and determine the playback segment corresponding to the speaker according to the start and end time of the target speech segment; Send the playback segment to the client for playback. 20. The method of claim 17, further comprising: Obtain a retrieval request submitted by the client, where the retrieval request includes content keywords; From the content viewpoints, determine target content viewpoints that match the content keywords; Obtain the target speech segment corresponding to the target content point of view, and determine the playback segment corresponding to the target content point of view according to the start and end time of the target speech segment; Send the playback segment to the client for playback. 21. A video file processing method, characterized in that, comprising: The client receives the point-of-view file sent by the server and obtained by processing the video of the vocal language communication activity. Perform speaker recognition on the separated audio files and image files, and extract from a set of determined speech segments associated with different speakers; wherein, when performing speaker recognition, perform speech segmentation on the audio file to obtain at least one speech segment Respectively corresponding start and end time information and speaker identification information, and performing face recognition on the image file to obtain face recognition results corresponding to image frames at different time points; wherein, at least one speech segment obtained by the speech segmentation Including: under the influence of the noise information contained in the audio file, the start and end time splitting error caused, and/or the speech segment with wrong speaker identification information; the start and end time information of the speech segment obtained by the speech segmentation, The speaker identification information and the face recognition result are aligned in time to determine the speaker corresponding to the at least one speech segment, so as to start and stop the speech segment obtained by the speech segmentation according to the face recognition result Time information and speaker identification information are corrected; Push the highlight file to the user associated with the client. 22. A speaker recognition device, comprising: The file separation unit is used to separate and obtain audio files and image files from video files to be identified; The voice segmentation unit is configured to perform voice segmentation on the audio file, and obtain the corresponding start and end time information and speaker identification information of at least one voice segment; wherein, the at least one voice segment obtained by the voice segmentation includes: Under the influence of the noise information contained in the file, the start and end time splitting error caused by the file, and/or the speech segment with wrong speaker identification information; A face recognition unit, configured to perform face recognition on the image file, and obtain face recognition results corresponding to image frames at different time points; An alignment processing unit, configured to perform temporal alignment processing on the start and end time information, speaker identification information, and the face recognition result of the speech segment obtained by the speech segmentation, and determine the speaker corresponding to the at least one speech segment , so as to correct the start and end time information and speaker identification information of the speech segment obtained from the speech segmentation according to the face recognition result. 23. A video file processing device, characterized in that it is applied to a server, comprising: The video file obtaining unit is used to obtain the video files produced by the vocal language communication activities; The speaker identification unit is used to perform speaker identification based on the audio files and image files separated from the video file, and obtain a collection of speech segments associated with different speakers; wherein, the collection of speech segments is passed through the following Obtaining method: perform speech segmentation on the audio file, and obtain start and end time information and speaker identification information corresponding to at least one speech segment, wherein the at least one speech segment obtained by the speech segmentation includes: the audio file contains Under the influence of the noise information, the start and end time splitting error caused, and/or the speech segment with wrong speaker identification information; the face recognition is performed on the image file, and the face recognition corresponding to the image frames at different time points is obtained As a result, the start and end time information and the speaker identification information of the speech segment obtained by the speech segmentation are corrected according to the face recognition result; A utterance content obtaining unit, configured to perform speech recognition on a set of speech segments associated with the speaker, and obtain the utterance content of the speaker; An association relationship establishing unit, configured to establish an association relationship between the speaker and the corresponding utterance content. 24. A video file processing device, characterized in that it is applied to a client, comprising: The point file receiving unit is used to receive the point file sent by the server to process the video of the audio language communication activity and obtain the point file, the point file is added with the point point of content of different speakers, and the point point of the content is Perform speaker identification on the audio files and image files separated from the video, and extract from the set of determined speech segments associated with different speakers; wherein, when performing speaker identification, perform voice segmentation on the audio file Obtaining start and end time information and speaker identification information corresponding to at least one speech segment, and performing face recognition on the image file, and obtaining face recognition results corresponding to image frames at different time points; wherein, the speech segmentation The obtained at least one speech segment includes: under the influence of the noise information contained in the audio file, the start-stop time split error caused, and/or the speech segment whose speaker identification information is wrong; the speech segment obtained by the speech segmentation Segment start and end time information, speaker identification information, and the face recognition result are aligned in time to determine the speaker corresponding to the at least one speech segment, so as to segment the speech according to the face recognition result Correct the start and end time information and speaker identification information of the obtained speech segment; A point-of-view file pushing unit, configured to push the point-of-view file to users associated with the client. 25. An electronic device, characterized in that it comprises: one or more processors; and A memory associated with the one or more processors, the memory is used to store program instructions, and when the program instructions are read and executed by the one or more processors, perform the following operations: Separate and obtain audio files and image files from video files to be identified; performing voice segmentation on the audio file, obtaining start and end time information and speaker identification information corresponding to at least one voice segment, and performing face recognition on the image file to obtain face recognition corresponding to image frames at different time points Result; wherein, at least one speech segment obtained by the speech segmentation includes: under the influence of the noise information contained in the audio file, the start-stop time splitting error caused, and/or the speech segment whose speaker identification information is wrong; The start and end time information of the speech segment obtained by the speech segmentation, the speaker identification information and the face recognition result are aligned in time, and the speaker corresponding to the at least one speech segment is determined, so that according to the person The face recognition result corrects the start and end time information and speaker identification information of the speech segment obtained from the speech segmentation. 26. An electronic device, characterized in that it comprises: one or more processors; and A memory associated with the one or more processors, the memory is used to store program instructions, and when the program instructions are read and executed by the one or more processors, perform the following operations: Access to video files produced by spoken language exchange activities; According to the audio file and the image file separated from the video file, perform speaker identification, and obtain a collection of speech segments associated with different speakers; wherein, the collection of speech segments is obtained in the following manner: for the audio Carrying out speech segmentation on the file to obtain at least one speech segment corresponding to start and end time information and speaker identification information, wherein the at least one speech segment obtained by the speech segmentation includes: under the influence of noise information contained in the audio file, The resulting start and end time split errors, and/or speech segments with incorrect speaker identification information; face recognition is performed on the image file, and face recognition results corresponding to image frames at different time points are obtained, and according to the person The face recognition result corrects the start and end time information and speaker identification information of the speech segment obtained by the speech segmentation; performing speech recognition on a set of speech segments associated with the speaker to obtain the speech content of the speaker; An association relationship between the speaker and the corresponding speech content is established. 27. An electronic device, comprising: one or more processors; and A memory associated with the one or more processors, the memory is used to store program instructions, and when the program instructions are read and executed by the one or more processors, perform the following operations: Receiving the highlights file sent by the server and obtained by processing the video of the audio language communication activity, the highlights file is added with the highlights of the contents of different speakers, and the contents of the highlights are based on the separation of the video from the video. The audio files and image files are identified by speakers, and the set of determined speech segments associated with different speakers is extracted; wherein, when the speaker is identified, the audio files are segmented to obtain at least one speech segment corresponding to each The start and end time information and speaker identification information of the image file, and face recognition is performed on the image file to obtain face recognition results corresponding to image frames at different time points; wherein, at least one speech segment obtained by the speech segmentation includes: Under the influence of the noise information contained in the audio file, the start and end time splitting error caused, and/or the speech segment with wrong speaker identification information; the start and end time information of the speech segment obtained by the speech segmentation, the speaker The identification information and the face recognition result are aligned in time, and the speaker corresponding to the at least one speech segment is determined, so as to start and end time information of the speech segment obtained by the speech segmentation according to the face recognition result , The speaker identification information is corrected; Push the highlight file to the user associated with the client.

Images