CN116129931B - An audio-visual combined speech separation model building method and speech separation method - Google Patents

Abstract

The invention provides an audio-visual combined voice separation model building method and a voice separation method, which belong to the technical field of voice separation. The model building method is as follows: acquiring the video and corresponding audio original data of several speakers, and pre-processing the acquired original data. Process and obtain speech spectrograms, face frames and mouth movement frames to construct data sets; build audio separation modules based on U‑Net networks, build face modules based on ResNet‑18 networks, and build mouth movements based on ShuffleNet‑V2 and TCN networks Module, the three are combined to form a new network model, and the model is trained to select the model with the highest accuracy; after the model is built, it is used for mixing audio separation. Compared with the method using a single video stream, the audio-visual combined speech separation model proposed by the present invention has achieved obvious performance improvement. Comparative experiments on public datasets verify the effectiveness of the method.

Description

An audio-visual combined speech separation model building method and speech separation method

technical field

The invention belongs to the technical field of speech separation, and in particular relates to a speech separation model building method and a speech separation method combining audio-visual.

Background technique

In an environment where multiple sound sources sound at the same time, humans can process the received sound signals with their own sensitive auditory system, focus on the target sound, and ignore other uninteresting sounds at the same time. Cherry defined it as "the cocktail party effect" in his book. Since then, a lot of attention has been paid to the problem of speech separation. The problem of speech separation is one of the key tasks to solve the cocktail party effect, which refers to extracting a single person's voice signal from multiple overlapping voice signals. With the development of intelligence, voice separation technology also plays a role in many voice interaction devices, such as hearing aids to help the hearing-impaired hear the sound of the outside world, voice control in smart homes to provide convenience for people, mobile phone voice assistant assistance Operation, assisting in analyzing voice clues of case intelligence, improving call efficiency and quality in online meetings, etc. However, the performance of the current speech separation technology is far behind the human auditory system, and how to efficiently achieve a speech separation effect close to that of humans is still a technical problem.

Early speech separation methods widely used include spectral subtraction, computer auditory scene analysis, hidden Markov model, etc. These methods are shallow models that cannot fully extract signal features, and their effects are often based on prior knowledge or specific microphone configurations. On the other hand, it lacks the ability to learn from large amounts of data. In recent years, with the development of deep learning technology, many well-performed speech separation models have been proposed.

In fact, the human ability to focus on a specific sound depends not only on the sound, but also on visual information such as the speaker's gender, age, opening and closing of lips, and so on. The ability of such non-speech information to enhance human ability to focus on target speech in complex environments has been demonstrated in psychological research. In recent years, more and more models that combine visual information to assist speech separation have been proposed. In 2018, Google proposed a deep learning-based joint audio-visual speech separation model, which significantly improved its separation performance compared to pure audio methods. Some technicians proposed a time-domain audio-visual speech separation architecture. The visual information is pre-trained through the lip embedding extractor, and word-level and phoneme lip shape embeddings are used to assist separation. The network directly predicts the target speech waveform. However, the above method only uses a single visual information, how to effectively extract and utilize audio and video features to make it more robust in the face of more complex scenes is still worth exploring.

Contents of the invention

In view of the above problems, the first aspect of the present invention provides an audio-visual combination speech separation model building method, comprising the following steps:

Step 1, obtaining raw data of video and corresponding audio of several speakers, said raw data being shot or downloaded and obtained in different scenes;

Step 2. Preprocess the raw data obtained in step 1; process the video into images frame by frame, and randomly select two speaker data from the raw data, mix the audio in it, and analyze the mixed voice Do the short-time Fourier transform to get the spectrogram of the speech, combine the face frames and mouth movement frames corresponding to the two speaker data to construct a data set, and divide it into a training set, a verification set and a test set;

Step 3, based on the U-Net network structure, establish a residual connection in the partial downsampling convolution block of the traditional U-Net to obtain a residual convolution block, and then between the convolution and ReLU activation functions of the compression path and the expansion path Add the BN layer to build an audio separation module; based on the ResNet-18 network structure, add the CBAM attention mechanism before and after the basic convolution block of ResNet-18 to build a face module; based on the ShuffleNet-V2 and TCN network structure, combined with 3D The convolutional layer is constructed as a mouth movement module; the above three network modules are combined to construct an AV-ResUnet network model; wherein the spectrogram of the mixed voice is input into the audio separation module, and the face frame is input into the face In the module, the mouth movement frame is input into the mouth movement module;

Step 4, use the training set and verification set described in step 2 to train and verify the AV-ResUnet network model built in step 3; select the model with the best verification effect in the training process as the final test model;

Step 5, use the data in the test set to test the final selected AV-ResUnet network model.

Preferably, the specific process of pre-processing in the step 2 is: first the video is processed into a frame-by-frame image, and a frame is selected as a face frame; each frame image uses the SFD face detector to obtain facial key points, and removes the Position-related differences, positioning the position of the lips and cutting them to a fixed size, and then gray-scaled them as mouth action frames, the number of frames is 64; then randomly select the data of two speakers from the original data, and convert the After the audio is mixed, short-time Fourier transform is performed on the mixed voice to obtain the spectrogram of the voice, and the data set is constructed by combining the facial frames and mouth movement frames corresponding to the two speaker data.

Preferably, the audio separation module is improved based on the U-Net network, including conv layer, res_conv layer, audio-visual feature fusion, up_conv layer and Tanh function;

The conv layer consists of a convolution kernel with a size of 4×4 and a step size of 2, a BN layer, and a ReLU activation function. There are two conv layers in the compression path and expansion path of the network, respectively unet_conv and unetup_conv;

The res_conv layer has 6 layers in total, namely res_conv1, res_conv2, res_conv3, res_conv4, res_conv5, res_conv6, each layer consists of two convolution kernels with a size of 3×3 and a step size of 1, two BN layers, and two ReLU activations function, a Maxpool layer and a residual connection;

Among them, the number of data input and output channels of res_conv1 and res_conv2 is different, and the number of channels of input and output of the remaining four layers is the same. The residual connection is divided into two types according to the number of channels of input data and output data. When the number of channels is the same, directly input After adding the output of the convolution, pooling is performed. When the number of channels is different, the input is processed by a convolution kernel and then added;

The audio-visual feature fusion is the process of merging the audio features obtained after the compression path processing with the visual features extracted by the visual network in the time dimension to obtain audio-visual fusion features;

The up_conv layer consists of an Upsample layer, a convolution kernel with a size of 3×3 and a step size of 1, a BN layer, and a ReLU activation function, and Upsample replaces Maxpool in the compression path;

The Tanh function compresses the data to the range of -1 to 1 and outputs the mask after separation, multiplies the mask after separation and the spectrogram of the mixed voice to obtain a separate speaker's speech spectrogram, and then passes the inverse short-time The Fourier transform restores the speaker's clean voice.

Preferably, the face module is improved based on the ResNet-18 network, including a conv7 layer, a CBAM layer, a res layer, a pooling layer and a linear layer;

The conv7 layer is composed of a convolution kernel with a size of 7×7 and a step size of 2, a BN layer and a ReLU activation function, and the output of the conv7 layer is used as the input of the CBAM layer;

The CBAM layer is composed of Channel Attention and Spatial Attention. The CBAM layer is located before the first res layer and after the last res layer, and is used to efficiently extract key areas of the face that are highly correlated with audio, ignoring the differences between the faces. outside the secondary area;

The res layer includes four layers res1, res2, res3, and res4, each containing two convolutional blocks, wherein each convolutional block in res1 consists of a 3×3 convolutional kernel, a BN layer, and a ReLU activation function. The convolution block can be expressed by the following formula:

y = ReLU(x + BN(conv3(ReLU(BN(conv3(x))))))

Among them, x represents the input of the convolution block, and y represents the output of the convolution block; conv3 is a 3×3 convolution operation, BN refers to the batch normalization layer; ReLU refers to the ReLU activation function;

The first convolution block in res2, res3 and res4 is the same as in res1, the second convolution block consists of a 3×3 convolution kernel, BN layer, downsampling layer and ReLU activation function, and the second convolution block It can be expressed by the following formula:

y = ReLU ( Downsample(x) + BN(conv3(ReLU (BN(conv3(x))))))

Among them, Downsample refers to the downsampling layer;

The pooling layer includes maximum pooling and average pooling, and the maximum pooling is located after the first CBAM layer to reduce the amount of parameters and simplify the complexity of the network; the average pooling is located after the second CBAM, and the average pooling The output of is used as the input of the final linear layer;

The output of the linear layer is used as the final facial feature extracted by the network, which is copied in the time dimension and combined with the lip feature to become the visual feature required by the model.

Preferably, the mouth movement module is constructed based on ShuffleNet-V2 and TCN network structure combined with a 3D convolutional layer, and the 3D convolutional layer is composed of volumes with a size of 5×7×7 and a step size of 1×2×2 Consists of product kernel, BN layer, ReLU activation function and 3D maximum pooling layer with a size of 1×3×3 and a step size of 1×2×2;

The ShuffleNet-V2 network includes a convolutional layer, a pooling layer, a fully connected layer, grouped convolution, and depth separable convolution; the TCN network is composed of a plurality of residual blocks, and the TCN network extracts the ShuffleNet-V2 network from the feature vector. The time-indexed sequence is mapped to a new sequence by using 1D temporal convolution, and finally a lip motion feature with a dimension of 512×64 is obtained.

Preferably, the model built in the step 4 uses the ideal ratio of the complex number domain to cover cIRM as the training target of the audio during the training process, and uses triplet loss to calculate the similarity of the audio and facial images. The calculation formula of the cIRM is as follows:

Among them, X _r and _Xi represent the real part and imaginary part of the mixed speech signal, S _r and S _i represent the real part and imaginary part of the clean speech.

Preferably, in said step 2, the time-domain mixed audio is converted into a spectrogram by short-time Fourier transform, the audio is passed through a 16kHz sampling rate, the length of the audio segment is 2.55s, and the STFT has 400 window lengths and 160 jumps size and 512 FFT sizes.

Preferably, the input of the audio separation module is specifically: regarding the visual input, the facial image frame size is 224×224, which is extracted by the network into facial features with a dimension of 128; the mouth movement frame input size is 88×88, which is extracted by the network. Extract the mouth features with a dimension of 512×64, and combine them with facial features to finally obtain a visual feature with a dimension of 640×64, which is used as the visual input of the audio separation module; about the audio input, the signal spectrum of the mixed audio The graph is used as the visual input of the audio separation module, and its dimension is 2×257×256. After passing through the network, a prediction mask with the same dimension as the input spectrogram is obtained.

The second aspect of the present invention provides an audio-visual combined speech separation method, comprising the following process:

Obtain a video and corresponding audio containing two speakers;

Process the acquired video and the corresponding audio, and extract the face frame and mouth movement frame of the speaker in the video respectively,

Input the face frame, mouth movement frame and corresponding audio into the speech separation model built by the construction method described in the first aspect;

Output each speaker after separation and the corresponding clean speech.

The third aspect of the present invention also provides an audio-visual combined speech separation device, the device includes at least one processor and at least one memory, the processor and the memory are coupled; the memory stores the information as described in the first aspect The computer execution program of the speech separation model built by the construction method; when the processor executes the computer execution program stored in the memory, the processor can realize the speech separation method.

Compared with the prior art, the present invention has the following beneficial effects:

The present invention proposes to use two parts of visual features of mouth movement and face information to assist the realization process of speech separation at the same time. The assistance of visual information allows the network to better utilize the intrinsic connection between visual information and audio information, and achieve better separation performance; in order to further improve the extraction of visual features and consider that the video also includes human faces Secondary information, in the process of facial feature extraction, uses a two-layer attention mechanism to help the network obtain the most critical face area and use visual information more efficiently; for the traditional U-Net network model, which tends to ignore the lack of data details, this paper In the invention, the residual connection mechanism is introduced into the U-Net network, and the residual connection is added in the convolution block of the compressed path to assist the network to better extract detailed features, and at the same time, the BN layer is added after the convolution layer to speed up the training and processing of the network. Convergence speed; Compared with simply using the frequency domain characteristics of the audio signal, the present invention performs STFT transformation on the mixed speech signal, and fully utilizes the amplitude information and phase information of the speech signal.

Description of drawings

Fig. 1 is a frame diagram of the audio-visual combination speech separation model proposed by the present invention.

Figure 2 is a network structure diagram of the audio separation module.

Figure 3 is a structural diagram of the convolutional layer in the improved U-Nnet network.

Figure 4 is a structural diagram of the residual convolution block in the improved U-Net network.

Figure 5 is a network structure diagram of the face module.

Figure 6 is a schematic diagram of the attention mechanism module of the CBAM layer.

Figure 7 is a network structure diagram of the mouth movement module.

Fig. 8 is a simplified block diagram of the structure of the speech separation device in Embodiment 2.

Detailed ways

The invention will be further described below in conjunction with specific embodiments.

Example 1:

The present invention proposes a kind of audio-visual combination speech separation method based on dual visual cues, as shown in Figure 1, mainly comprises the following steps:

Step 1, obtaining raw data of video and corresponding audio of several speakers, said raw data being shot or downloaded and obtained in different scenes;

Step 2. Preprocess the raw data obtained in step 1; process the video into images frame by frame, and randomly select two speaker data from the raw data, mix the audio in it, and analyze the mixed voice Do the short-time Fourier transform to get the spectrogram of the speech, combine the face frames and mouth movement frames corresponding to the two speaker data to construct a data set, and divide it into a training set, a verification set and a test set;

Step 3, based on the U-Net network structure, establish a residual connection in the partial downsampling convolution block of the traditional U-Net to obtain a residual convolution block, and then between the convolution and ReLU activation functions of the compression path and the expansion path Add the BN layer to build an audio separation module; based on the ResNet-18 network structure, add the CBAM attention mechanism before and after the basic convolution block of ResNet-18 to build a face module; based on the ShuffleNet-V2 and TCN network structure, combined with 3D The convolutional layer is constructed as a mouth movement module; the above three network modules are combined to construct an AV-ResUnet network model; wherein the spectrogram of the mixed voice is input into the audio separation module, and the face frame is input into the face In the module, the mouth movement frame is input into the mouth movement module;

Step 4, use the training set and verification set described in step 2 to train and verify the AV-ResUnet network model built in step 3; select the model with the best verification effect in the training process as the final test model;

Step 5, use the data in the test set to test the final selected AV-ResUnet network model.

In this embodiment, experiments are carried out on the VoxCeleb2 data set, which contains more than 1,000,000 voice clips and corresponding video clips downloaded from YouTube, and the distribution of male and female ratio is relatively balanced, and the speakers come from many countries.

1. Get raw data

Since the videos included in the VoxCeleb2 dataset were shot in a large number of challenging visual and auditory environments. These included interviews on red carpets, outdoor stadiums, and in studios, which resulted in videos of varying quality. The video clips of some speakers in the data set are very blurry, and it is a big challenge for the network to extract useful information from them. Therefore, in this embodiment, some blurred videos in the data set are deleted to ensure better video and audio quality.

2. Data preprocessing

The acquired raw data is preprocessed; first, the video is processed into a frame-by-frame image, and one frame is selected as a face frame with a resolution of 224×224; each frame of image uses the SFD face detector to obtain facial key points, and the video The face in the image is aligned with the reference plane, and the similarity transformation is used to remove the position-related differences and position the lips. Then, it is cropped to a size of 96×96, and after grayscale processing, it is used as a mouth motion frame and saved as a frame. The .h5 file is convenient for the follow-up model to read the data; then randomly select the data of two speakers from the original data, mix the audio and perform short-time Fourier transform on the mixed speech to obtain the spectrogram of the speech, and combine the two Construct a dataset of face frames and mouth movement frames corresponding to the speaker data.

3. Model building

In this embodiment, the audio separation module is improved based on the U-Net network, including a conv layer, a res_conv layer, audio-visual feature fusion, an up_conv layer, and a Tanh function; the specific structure is shown in Figure 2.

The conv layer consists of a convolution kernel with a size of 4×4 and a step size of 2, a BN (Batch Normalization) layer, and a ReLU activation function. There are two conv layers in the compression path and expansion path of the network, respectively. unet_conv and unetup_conv; see Figure 3 for the detailed structure; (a) represents the conv layer in the U-Net network, and (b) represents the up_conv layer in the U-Net network.

The res_conv layer has 6 layers, namely res_conv1, res_conv2, res_conv3, res_conv4, res_conv5, and res_conv6. Each layer consists of two convolution kernels with a size of 3×3 and a step size of 1, two BN layers, and two ReLU activation functions. A Maxpool layer and a residual connection;

Among them, the number of data input and output channels of res_conv1 and res_conv2 is different, and the number of channels of input and output of the remaining four layers is the same. The residual connection is divided into two types according to the number of channels of input data and output data. When the number of channels is the same, directly input After adding the output of the convolution, pooling is performed. When the number of channels is different, the input is processed by a convolution kernel and then added. The residual connection can effectively avoid the problem of gradient disappearance during the training process of the network and help the network Better extract small details that are not easy to distinguish. The specific structure is shown in Figure 4. (a) represents the 1-2th residual convolution block, and (b) represents the 4th-6th residual convolution block. The number of input and output channels of each layer of the improved U-Net network is shown in Table 1.

Table 1 The number of channels in each layer of the improved U-Net network

Audio-visual feature fusion is the process of merging the audio features processed by the compressed path with the visual features extracted by the visual network in the time dimension to obtain audio-visual fusion features;

The up_conv layer consists of an Upsample layer, a convolution kernel with a size of 3x3 and a step size of 1, a BN layer, and a ReLU activation function. Upsample replaces Maxpool in the compression path. The specific structure is shown in Figure 3;

The tanh function compresses the data to the range -1 to 1 and outputs the separated mask, multiplies the separated mask and the spectrogram of the mixed voice to obtain a separate speaker's speech spectrogram, and then passes the inverse short-time Fourier The transformation restores the speaker's clean voice.

Face module, improved based on ResNet-18 network, including conv7 layer, CBAM layer, res layer, pooling layer and linear layer;

The conv7 layer is composed of a convolution kernel with a size of 7×7 and a step size of 2, a BN layer, and a ReLU activation function. The output of the conv7 layer is used as the input of the CBAM layer. The specific structure of the face module module is shown in Figure 5 .

The CBAM (Convolutional Block Attention Module) layer is composed of Channel Attention and Spatial Attention. The CBAM layer is located before the first res layer and after the last res layer, and is used to efficiently extract key areas of the face with high audio correlation. Ignore the secondary area outside the face; the CBAM layer structure is shown in Figure 6.

The res layer includes four layers res1, res2, res3, and res4, each containing 2 convolution blocks, where each convolution block in res1 consists of a 3×3 convolution kernel, a BN layer, and a ReLU activation function. The convolution A block can be represented by the following formula:

y = ReLU(x + BN(conv3(ReLU(BN(conv3(x))))))

Among them, x represents the input of the convolution block, and y represents the output of the convolution block; conv3 is a 3×3 convolution operation, BN refers to the batch normalization layer; ReLU refers to the ReLU activation function;

The first convolution block in res2, res3 and res4 is the same as in res1, the second convolution block consists of a 3×3 convolution kernel, BN layer, downsampling layer and ReLU activation function, and the second convolution block It can be expressed by the following formula:

y = ReLU ( Downsample(x) + BN(conv3(ReLU (BN(conv3(x))))))

Among them, Downsample refers to the downsampling layer; the rest are the same as the first convolution block.

The pooling layer includes maximum pooling and average pooling. The maximum pooling is located after the first CBAM layer to reduce the amount of parameters and simplify the complexity of the network; the average pooling is located after the second CBAM, and the output of the average pooling is As the input of the final linear layer; the output of the linear layer is the facial feature extracted by the network, which is copied in the time dimension and combined with the lip feature to become the visual feature required by the model;

Mouth movement module, based on ShuffleNet-V2 and TCN network structure, combined with 3D convolutional layer construction, the 3D convolutional layer consists of a convolution kernel with a size of 5×7×7 and a step size of 1×2×2, a BN layer , ReLU activation function and a 3D maximum pooling layer with a size of 1×3×3 and a step size of 1×2×2;

The ShuffleNet-V2 network consists of a convolutional layer, a pooling layer, a fully connected layer, grouped convolution, and depth-separable convolution; The temporal index sequence of feature vectors extracted by the network is mapped to a new sequence by using 1D temporal convolution, and finally a lip motion feature with a dimension of 512×64 is obtained.

4. Model training

In the present embodiment, a realization platform of audio-visual combined speech separation method based on dual vision clues is based on Linux operating system, the programming language is Python3.8, the deep learning framework is Pytorch1.11.0, the CUDA version is 11.1, and NVIDIA RTX 2080Ti graphics card is used . Using Adam as the optimizer, the learning rate is 0.00001, the batch size is 8, the total batch size is 5000, and the latest model is saved every 500 iterations. During the training process, every 100 iterations, use the verification set to check the training effect, and save the current optimal model. During the experiment, considering that the VoxCeleb2 dataset is too large and the training time is too long, which is not conducive to the experiment, in order to save time and cost and ensure fairness, the same part of the data is used for training, which does not affect the comparison of the models.

The data in the data set is a single sound. During training, the sound signals of two different speakers are randomly mixed, and the time-domain mixed audio is converted into a spectrogram through STFT. The audio is sampled at a rate of 16kHz, and the length of the audio segment is 2.55s. STFT With 400 window lengths, 160 hop sizes and 512 FFT sizes. The size of the facial image frame is 224x224, which is extracted by the network into facial features with a dimension of 128, and the input of the mouth movement frame is 64 frames with a size of 96x96 mouth grayscale frame, which is extracted by the network into a mouth feature with a dimension of 512x64, which is compared with The facial features are combined to finally obtain a visual feature with a dimension of 640x64, which is used as the visual input of the audio separation module; the signal spectrogram dimension of the mixed audio is 2x257x256, and a prediction mask with the same dimension as the input spectrogram is obtained after the network. The prediction mask is multiplied by the spectrogram of the mixed speech to obtain the separated speaker speech spectrogram, and then the speaker's clean speech signal is recovered by inverse short-time Fourier transform (STFT).

5. Experimental results

In this embodiment, the separation performance of the proposed method and the audio-visual speech separation model using a single visual cue is compared, and the separation performance of the improved model and the basic model is compared to verify the effectiveness of the proposed scheme of the present invention. Commonly used evaluation indicators for speech separation tasks are active distortion ratio SDR, perceptual evaluation of speech quality PESQ, short-term objective intelligibility STOI, etc. SDR indicates the overall distortion of the signal; STOI measures the correlation between the reference (clean) utterance and the short-term temporal envelope of the separated utterance; for speech quality, PESQ is the standard measure, applying an auditory transformation to produce a loudness spectrum and compare clean Loudness spectra of the reference and separated signals. In this embodiment, PESQ, SDR and STOI are used as evaluation indexes for comparative experiments. In this embodiment, only the results of mixing two speakers are compared and analyzed.

Attention mechanism effect verification:

In the video feature extraction network, the CBAM attention mechanism is added to the facial module of the model proposed by the present invention to help the network extract key facial information, and ignore the secondary information that is useless for separation, so as to improve the performance of separation. In this example, the model effects of Squeeze Excitation (SE) and CBAM attention mechanism are compared. It should be noted that the results shown in the table below are achieved by the separation model based on the original U-Net network. Two data in the test set are selected to test the effect. The evaluation index results of the separated speech are shown in Table 2. The experimental results show that adding two layers of CBAM attention mechanism can help improve the effect of speech separation.

Table 2 Experimental results of different attention mechanisms

It can be seen that the addition of the CBAM attention mechanism helps the visual feature extraction module to extract facial information more accurately, that is, the key faces in the video stream contain features that are more beneficial to the separation task. The experimental results further verify the feasibility of this idea . The separation model with two layers of CBAM attention mechanism improves the PESQ score by 0.09 and the SDR improvement by 0.8dB compared to the model without the attention mechanism. Different from the CBAM attention mechanism, the effect of adding SE attention mechanism does not improve the separation effect, which may be due to the fact that the processing mechanism of CBAM for visual information is closer to the processing mechanism of the human brain for visual information in audio-visual mode.

Residual connection effect verification:

Inspired by the application of residual connections in the field of image processing can help the network to better extract image details and the ability of residual connections to help networks avoid degradation problems. In the audio signal processing network, the proposed model of the present invention adds residual connections on the basis of U-Net. The function of the residual connection is to act as a bridge to pass the information of the previous layer to the next layer when the deep network degenerates (the effect of deepening the number of network layers decreases instead). The model proposed in the present invention adds residual connections to the convolutional layer of the compressed path, and verifies the improvement of the separation effect by adding different numbers and types of residual connections. Select two data in the test set for verification, and the evaluation index results of the speech after separation are shown in the table below. As shown in Table 3, the results show that the network model with 6 residuals improves the performance of separating audio compared to the U-Net network without residual connections.

Table 3 Experimental results of different residuals

Visual information effect verification:

In order to verify the effectiveness of using dual visual information to improve the performance of speech separation, this embodiment compares the performance of speech separation models that combine audio with mouth information, audio with facial information, and audio with both face and lip information. Specifically, in the experiment, the pure mouth information features, pure facial information features and mouth and face fusion features obtained by the visual feature extraction module in the proposed model were fused into the separation module as the final visual features, and the network structure of the separation module remain unchanged, the results are shown in Table 4.

Table 4 Experimental effects of different visual cues

Model separation works differently when different visual cues are introduced. It can be seen from the table that compared with simply using mouth movements or facial information, the dual visual cues used in this paper can make more comprehensive use of the speaker's visual and speech information. Compared with the method using only face information, the PESQ of the method proposed in the present invention is increased by 0.23, the SDR is increased by 2.5dB, and the STOI is increased by 0.08, achieving better separation performance.

In different application scenarios, the speech separation model built in the present invention can be used for speech separation:

First obtain the video and corresponding audio containing two speakers;

Process the acquired video and the corresponding audio, and extract the face frame and mouth movement frame of the speaker in the video respectively,

Input the face frame, mouth movement frame and corresponding audio into the speech separation model built by the above method;

Output each speaker after separation and the corresponding clean speech.

Example 2:

As shown in Fig. 8, the present invention simultaneously provides a kind of audio-visual combination speech separation equipment, and equipment comprises at least one processor and at least one memory, also comprises communication interface and internal bus simultaneously; Computer execution program is stored in the memory; The computer execution program of the speech separation model built by the construction method described in Embodiment 1 is stored; when the processor executes the computer execution program stored in the memory, the processor can realize the speech separation method. The internal bus may be an Industry Standard Architecture (Industry Standard Architecture, ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (.Xtended Industry Standard Architecture, EISA) bus, etc. The bus can be divided into address bus, data bus, control bus and so on. For ease of representation, the buses in the drawings of the present application are not limited to only one bus or one type of bus. The storage may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk storage, and may also be a U disk, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, and the like.

A device may be provided as a terminal, server, or other form of device.

Fig. 8 is a block diagram of a device shown for example. A device may include one or more of the following components: a processing component, a memory, a power supply component, a multimedia component, an audio component, an input/output (I/O) interface, a sensor component, and a communication component. The processing components typically control the overall operations of the electronic device, such as those associated with display, telephone calls, data communications, camera operations, and recording operations. The processing component may include one or more processors to execute instructions to complete all or part of the steps of the above method. Additionally, a processing component may include one or more modules to facilitate interaction between the processing component and other components. For example, the processing component may include a multimedia module to facilitate interaction between the multimedia component and the processing component.

The memory is configured to store various types of data to support operations at the electronic device. Examples of such data include instructions for any application or method operating on the electronic device, contact data, phonebook data, messages, pictures, videos, etc. The memory can be realized by any type of volatile or non-volatile storage devices or their combination, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read Only Memory (PROM), Read Only Memory (ROM), Magnetic Memory, Flash Memory, Magnetic or Optical Disk.

Power components provide power to various components of electronic equipment. Power components may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic devices. The multimedia component includes a screen providing an output interface between said electronic device and a user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensor may not only sense a boundary of a touch or swipe action, but also detect duration and pressure associated with the touch or swipe action. In some embodiments, the multimedia component includes a front camera and/or a rear camera. When the electronic device is in an operation mode, such as a shooting mode or a video mode, the front camera and/or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have focal length and optical zoom capability.

The audio component is configured to output and/or input audio signals. For example, the audio component includes a microphone (MIC), which is configured to receive an external audio signal when the electronic device is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal may be further stored in a memory or sent via a communication component. In some embodiments, the audio component further includes a speaker for outputting audio signals. The I/O interface provides an interface between the processing component and the peripheral interface module, and the above peripheral interface module can be a keyboard, a click wheel, a button, and the like. These buttons may include, but are not limited to: a home button, volume buttons, start button, and lock button.

A sensor assembly includes one or more sensors that provide status assessments of various aspects of an electronic device. For example, the sensor component can detect the open/closed state of the electronic device, the relative positioning of components, such as the display and keypad of the electronic device, the sensor component can also detect the position change of the electronic device or a component of the electronic device, and the user The presence or absence of contact with the electronic device, the orientation or acceleration/deceleration of the electronic device and the temperature change of the electronic device. The sensor assembly may include a proximity sensor configured to detect the presence of nearby objects in the absence of any physical contact. The sensor assembly may also include optical sensors, such as CMOS or CCD image sensors, for use in imaging applications. In some embodiments, the sensor assembly may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor or a temperature sensor.

The communication component is configured to facilitate wired or wireless communication between the electronic device and other devices. Electronic devices can access wireless networks based on communication standards, such as WiFi, 2G or 3G, or a combination thereof. In one exemplary embodiment, the communication component receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication assembly further includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

In an exemplary embodiment, the electronic device may be programmed by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable A gate array (FPGA), controller, microcontroller, microprocessor or other electronic component implementation for performing the methods described above.

The above descriptions are only preferred embodiments of the present application, and are not intended to limit the present application. For those skilled in the art, there may be various modifications and changes in the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this application shall be included within the protection scope of this application.

Although the specific implementation of the present invention has been described above, it is not a limitation to the protection scope of the present invention. Those skilled in the art should understand that on the basis of the technical solution of the present invention, those skilled in the art can do it without creative work. Various modifications or deformations are still within the protection scope of the present invention.

Claims (9)

1. An audio-visual combined voice separation model building method is characterized by comprising the following steps:

step 1, acquiring original data of videos and corresponding audios of a plurality of speakers, wherein the original data are shot or downloaded in different scenes;

step 2, preprocessing the original data obtained in the step 1; processing videos into images of one frame and one frame respectively, randomly selecting data of two speakers from original data, mixing audio in the data, performing short-time Fourier transform on the mixed voice to obtain a spectrogram of the voice, combining face frames and mouth action frames corresponding to the data of the two speakers to construct a data set, and dividing the data set into a training set, a verification set and a test set;

step 3, based on a U-Net network structure, establishing residual connection in a partial downsampling convolution block of the traditional U-Net to obtain a residual convolution block, and then adding a BN layer between convolution of a compression path and an expansion path and a ReLU activation function to construct an audio separation module; based on the ResNet-18 network structure, adding a CBAM attention mechanism before and after a basic convolution block of ResNet-18 respectively to construct a face module; based on the SheffleNet-V2 and TCN network structure, the method combines a 3D convolution layer to construct a mouth action module; combining the three network modules to construct an AV-Resunate network model; wherein a spectrogram of the mixed speech is input into the audio separation module, a face frame is input into the face module, and a mouth action frame is input into the mouth action module;

The input of the audio separation module is specifically as follows: regarding visual input, the face frame size is 224×224, extracted via a network into facial features of dimension 128; the mouth action frame is input into 88 x 88, extracted into 512 x 64 mouth features through a network, and combined with facial features to finally obtain 640 x 64 visual features, wherein the visual features are used as visual input of an audio separation module; regarding audio input, the signal spectrogram of the mixed audio is used as the audio input of the audio separation module, the dimension is 2 multiplied by 257 multiplied by 256, and a prediction mask which is consistent with the dimension of the input spectrogram is obtained after the network; multiplying the prediction mask with the spectrograms of the mixed voices respectively to obtain separated individual speaker voice spectrograms, and recovering a speaker clean voice signal through inverse short-time Fourier transform;

step 4, training and verifying the AV-Resunate network model built in the step 3 by using the training set and the verification set in the step 2; selecting a model with the best verification effect in the training process as a final test model;

and 5, testing the finally selected AV-Resunate network model by using the data in the test set.

2. The audio-visual combined speech separation model building method according to claim 1, wherein the specific process of preprocessing in the step 2 is: firstly, processing a video into an image frame by frame, and selecting a frame as a face frame; each frame of image is used for acquiring facial key points by using an SFD (small form-factor detector), removing the difference related to the position, positioning the position of the lips, then cutting the lips into a fixed size, and taking the lips as a mouth action frame after gray-scale treatment, wherein the frame number is 64; and then randomly selecting the data of two speakers from the original data, mixing the audio frequencies in the data, and then performing short-time Fourier transform on the mixed voice to obtain a spectrogram of the voice, and combining face frames and mouth action frames corresponding to the data of the two speakers to construct a data set.

3. The audio-visual combined speech separation model building method according to claim 1, wherein: the audio separation module is improved based on a U-Net network and comprises a conv layer, a res_conv layer, audio-visual feature fusion, an up_conv layer and a Tanh function;

the conv layer consists of a convolution kernel with the size of 4 multiplied by 4 and the step length of 2, a BN layer and a ReLU activation function, wherein two conv layers are respectively in a compression path and an expansion path of the network, namely unet_conv and unet_conv;

the res_conv layers comprise 6 layers, namely res_conv1, res_conv2, res_conv3, res_conv4, res_conv5 and res_conv6, and each layer consists of two convolution kernels with the 3×3 step length of 1, two BN layers, two ReLU activation functions, one Maxpool layer and one residual error connection;

the method comprises the steps of dividing the input data into two types according to the difference of the channel numbers of the input data and the output data, directly adding the input data and the convolved output to be pooled when the channel numbers are the same, and adding the input data after one convolution kernel processing when the channel numbers are different;

The audio-visual feature fusion is a process of fusing audio features obtained after the compression path processing with visual features extracted by a visual network in a time dimension to obtain audio-visual fusion features;

the up_conv layer consists of an Upsample layer, a convolution kernel with the size of 3 multiplied by 3 and the step length of 1, a BN layer and a ReLU activation function, wherein Upsample replaces Maxpool in a compression path;

and the Tanh function compresses data to a section from-1 to 1, outputs a separated mask, multiplies the separated mask by a spectrogram of the mixed voice to obtain an independent speaker voice spectrogram, and then recovers the clean voice of the speaker through inverse short time Fourier transform.

4. The audio-visual combined speech separation model building method according to claim 1, wherein: the face module is improved based on a ResNet-18 network and comprises a conv7 layer, a CBAM layer, a res layer, a pooling layer and a linear layer;

the conv7 layer consists of a convolution kernel with the size of 7 multiplied by 7 and the step length of 2, a BN layer and a ReLU activation function, and the output of the conv7 layer is used as the input of the CBAM layer;

the CBAM layer consists of Channel Attention and Spatial Attention, is respectively positioned before the first res layer and after the last res layer and is used for efficiently extracting and audio-related face key areas and ignoring secondary areas outside the face;

The res layer comprises four layers res1, res2, res3 and res4, and each of the four layers comprises 2 convolution blocks, wherein each convolution block in res1 consists of a 3×3 convolution kernel, a BN layer and a ReLU activation function, and the convolution blocks can be represented by the following formula:

y = ReLU(x + BN(conv3(ReLU(BN(conv3(x))))))

wherein x represents the input of the convolution block and y represents the output of the convolution block; conv3 is a 3×3 convolution operation, BN batch normalization layer; reLU refers to a ReLU activation function;

the first convolution block in res2, res3 and res4 is the same as res1, the second convolution block is composed of a 3×3 convolution kernel, BN layer, downsampling layer and ReLU activation function, and the second convolution block can be expressed by the following formula:

y = ReLU ( Downsample(x) + BN(conv3(ReLU (BN(conv3(x))))))

wherein downsampled refers to a downsampling layer;

the pooling layer comprises maximum pooling and average pooling, wherein the maximum pooling is positioned after the first CBAM layer and is used for reducing the parameter number and simplifying the complexity of the network; the average pooling is located after the second CBAM, the average pooled output being the input to the final linear layer;

the output of the linear layer is taken as the final facial feature extracted by the network, and the final facial feature is combined with the mouth feature after being copied in the time dimension to form the visual feature required by the model.

5. The audio-visual combined speech separation model building method according to claim 1, wherein: the mouth action module is constructed based on a SheffleNet-V2 and TCN network structure and combined with a 3D convolution layer, wherein the 3D convolution layer consists of a convolution kernel with the size of 5 multiplied by 7 and the step size of 1 multiplied by 2, a BN layer, a ReLU activation function and a 3D maximum pooling layer with the size of 1 multiplied by 3 and the step size of 1 multiplied by 2;

The ShuffleNet-V2 network includes a convolutional layer, a pooling layer, a fully-connected layer, a packet convolution, and a depth separable convolution; the TCN network is composed of a plurality of residual blocks, and maps the time index sequence of the feature vector extracted by the ShuffleNet-V2 network into a new sequence by using 1D time convolution, so as to finally obtain a mouth feature with dimension of 512×64.

6. The audio-visual combined speech separation model building method according to claim 1, wherein: in the training process, the model constructed in the step 4 uses complex domain ideal ratio masking cIRM as a training target of audio, and uses a triple loss to calculate similarity between the audio and the facial image, wherein the calculation formula of the cIRM is as follows:

wherein X is _r And X _i Representing the real and imaginary parts of the mixed speech signal, S _r And S is _i Representing the real and imaginary parts of clean speech.

7. The audio-visual combined speech separation model building method according to claim 1, wherein: in the step 2, the time domain mixed audio is converted into a spectrogram through short-time Fourier transform, the audio is subjected to 16kHz sampling rate, the audio fragment length is 2.55s, and the STFT has 400 window lengths, 160 jump sizes and 512 FFT sizes.

8. An audiovisual combined speech separation method, characterized by comprising the following processes:

acquiring video and corresponding audio containing two speakers;

processing the acquired video and corresponding audio, respectively extracting face frames and mouth action frames of a speaker in the video,

inputting a face frame, a mouth action frame and corresponding audio into a speech separation model constructed by the construction method according to any one of claims 1 to 7;

outputting each separated speaker and the corresponding clean voice.

9. An audiovisual combined speech separation device, characterized by: the apparatus includes at least one processor and at least one memory, the processor and the memory coupled; a computer-implemented program of a speech separation model constructed by the construction method according to any one of claims 1 to 7 is stored in the memory; the processor may be caused to implement a voice separation method when executing a computer-implemented program stored in the memory.

Images