CN106960672A - Method and device for bandwidth expansion of stereo audio - Google Patents

Abstract

The invention discloses a bandwidth extension method and device of stereo audio. The method comprises the following steps: decomposing the stereo signal into direct sound and diffuse sound; performing bandwidth extension on the diffused sound according to a preset frequency band extension method; separating direct sound into a plurality of point sound sources in different directions, and respectively performing bandwidth expansion on the plurality of point sound sources to obtain a plurality of point sound sources after bandwidth expansion; remixing the plurality of point sound sources after bandwidth expansion according to the pre-estimated azimuth information to obtain direct sound after bandwidth expansion; and reconstructing a broadband stereo audio signal according to the direct sound after bandwidth expansion and the diffuse sound after bandwidth expansion. By means of the technical scheme, the problem that in the prior art, signal bandwidth expansion is realized only according to subjective quality of a single channel reconstruction signal, correlation between signal energy and phases in two channels is not considered, and the reconstruction of stereo signals seriously influences judgment of a listener on the position and distance of a sound source is solved.

Description

Method and device for bandwidth expansion of stereo audio

technical field

The invention relates to the application field of network technology, in particular to a method and device for extending the bandwidth of stereo audio.

Background technique

In digital audio signal processing technology, an audio signal covering the entire frequency range of 20Hz to 20KHz that can be perceived by the human ear is usually called full-band audio. This type of signal is mainly used for high-fidelity reproduction of music signals. The current audio instant messaging system cannot provide sufficient network transmission rate and terminal processing capability, which will inevitably limit the effective bandwidth of the reconstructed signal, give priority to quantizing the low-frequency components of the encoded audio signal, and then improve the coding efficiency of the audio communication system.

The traditional telephone voice communication system usually transmits narrowband signals, whose frequency is distributed in the range of 300-3400Hz, and the sampling rate is 8kHz. The relevant subjective listening test results show that 91% of syllable intelligibility and 99% of sentence intelligibility are preserved in narrowband speech. However, compared with real speech, the naturalness and subjective quality of narrowband signals transmitted in actual calls are significantly reduced. Due to the absence of high-frequency components, narrowband speech cannot distinguish parts of unvoiced or plosive sounds well, and weakens its ability to describe the characteristics of the speaker. In order to effectively overcome the shortcomings of narrowband audio, wideband audio is widely used in the field of telephone voice communication, and its effective bandwidth is extended to 50Hz ~ 7kHz, which covers most of the frequency spectrum that represents the important characteristics of voice signals, and realizes close to AM broadcasting sound quality level. However, limited by many problems such as history, economy, and technology, it will take a long period of transition for traditional fixed and mobile communications to fully realize the transition from narrowband to wideband audio.

As an effective audio enhancement method, the frequency band extension method can artificially recover from the reconstructed wideband audio at the receiving end by analyzing the time-frequency characteristics of the original audio signal without changing the narrowband signal source coding and network transmission. The high-frequency components truncated by the encoding end are output, so as to achieve the purpose of enhancing the auditory quality of the reconstructed audio. For hearing-impaired persons, the frequency band extension method can further improve their phonemic and semantic discrimination. In the past ten years, many research institutions and researchers have successively proposed many solutions for the frequency band extension of monophonic speech signals. These methods usually start from two aspects of spectrum envelope expansion and spectrum detail expansion respectively, and then synthesize the high-frequency components of the signal. The principle is shown in Figure 1. Firstly, time-frequency feature extraction is performed on the narrowband signal according to the principle of human auditory perception; next, the spectral envelope and energy of the high-frequency components are estimated by using the mapping relationship between high and low frequency features described by side information or prior knowledge; At the same time, an appropriate spectrum patching method is selected to expand the spectrum details; finally, combined with the expanded spectrum envelope and spectrum details, the effective reconstruction of the high-frequency components of the wideband audio signal is realized.

For stereo audio, the traditional frequency band extension method mostly performs independent reconstruction of high-frequency components for two channels. This type of method only expands the signal bandwidth based on the subjective quality of the reconstructed signal of a single channel, and does not take into account the signals in the two channels. The correlation between energy and phase, the reconstructed stereo signal seriously affects the listener's judgment on the position and distance of the sound source.

Contents of the invention

In view of the above problems, the present invention provides a method and device for bandwidth expansion of stereo audio.

The bandwidth extension method of stereo audio provided by the present invention comprises the following steps:

Decompose the stereo signal into direct sound and diffuse sound;

performing bandwidth expansion on the diffuse sound according to a preset frequency band expansion method;

separating the direct sound into a plurality of point sound sources in different orientations, respectively performing bandwidth expansion on the plurality of point sound sources, to obtain a plurality of point sound sources after bandwidth expansion;

remixing the multiple point sound sources after the bandwidth expansion according to the pre-estimated orientation information to obtain the direct sound after the bandwidth expansion;

A broadband stereo audio signal is reconstructed according to the bandwidth-expanded direct sound combined with the bandwidth-extended diffuse sound.

The present invention also provides a stereo audio bandwidth expansion device, including: a decomposition module, a diffuse sound expansion module, a direct sound separation and expansion module, and a reconstruction module;

The decomposition module is used to decompose the stereo signal into direct sound and diffuse sound;

The diffuse sound extension module is configured to extend the bandwidth of the diffuse sound according to a preset frequency band extension method;

The direct sound separation and expansion module is used to separate the direct sound into a plurality of point sound sources in different orientations, respectively perform bandwidth expansion on the plurality of point sound sources, and obtain a plurality of point sound sources after bandwidth expansion;

The reconstruction module is configured to remix the multiple point sound sources after the bandwidth expansion according to the pre-estimated orientation information to obtain the direct sound after the bandwidth expansion, and combine the bandwidth expansion with the direct sound after the bandwidth expansion The subsequent diffuse sound reconstructs a wideband stereo audio signal.

The beneficial effects of the present invention are as follows:

In the embodiment of the present invention, firstly, the input stereo signal is decomposed into two components of direct sound and diffuse sound by utilizing the spectral correlation between channels, and then the diffuse sound component is directly expanded by using the traditional frequency band expansion method; According to the sparsity of the frequency structure, it is separated into multiple point sound sources with different orientations, and the bandwidth is expanded respectively. Finally, the expanded point sound sources are remixed according to their orientation information in the original stereo, and combined with the diffuse sound after bandwidth expansion. components to reconstruct a wideband stereo audio signal. The present invention solves the problem of expanding the signal bandwidth only based on the subjective quality of the reconstructed signal of a single channel in the prior art, without considering the correlation of signal energy and phase in the two channels, and the reconstructed stereo signal seriously affects the listener The problem of judging the location and distance of the sound source.

Description of drawings

Fig. 1 is the basic flowchart of monophonic speech signal frequency band extension method in the prior art;

Fig. 2 is the flow chart of the bandwidth extension method of stereo audio of the method embodiment of the present invention;

FIG. 3 is a schematic structural diagram of a stereo audio bandwidth extension device according to an embodiment of the present invention;

Fig. 4 is the functional block diagram of the bandwidth expansion method of the stereo audio of the example 1 of the present invention;

Fig. 5 is a functional block diagram of a state-space model based on a deep neural network in Example 1 of the present invention.

detailed description

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided for more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

In order to solve the problem of extending the signal bandwidth only based on the subjective quality of the reconstructed signal of a single channel in the prior art, the correlation between the signal energy and phase in the two channels is not considered, and the reconstruction of the stereo signal seriously affects the listener's perception of the sound quality. For the determination of source position and distance, the present invention provides a method and device for bandwidth expansion of stereo audio. The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

According to the method embodiment of the present invention, a method for extending the bandwidth of stereo audio is provided. FIG. 1 is a flow chart of the method for extending the bandwidth of stereo audio according to the method embodiment of the present invention. As shown in FIG. 1 , according to the method embodiment of the present invention The bandwidth extension method for stereo audio includes the following processing:

Step 201, decompose the stereo signal into direct sound and diffuse sound.

Specifically, step 201 includes the following steps:

decomposing the stereo signal into left and right channels;

Carry out time-frequency conversion to the left channel and the right channel after the frame division processing respectively, and obtain the left channel short-term spectral component and the right channel short-term spectral component of the stereo signal;

According to the short-time spectral components of the left channel and the short-time spectral components of the right channel, respectively, the sum P _sum between the left and right channel signal energy spectra, the difference P _diff between the left and right channel signal energy spectra, and the left and right channel Cross-correlation P _cc between signal energy spectra;

Using the P _sum , P _diff , and P _cc to obtain the direct acoustic matrix by the least square method;

separating direct sound from said stereo signal using said direct sound matrix;

The diffuse sound is obtained by subtracting the direct sound from the stereo signal.

More specifically, according to the short-time spectral components of the left channel and the short-term spectral components of the right channel, the sum P _sum between the left and right channel signal energy spectra and the difference between the left and right channel signal energy spectra are obtained P _diff , the cross-correlation P _cc between the left and right channel signal energy spectra, including:

According to the formula P _sum =| _{S L (} t,f)| ² + |S _R (t, f)| ² Calculate the sum P _sum between the left and right channel signal energy spectra;

According to the formula P _diff =| _{S L (} t,f)| ² - |S _R (t, f)| ² calculates the difference P _diff between the left and right channel signal energy spectra;

Using the left channel short-term spectral component S _L (t, f) and the right channel short-term spectral component S _R (t, f) according to the formula P _cc =R{S _L (t,f)S _R ^* (t,f)} calculates the cross-correlation P _cc between the left and right channel signal energy spectra, where R{} is the real part operation.

More specifically, said separating the direct sound from the stereo signal by using the direct sound matrix includes:

Using the direct sound matrix M _D (t, f) to separate the direct sound S'(t, f) from the stereo signal S(t, f) according to formula 1;

S'(t, f) = M _D (t, f) [S _L (t, f) S _R (t, f)] ^T Formula 1.

Step 202, performing bandwidth extension on the diffuse sound according to a preset frequency band extension method.

Specifically, step 202 directly uses the traditional frequency band extension method to extend the bandwidth of the diffuse sound, which will not be described in detail in the present invention.

In step 203, the direct sound is separated into multiple point sound sources in different directions, and bandwidth expansion is performed on the multiple point sound sources respectively to obtain multiple point sound sources after bandwidth expansion.

Specifically, in step 203, the direct sound is separated into multiple point sound sources with different orientations, including:

Calculate the direction information of the direct sound at each time-frequency point, cluster the direction information of all time-frequency points, and obtain the cluster centers of the direction information, and the cluster centers correspond to the direction information of each point sound source;

Obtain a masking matrix according to the direction information of the direct sound at a certain time-frequency point and the cluster center of the direction information;

The direct sound is separated by using the masking matrix to obtain multiple point sound sources with different orientations.

Specifically, the bandwidth expansion of multiple point sound sources includes:

Input multiple point sound sources into the preset state space model to fit the mapping relationship between the short-time spectrum of the narrowband signal and the short-time spectrum of the wideband signal, and analyze the short-time spectrum of the wideband signal according to the preset error criterion The spectrum envelope of the high-frequency component is estimated, combined with the low-frequency spectrum envelope and the spectrum details expanded by the appropriate spectrum patching method, multiple point sound sources with bandwidth expansion are obtained.

More specifically, the mapping relationship between the short-time spectrum of the narrowband signal and the short-time spectrum of the wideband signal is fitted in the state space model, and the spectrum envelope of the high-frequency component is calculated according to a preset error criterion estimates, including:

Using the hidden state vector at the previous moment and the short-time spectrum of the narrowband signal at the previous moment, the hidden state vector in the state space model is obtained;

The short-time spectrum of the wideband signal is obtained by using the hidden state vector in the state space model and the short-time spectrum of the narrowband signal at the current moment.

Step 204: Remix the multiple bandwidth-expanded point sound sources according to the preset orientation information to obtain the bandwidth-expanded direct sound, and reconstruct according to the bandwidth-expanded direct sound combined with the bandwidth-extended diffuse sound Output wideband stereo audio signal.

Specifically, the pre-estimated orientation information is estimated according to the clustering center of the orientation information, and the estimation method is a conventional technical means in the art, which will not be described in detail in the present invention.

Specifically, using Formula 2 to reconstruct a broadband stereo audio signal according to the bandwidth-expanded direct sound combined with the bandwidth-expanded diffuse sound;

In Equation 2, Represents the short-term spectrum of the wideband extended stereo signal; Represents the short-term spectrum of the direct sound after broadband expansion; Represents the short-term spectrum of diffuse sound after bandwidth expansion.

Corresponding to the method embodiment of the present invention, a stereo audio bandwidth expansion device is provided. FIG. 3 is a schematic structural diagram of the stereo audio bandwidth expansion device in the device embodiment of the present invention. As shown in FIG. 3 , the device according to the present invention The stereo audio bandwidth expansion device of the embodiment includes: a decomposition module 30 , a diffuse sound expansion module 32 , a direct sound separation and expansion module 34 , and a reconstruction module 36 , and each module of the embodiment of the present invention will be described in detail below.

Specifically, the decomposition module 30 is configured to decompose the stereo signal into direct sound and diffuse sound;

The diffuse sound extension module 32 is configured to extend the bandwidth of the diffuse sound according to a preset frequency band extension method;

The direct sound separation and expansion module 34 is used to separate the direct sound into a plurality of point sound sources in different orientations, respectively perform bandwidth expansion on the plurality of point sound sources, and obtain a plurality of point sound sources after bandwidth expansion;

The reconstruction module 36 is configured to remix the multiple point sound sources after the bandwidth expansion according to the pre-estimated orientation information to obtain the direct sound after the bandwidth expansion, and to obtain the direct sound after the bandwidth expansion according to the direct sound after the bandwidth expansion. A wideband stereo audio signal is reconstructed by combining the diffused sound after bandwidth expansion.

The decomposition module 30 is specifically used for:

decomposing the stereo signal into left and right channels;

Carry out time-frequency conversion to the left channel and the right channel after the frame division processing respectively, and obtain the left channel short-term spectral component and the right channel short-term spectral component of the stereo signal;

According to the short-time spectral components of the left channel and the short-time spectral components of the right channel, respectively, the sum P _sum between the left and right channel signal energy spectra, the difference P _diff between the left and right channel signal energy spectra, and the left and right channel Cross-correlation P _cc between signal energy spectra;

Using the P _sum , P _diff , and P _cc to obtain the direct acoustic matrix by the least square method;

separating direct sound from said stereo signal using said direct sound matrix;

The diffuse sound is obtained by subtracting the direct sound from the stereo signal.

The direct sound separation and expansion module 34 is specifically used for:

Calculate the direction information of the direct sound at each time-frequency point, cluster the direction information of all time-frequency points, and obtain the cluster centers of the direction information, and the cluster centers correspond to the direction information of each point sound source;

Obtain a masking matrix according to the direction information of the direct sound at a certain time-frequency point and the cluster center of the direction information;

The direct sound is separated by using the masking matrix to obtain multiple point sound sources with different orientations.

The direct sound separation and expansion module 34 is specifically used for:

Input multiple point sound sources into the preset state space model to fit the mapping relationship between the short-time spectrum of the narrowband signal and the short-time spectrum of the wideband signal, and analyze the short-time spectrum of the wideband signal according to the preset error criterion The spectrum envelope of the high-frequency component is estimated, combined with the low-frequency spectrum envelope and the spectrum details expanded by the appropriate spectrum patching method, and finally the direct sound after bandwidth expansion is obtained.

In order to illustrate the technical solution of the present invention in more detail, give example 1, Fig. 4 is the functional block diagram of the bandwidth extension method of the stereo audio of the present invention example 1, as shown in Fig. 4, a kind of bandwidth extension method of stereo audio includes the following step:

1. Direct sound/diffuse sound separation

The stereo extension system proposed in this paper uses discrete Fourier transform or orthogonal mirror filter bank to convert the framed left and right channel audio signals into the frequency domain respectively, and divides them into multiple uniform subbands according to the principle of human auditory perception or critical bands. Then, the short-term spectrum S(t,f) of the input stereo signal can be expressed as S(t,f)=[S _L (t,f) _SR (t,f)] ^T

Among them, t and f represent the time frame and sub-band number of the signal, respectively; _SL (t, f) and S _R (t, f) represent the short-term spectral components of the left and right channels of the stereo signal, respectively.

In order to effectively separate the direct sound from the diffuse sound, the system also needs to calculate the sum P _sum and difference P _diff between the left and right channel signal energy spectra and the cross-correlation P _cc of the two channels.

P _sum ＝|S _L (t,f)| ² +|S _R (t,f)| ²

P _diff ＝|S _L (t,f)| ² -|S _R (t,f)| ²

P _cc ＝R{S _L (t,f)S _R ^* (t,f)}

Among them, R{} is the operation of taking the real part. In order to improve the stability of the separation algorithm, time smoothing is performed on the calculated P _sum , P _diff and P _cc respectively.

The direct sound components in the left and right stereo channels are highly correlated and can be represented as a point source signal traveling from a certain direction. Accordingly, the system proposed in this paper uses a direct sound matrix to directly separate the direct sound component S'(t,f) from the original stereo two-channel signal S(t,f), as shown in the following formula,

S'(t,f)＝[S _L '(t,f)S _R '(t,f)] ^T ＝M _D (t,f)[S _L (t,f)S _R (t,f) ] ^T = M _D (t,f)S(t,f)

Among them, S _L '(t,f) and S _R '(t,f) respectively represent the short-term spectral components of the left and right channels of the direct sound, and M _D (t, f) is the matrix of the direct sound. According to the literature [M Vinton, D McGrath, C Robinson, P Brown, next generationsurround decoding and upmixing for consumer ad professional applications. ^{AES57 th} International conference, USA, 2015], the direct acoustic matrix M _D (t, f) can be used The least square method is obtained, so that the expected square error between the estimated direct sound component and the real component is minimized, that is,

Then the direct acoustic matrix M _D (t, f) can be calculated by the following formula,

The diffuse sound component S”(t,f) can be expressed as the difference between the original stereo signal and the direct sound component,

S"(t,f)=S(t,f)-S'(t,f)

2. Sound source separation of direct sound components

According to S'(t,f)=[S _L '(t,f)S _R '(t,f)] ^T use formula 3 to get the direction information θ of the direct sound S'(t,f) at a certain time-frequency point (t, f), the direction information θ (t, f) of the direct sound S' (t, f) at a certain time-frequency point is the same as the direction information θ _i of the point sound source;

Cluster the direction information θ(t,f) of all time-frequency points to obtain the cluster centers C _i of direction information, i=1, 2...N; these cluster centers correspond to each point sound source S ₁ (t ,f), S ₂ (t,f), S ₃ (t,f)...S _N (t,f) direction information θ ₁ , θ ₂ , θ ₃ ...θ _N ;

According to the direction information θ(t,f) of the direct sound S'(t,f) at a certain time-frequency point and the clustering center C _i , the masking matrix m _i (t,f) is obtained;

Use the masking matrix m _i (t, f) to separate the direct sound S'(t, f) according to formula 4 to obtain the direct point sound source

3. Bandwidth expansion

According to the method described above, the diffuse sound component S”(t,f) and the direct sound component S’(t,f) are separated from the stereo signal, and the direct sound component S’(t,f) is further separated by using time-frequency sparsity t, f) separated into multiple point sound sources Next, the diffuse sound S”(t, f) and the direct point sound source can be respectively analyzed according to the monophonic frequency band extension method Perform independent bandwidth expansion.

In this paper, the state-space model is used to directly fit the mapping relationship between the narrow-band spectrum parameters, and the spectrum envelope of the high-frequency components is estimated according to a certain error criterion in the actual extension.

S _Y (t,f)=F[S _X (t,f)]

In the formula, S _X (t, f) and S _Y (t, f) represent the short-term spectrum of the narrowband and wideband signals, respectively, and F[] represents the mapping (or estimation) function.

According to the state space model, the mapping function F[] can be described by two processes of the state evolution function F _state [] and the observation function F _obs [], as shown in the following formula,

S _hidden (t,f)＝F _state [S _hidden (t-1,f),S _X (t-1,f),N ₁ (t,f)]

S _Y (t,f)＝F _obs [S _hidden (t,f),S _X (t,f),N ₂ (t,f)]

Among them, S _hidden (t, f) is the hidden state vector in the model, and N ₁ (t, f) and N ₂ (t, f) respectively describe the errors of the state evolution function F _state and the observation function F _obs . In the above model, the hidden state vector S _hidden (t, f) at the current moment is composed of the hidden state vector S _hidden (t-1, f) at the previous moment and the short-term spectrum S _X (t-1, f) of the narrowband signal at the previous moment f) and the short-time spectrum S _Y (t, f) of the broadband signal at the current moment is further determined by the hidden state vector S _hidden (t, f) at the current moment and the short-time spectrum S _X (t, f) of the narrowband signal at the current moment )Decide. Using the hidden state recursive structure contained in the state-space model can more accurately fit the complex mapping relationship between narrow-bandwidth spectral parameters. The model can be implemented by generalized Kalman filter method, or by two independent deep neural networks. to fulfill. The basic principle of the state space model based on deep neural network is shown in Fig. 5. Here, the state evolution function F _state and the observation function F _obs can be implemented by various forward and recursive deep neural networks such as stacked autoencoders, multi-layer perceptrons, delayed recurrent networks, and long-short-term memory networks.

4. Stereo signal synthesis

The diffuse sound S”(t, f) and the direct point sound source can be respectively analyzed by using the mono frequency band extension method 2,...,N to get the corresponding broadband spectrum S _Y (t, f). Next, the direction information θ _i of each point sound source can be used to reproduce the broadband direct sound

in, is the extended broadband spectrum of the point sound source. is the short-term spectrum of the extended broadband direct sound. Ultimately, combined with the extended broadband diffuse sound wideband stereo signal possible reappearance,

In the embodiment of the present invention, firstly, the input stereo signal is decomposed into two components of direct sound and diffuse sound by utilizing the spectral correlation between channels, and then the diffuse sound component is directly expanded by using the traditional frequency band expansion method; According to the sparsity of the frequency structure, it is separated into multiple point sound sources with different orientations, and the bandwidth is expanded respectively. Finally, the expanded point sound sources are remixed according to their orientation information in the original stereo, and combined with the diffuse sound after bandwidth expansion. components to reconstruct a wideband stereo audio signal. The present invention solves the problem of expanding the signal bandwidth only based on the subjective quality of the reconstructed signal of a single channel in the prior art, without considering the correlation of signal energy and phase in the two channels, and the reconstructed stereo signal seriously affects the listener The problem of judging the location and distance of the sound source.

The above description is only an embodiment of the present invention, and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and changes. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the scope of the claims of the present invention.

Claims (10)

1. A method of bandwidth extension for stereo audio, comprising:

decomposing the stereo signal into direct sound and diffuse sound;

performing bandwidth extension on the diffused sound according to a preset frequency band extension method;

separating the direct sound into a plurality of point sound sources in different directions, and respectively performing bandwidth expansion on the plurality of point sound sources to obtain a plurality of point sound sources with expanded bandwidths;

and remixing the plurality of point sound sources after bandwidth expansion according to the pre-estimated azimuth information to obtain direct sound after bandwidth expansion, and reconstructing a broadband stereo audio signal according to the direct sound after bandwidth expansion and the diffuse sound after bandwidth expansion.

2. The method of bandwidth extension of stereo audio according to claim 1, wherein said decomposing the stereo signal into direct sound and diffuse sound comprises:

decomposing the stereo signal into a left channel and a right channel;

respectively carrying out time-frequency transformation on the left channel and the right channel subjected to the framing processing to obtain a left channel short-time frequency spectrum component and a right channel short-time frequency spectrum component of the stereo signal;

respectively obtaining the sum P between the energy spectrums of the left and right sound channel signals according to the left and right sound channel short-time frequency spectrum components_sumThe difference P between the energy spectra of the left and right channel signals_diffCross-correlation P between the energy spectra of the left and right channel signals_cc；

Using said P_sum、P_diffAnd P_ccObtaining a direct sound matrix through a least square method;

separating direct sound from the stereo signal using the direct sound matrix;

and eliminating the direct sound from the stereo signal to obtain diffuse sound.

3. The method of bandwidth extension for stereo audio according to claim 1,

the separating the direct sound into a plurality of point sound sources of different azimuths comprises:

calculating the direction information of direct sound on each time frequency point, clustering the direction information of all the time frequency points to obtain a clustering center of the direction information, wherein the clustering center respectively corresponds to the direction information of each point sound source;

obtaining a masking matrix according to the direction information of the direct sound at a certain time frequency point and the clustering center of the direction information;

and separating the direct sound by using the masking matrix to obtain a plurality of point sound sources in different directions.

4. The method of bandwidth extension for stereo audio according to claim 1,

the bandwidth extension is respectively carried out on the plurality of point sound sources, and the bandwidth extension comprises the following steps:

respectively inputting a plurality of point sound sources into a preset state space model to fit the mapping relation between the short-time frequency spectrum of the narrow-band signal and the short-time frequency spectrum of the broadband signal, estimating the spectrum envelope of the high-frequency component of the short-time frequency spectrum of the broadband signal according to a preset error criterion, and combining the low-frequency spectrum envelope and the spectrum details after being expanded by adopting a proper spectrum repairing method to obtain the plurality of point sound sources after bandwidth expansion.

5. The method of bandwidth extension for stereo audio according to claim 4,

the fitting of the mapping relationship between the short-time spectrum of the narrowband signal and the short-time spectrum of the wideband signal in the state space model and the estimation of the spectrum envelope of the high-frequency component according to a preset error criterion comprises:

obtaining a hidden state vector in the preset state space model by using the hidden state vector at the previous moment and the short-time frequency spectrum of the narrowband signal at the previous moment;

and obtaining the short-time frequency spectrum of the broadband signal by using the hidden state vector in the preset state space model and the short-time frequency spectrum of the narrow-band signal at the current moment.

6. An apparatus for bandwidth extension of stereo audio, comprising: the system comprises a decomposition module, a diffuse sound expansion module, a direct sound separation and expansion module and a reconstruction module;

the decomposition module is used for decomposing the stereo signal into direct sound and diffuse sound;

the diffuse sound expansion module is used for performing bandwidth expansion on diffuse sound according to a preset frequency band expansion method;

the direct sound separation and expansion module is used for separating the direct sound into a plurality of point sound sources in different directions, and performing bandwidth expansion on the plurality of point sound sources respectively to obtain a plurality of point sound sources after bandwidth expansion;

and the reconstruction module is used for remixing the plurality of point sound sources after the bandwidth expansion according to the pre-estimated azimuth information to obtain direct sound after the bandwidth expansion, and reconstructing a broadband stereo audio signal according to the direct sound after the bandwidth expansion and the diffuse sound after the bandwidth expansion.

7. The apparatus of claim 6, wherein the decomposition module is specifically configured to:

decomposing the stereo signal into a left channel and a right channel;

respectively carrying out time-frequency transformation on the left channel and the right channel subjected to the framing processing to obtain a left channel short-time frequency spectrum component and a right channel short-time frequency spectrum component of the stereo signal;

respectively obtaining the sum P between the energy spectrums of the left and right sound channel signals according to the left and right sound channel short-time frequency spectrum components_sumThe difference P between the energy spectra of the left and right channel signals_diffCross-correlation P between the energy spectra of the left and right channel signals_cc；

Using said P_sum、P_diffAnd P_ccObtaining a direct sound matrix through a least square method;

separating direct sound from the stereo signal using the direct sound matrix;

and eliminating the direct sound from the stereo signal to obtain diffuse sound.

8. The stereo audio bandwidth extension device of claim 6, wherein the direct sound separation and extension module is specifically configured to:

calculating the direction information of direct sound on each time frequency point, clustering the direction information of all the time frequency points to obtain a clustering center of the direction information, wherein the clustering center respectively corresponds to the direction information of each point sound source;

obtaining a masking matrix according to the direction information of the direct sound at a certain time frequency point and the clustering center of the direction information;

and separating the direct sound by using the masking matrix to obtain a plurality of point sound sources in different directions.

9. The stereo audio bandwidth extension device of claim 6, wherein the direct sound separation and extension module is specifically configured to:

respectively inputting a plurality of point sound sources into a preset state space model to fit the mapping relation between the short-time frequency spectrum of the narrow-band signal and the short-time frequency spectrum of the broadband signal, estimating the spectrum envelope of the high-frequency component of the short-time frequency spectrum of the broadband signal according to a preset error criterion, and obtaining the direct sound after bandwidth expansion by combining the low-frequency spectrum envelope and the spectrum details after expansion by adopting a proper spectrum repairing method.

10. The stereo audio bandwidth extension device of claim 9, wherein the direct sound separation and extension module is specifically configured to:

obtaining a hidden state vector in the preset state space model by using the hidden state vector at the previous moment and the short-time frequency spectrum of the narrowband signal at the previous moment;

and obtaining the short-time frequency spectrum of the broadband signal by using the hidden state vector in the preset state space model and the short-time frequency spectrum of the narrow-band signal at the current moment.