CN106960672B - Bandwidth extension method and device for stereo audio - Google Patents

Abstract

The invention discloses a bandwidth extension method and device of stereo audio. The method comprises the following steps: decomposing the stereo signal into direct sound and diffuse sound; performing bandwidth extension on the diffused sound according to a preset frequency band extension method; separating direct sound into a plurality of point sound sources in different directions, and respectively performing bandwidth expansion on the plurality of point sound sources to obtain a plurality of point sound sources after bandwidth expansion; remixing the plurality of point sound sources after bandwidth expansion according to the pre-estimated azimuth information to obtain direct sound after bandwidth expansion; and reconstructing a broadband stereo audio signal according to the direct sound after bandwidth expansion and the diffuse sound after bandwidth expansion. By means of the technical scheme, the problem that in the prior art, signal bandwidth expansion is realized only according to subjective quality of a single channel reconstruction signal, correlation between signal energy and phases in two channels is not considered, and the reconstruction of stereo signals seriously influences judgment of a listener on the position and distance of a sound source is solved.

Description

Bandwidth expansion method and device for stereo audio

technical field

The invention relates to the application field of network technology, in particular to a method and device for bandwidth expansion of stereo audio.

Background technique

In digital audio signal processing technology, audio signals covering the entire frequency range of 20Hz to 20KHz perceivable by the human ear are usually called full-band audio. Such signals are mainly used for high-fidelity reproduction of music signals. The current audio instant messaging system cannot provide sufficient network transmission rate and terminal processing capacity, which will inevitably limit the effective bandwidth of the reconstructed signal, and prioritize the low-frequency components of the encoded audio signal, thereby improving the encoding efficiency of the audio communication system.

The traditional telephone voice communication system usually transmits narrow-band signals, and its frequency is distributed in the range of 300 to 3400 Hz, and the sampling rate is 8 kHz. Correlative subjective listening test results show that 91% of syllable intelligibility and 99% of sentence intelligibility are preserved in narrowband speech. However, compared with real speech, the naturalness and subjective quality of narrowband signals transmitted in actual calls are significantly reduced. Due to the absence of high-frequency components, narrow-band speech cannot well distinguish partial unvoiced or plosive sounds, and impairs its ability to describe speaker characteristics. In order to effectively overcome the shortcomings of narrowband audio, wideband audio is widely used in the field of telephone voice communication. Its effective bandwidth is extended to 50Hz ~ 7kHz, which better covers most of the spectrum that characterizes the important characteristics of voice signals, and achieves close to AM broadcasting. sound quality level. However, limited by many problems such as history, economy, technology, etc., it will take a long transition period for traditional fixed and mobile communications to fully realize the transition from narrowband to wideband audio.

As an effective audio enhancement method, the band extension method can artificially recover from the reconstructed wideband audio at the receiving end by analyzing the time-frequency characteristics of the original audio signal without changing the source coding and network transmission of the narrowband signal. The high-frequency components cut off by the encoding end are extracted, so as to achieve the purpose of enhancing the auditory quality of the reconstructed audio. For the hearing-impaired, the band extension method can further improve phonemic and semantic discrimination. In the past ten years, many research institutions and researchers have successively proposed many solutions for the frequency band extension of monophonic speech signals. These methods usually start from two aspects of spectrum envelope expansion and spectrum detail expansion, and then synthesize the high-frequency components of the signal. The principle is shown in Figure 1. Firstly, according to the principle of human auditory perception, the time-frequency features of the narrowband signal are extracted; then, the spectral envelope and energy of the high-frequency components are estimated with the help of the mapping relationship between the high-frequency and low-frequency features described by the side information or prior knowledge; At the same time, an appropriate spectral patching method is selected to spread the spectral details; finally, the effective reconstruction of the high-frequency components of the wideband audio signal is achieved by combining the expanded spectral envelope and spectral details.

For stereo audio, traditional frequency band expansion methods mostly reconstruct high-frequency components independently for two channels. These methods only expand the signal bandwidth according to the subjective quality of the reconstructed signal of a single channel, and do not consider the signals in the two channels. The correlation of energy and phase, which reconstructs the stereo signal seriously affects the listener's determination of the location and distance of the sound source.

SUMMARY OF THE INVENTION

In view of the above problems, the present invention provides a method and device for bandwidth expansion of stereo audio.

The bandwidth expansion method of stereo audio provided by the present invention comprises the following steps:

Decompose the stereo signal into direct sound and diffuse sound;

Bandwidth expansion is performed on the diffused sound according to a preset frequency band expansion method;

The direct sound is separated into a plurality of point sound sources with different orientations, and bandwidth expansion is performed on the plurality of point sound sources respectively, so as to obtain a plurality of point sound sources after the bandwidth expansion;

remixing the multiple point sound sources after the bandwidth expansion according to the pre-estimated azimuth information to obtain the direct sound after the bandwidth expansion;

A wideband stereo audio signal is reconstructed according to the bandwidth-expanded direct sound combined with the bandwidth-expanded diffused sound.

The invention also provides a bandwidth expansion device for stereo audio, comprising: a decomposition module, a diffused sound expansion module, a direct sound separation and expansion module, and a reconstruction module;

The decomposition module is used to decompose the stereo signal into direct sound and diffused sound;

The diffused sound expansion module is used to expand the bandwidth of the diffused sound according to a preset frequency band expansion method;

The direct sound separation and expansion module is used to separate the direct sound into a plurality of point sound sources with different orientations, and respectively perform bandwidth expansion on the plurality of point sound sources to obtain a plurality of point sound sources after bandwidth expansion;

The reconstruction module is used to remix the multiple point sound sources after the bandwidth expansion according to the pre-estimated azimuth information to obtain the direct sound after the bandwidth expansion, and combine the bandwidth expansion according to the direct sound after the bandwidth expansion. The diffused sound after reconstruction reconstructs a wideband stereo audio signal.

The beneficial effects of the present invention are as follows:

In the embodiment of the present invention, the input stereo signal is firstly decomposed into two components: direct sound and diffused sound by using the spectral correlation between channels, and then the diffused sound component is directly expanded by using the traditional frequency band expansion method; The sparseness of the frequency structure is separated into multiple point sound sources with different orientations, and the bandwidths are expanded respectively. Finally, the expanded point sound sources are remixed according to their orientation information in the original stereo, and combined with the diffused sound after bandwidth expansion. components to reconstruct a wideband stereo audio signal. The invention solves the problem that in the prior art, the expansion of the signal bandwidth is only realized according to the subjective quality of the reconstructed signal of a single channel, and the correlation between the signal energy and the phase in the two channels is not considered, and the reconstructed stereo signal seriously affects the listener. The problem of determining the position and distance of the sound source.

Description of drawings

Fig. 1 is the basic flow chart of the method for expanding the frequency band of a monophonic voice signal in the prior art;

Fig. 2 is the flow chart of the bandwidth expansion method of stereo audio of the method embodiment of the present invention;

3 is a schematic structural diagram of a bandwidth expansion device for stereo audio according to an apparatus embodiment of the present invention;

Fig. 4 is the principle block diagram of the bandwidth expansion method of the stereo audio of the example 1 of the present invention;

FIG. 5 is a schematic block diagram of a state space model based on a deep neural network in Example 1 of the present invention.

Detailed ways

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be more thoroughly understood, and will fully convey the scope of the present disclosure to those skilled in the art.

In order to solve the problem of expanding the signal bandwidth only based on the subjective quality of the reconstructed signal of a single channel in the prior art, the correlation between the signal energy and phase in the two channels is not considered, and the reconstructed stereo signal seriously affects the listener's perception of the sound. Regarding the determination of source location and distance, the present invention provides a method and device for bandwidth expansion of stereo audio. The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to illustrate the present invention, but not to limit the present invention.

According to the method embodiment of the present invention, a method for bandwidth expansion of stereo audio is provided. FIG. 1 is a flowchart of the method for bandwidth expansion of stereo audio according to the method embodiment of the present invention. As shown in FIG. 1 , according to the method embodiment of the present invention The bandwidth expansion method of stereo audio includes the following processing:

Step 201, decompose the stereo signal into direct sound and diffuse sound.

Specifically, step 201 includes the following steps:

decompose the stereo signal into left and right channels;

Perform time-frequency transformation on the left channel and right channel after frame division respectively, and obtain the left channel short-term spectral component and the right channel short-term spectral component of the stereo signal;

According to the short-term spectral components of the left channel and the short-term spectral components of the right channel, respectively, the sum P _sum between the left and right channel signal energy spectra, the difference P _diff between the left and right channel signal energy spectra, the left and right channels the cross-correlation P _cc between the signal energy spectra;

Utilize described P _sum , P _diff , and P _cc to obtain direct sound matrix by least squares method;

Utilize the direct sound matrix to separate the direct sound from the stereo signal;

The diffuse sound is obtained by subtracting the direct sound from the stereo signal.

More specifically, according to the short-term spectral components of the left channel and the short-term spectral components of the right channel, the sum P _sum between the left and right channel signal energy spectra and the difference between the left and right channel signal energy spectra are obtained. P _diff , the cross-correlation P _cc between the left and right channel signal energy spectra, including:

Using the left channel short-term spectral component _SL (t, f) and the right channel short-term spectral component _SR (t, f) according to the formula P _sum = | _SL (t, f)| ² + |S _R (t,f)| ² Calculate the sum P _sum between the left and right channel signal energy spectra;

Using the left channel short-term spectral component _SL (t,f) and the right channel short-term spectral component _SR (t,f) according to the formula P _diff =| _SL (t,f)| ² − |S _R (t,f)| ² Calculate the difference P _diff between the left and right channel signal energy spectra;

Using the left channel short-term spectral component _SL (t, f) and the right channel short-term spectral component _SR (t, f) according to the formula P _cc =R{ _SL (t,f)S _R ^* (t,f)} Calculate the cross-correlation P _cc between the left and right channel signal energy spectra, where R{} is the real part operation.

More specifically, the use of the direct sound matrix to separate the direct sound from the stereo signal includes:

Use the direct sound matrix MD (t, f) to separate the direct sound S'(t, f) from the stereo signal _S (t, f) according to formula 1;

S'(t, f) = M _D (t, f) [ _SL (t, f) S _R (t, f)] ^T formula 1.

Step 202, performing bandwidth expansion on the diffused sound according to a preset frequency band expansion method.

Specifically, step 202 directly utilizes the traditional frequency band extension method to perform bandwidth extension on the diffused sound, which is not described in detail in the present invention.

Step 203 , separating the direct sound into a plurality of point sound sources with different azimuths, and performing bandwidth expansion on the plurality of point sound sources respectively to obtain a plurality of point sound sources with expanded bandwidths.

Specifically, in step 203, the direct sound is separated into a plurality of point sound sources with different orientations, including:

Calculate the direction information of the direct sound at each time-frequency point, and cluster the direction information of all the time-frequency points to obtain a cluster center of the direction information, and the cluster centers respectively correspond to the direction information of each point sound source;

Obtain a masking matrix according to the direction information of the direct sound at a certain time-frequency point and the cluster center of the direction information;

The direct sound is separated by using the masking matrix to obtain a plurality of point sound sources in different directions.

Specifically, the bandwidth expansion of the multiple point sound sources includes:

Input multiple point sound sources into the preset state space model to fit the mapping relationship between the short-term spectrum of the narrowband signal and the short-term spectrum of the wideband signal, and analyze the short-term spectrum of the wideband signal according to the preset error criterion. The spectral envelope of the high-frequency components is estimated, and combined with the low-frequency spectral envelope and the spectral details expanded by appropriate spectral patching methods, multiple point sound sources with expanded bandwidth are obtained.

More specifically, in the state space model, the mapping relationship between the short-term spectrum of the narrowband signal and the short-term spectrum of the broadband signal is fitted, and the spectral envelope of the high-frequency component is performed according to a preset error criterion. Estimates, including:

Using the hidden state vector at the previous moment and the short-term spectrum of the narrowband signal at the previous moment, the hidden state vector in the state space model is obtained;

Using the hidden state vector in the state space model and the short-term spectrum of the narrowband signal at the current moment, the short-term spectrum of the wideband signal is obtained.

Step 204, remixing the multiple bandwidth-expanded point sound sources according to the preset azimuth information to obtain the direct sound after the bandwidth expansion, and reconstructing the diffused sound after combining with the bandwidth-expanded direct sound according to the bandwidth expansion. output wideband stereo audio signal.

Specifically, the pre-estimated orientation information is estimated and obtained according to the cluster center of the orientation information, and the estimation method is a conventional technical means in the field, which is not described in detail in the present invention.

Specifically, use formula 2 to reconstruct a wideband stereo audio signal according to the bandwidth-expanded direct sound combined with the bandwidth-expanded diffused sound;

表示宽带扩展后立体声信号的短时频谱；

表示宽带扩展后直达声的短时频谱；

表示带宽扩展后扩散声的短时频谱。In Equation 2,

Represents the short-term spectrum of the wideband extended stereo signal;

Represents the short-term spectrum of the direct sound after broadband expansion;

Represents the short-term spectrum of the diffuse sound after bandwidth expansion.

Corresponding to the method embodiment of the present invention, a device for bandwidth expansion of stereo audio is provided. FIG. 3 is a schematic structural diagram of the device for bandwidth expansion of stereo audio according to an embodiment of the device of the present invention. As shown in FIG. The stereo audio bandwidth expansion device of the embodiment includes: a decomposition module 30 , a diffused sound expansion module 32 , a direct sound separation and expansion module 34 , and a reconstruction module 36 .

Specifically, the decomposition module 30 is used to decompose the stereo signal into direct sound and diffused sound;

The diffused sound expansion module 32 is configured to perform bandwidth expansion on the diffused sound according to a preset frequency band expansion method;

The direct sound separation and expansion module 34 is used to separate the direct sound into a plurality of point sound sources with different orientations, and respectively perform bandwidth expansion on the plurality of point sound sources to obtain a plurality of point sound sources after the bandwidth expansion;

The reconstruction module 36 is used to remix the multiple point sound sources after the bandwidth expansion according to the pre-estimated azimuth information to obtain the direct sound after the bandwidth expansion, which is used for the direct sound after the bandwidth expansion. A wideband stereo audio signal is reconstructed by combining the diffused sound after bandwidth expansion.

The decomposition module 30 is specifically used for:

decompose the stereo signal into left and right channels;

Perform time-frequency transformation on the left channel and right channel after frame division respectively, and obtain the left channel short-term spectral component and the right channel short-term spectral component of the stereo signal;

According to the short-term spectral components of the left channel and the short-term spectral components of the right channel, respectively, the sum P _sum between the left and right channel signal energy spectra, the difference P _diff between the left and right channel signal energy spectra, the left and right channels the cross-correlation P _cc between the signal energy spectra;

Utilize described P _sum , P _diff , and P _cc to obtain direct sound matrix by least squares method;

Utilize the direct sound matrix to separate the direct sound from the stereo signal;

The diffuse sound is obtained by subtracting the direct sound from the stereo signal.

The direct sound separation and expansion module 34 is specifically used for:

Calculate the direction information of the direct sound at each time-frequency point, and cluster the direction information of all the time-frequency points to obtain a cluster center of the direction information, and the cluster centers respectively correspond to the direction information of each point sound source;

Obtain a masking matrix according to the direction information of the direct sound at a certain time-frequency point and the cluster center of the direction information;

The direct sound is separated by using the masking matrix to obtain a plurality of point sound sources in different directions.

The direct sound separation and expansion module 34 is specifically used for:

Input multiple point sound sources into the preset state space model to fit the mapping relationship between the short-term spectrum of the narrowband signal and the short-term spectrum of the wideband signal, and analyze the short-term spectrum of the wideband signal according to the preset error criterion. The spectral envelope of high-frequency components is estimated, combined with the low-frequency spectral envelope and the spectral details expanded by appropriate spectral patching methods, and finally the direct sound after bandwidth expansion is obtained.

In order to describe the technical solution of the present invention in more detail, Example 1 is given. FIG. 4 is a schematic block diagram of the bandwidth expansion method of stereo audio according to Example 1 of the present invention. As shown in FIG. 4 , a bandwidth expansion method of stereo audio includes the following step:

1. Direct sound/diffuse sound separation

The stereo expansion system proposed in this paper uses discrete Fourier transform or orthogonal mirror filter bank to convert the framed left and right channel audio signals to the frequency domain, and divides them into multiple uniform sub-bands according to the principle of human auditory perception or critical band. Then, the short-term spectrum S(t,f) of the input stereo signal can be expressed as S(t,f)=[ _SL (t,f)S _R (t,f)] ^T

Among them, t and f represent the time frame and subband number of the signal, respectively; _SL (t, f) and _SR (t, f) represent the short-term spectral components of the left and right channels of the stereo signal, respectively.

In order to effectively separate the direct sound and the diffuse sound, the system also needs to calculate the sum P _sum and the difference P _diff between the left and right channel signal energy spectra and the cross-correlation P _cc of the two channels, respectively.

P _sum = | _SL (t,f)| ² +|S _R (t,f)| ²

P _diff =|S _L (t,f)| ² -|S _R (t,f)| ²

P _cc =R{S _L (t,f)S _R ^* (t,f)}

Among them, R{} is the operation of taking the real part. In order to improve the stability of the separation algorithm, time smoothing is performed on the calculated P _sum , P _diff and P _cc respectively.

The direct sound components in the stereo left and right channels are highly correlated and can be represented as point sound source signals propagating from a certain direction. Accordingly, the system proposed in this paper uses a direct sound matrix to directly separate the direct sound component S'(t, f) from the original stereo binaural signal S(t, f), as shown in the following formula:

S'(t,f)=[S _L '(t,f)S _R '(t,f)] ^T =MD (t,f)[ _{S L} (t,f)S _R (t,f) ] ^T = M _D (t,f)S(t,f)

Among them, _SL '(t, f) and _SR '(t, f) represent the short-term spectral components of the left and right channels of the direct sound, respectively, and _MD (t, f) is the direct sound matrix. According to the literature [M Vinton, D McGrath, C Robinson, P Brown, next generationsurround decoding and upmixing for consumer ad professional applications. ^{AES57 th} International conference, USA, 2015], the direct sound matrix M _D (t, f) can be used The least squares method is obtained, so that the expected square error between the estimated direct sound component and the real component is minimized, namely

Then the direct sound matrix M _D (t, f) can be calculated by the following formula,

The diffuse sound component S”(t,f) can be expressed as the difference between the original stereo signal and the direct sound component,

S"(t,f)=S(t,f)-S'(t,f)

2. Sound source separation of direct sound components

According to S'(t,f)=[S _L '(t,f)S _R '(t,f)] ^T , the direction information θ of the direct sound S'(t,f) at a certain time-frequency point is obtained by formula 3 (t, f), the direction information θ(t, f) of the direct sound S'(t, f) at a certain time-frequency point is the same as the direction information θ _i of the point sound source;

The direction information θ(t, f) of all time-frequency points is clustered to obtain the cluster center C _i of the direction information, i=1, 2...N; these cluster centers correspond to each point sound source S ₁ (t ,f), S ₂ (t,f), S ₃ (t,f)…S _N (t,f) direction information θ ₁ , θ ₂ , θ ₃ …θ _N ;

According to the direction information θ(t, f) of the direct sound S'(t, f) at a certain time-frequency point and the cluster center C _i , the masking matrix m _i (t, f) is obtained;

Use the masking matrix m _i (t, f) to separate the direct sound S'(t, f) according to formula 4, and obtain the direct point sound source

3. Bandwidth expansion

接下来可以根据单声道频带扩展方法分别对扩散声S”(t,f)和直达点声源

进行独立的带宽扩展。According to the method described above, the diffuse sound component S"(t,f) and the direct sound component S'(t,f) are separated from the stereo signal respectively, and the direct sound component S'( t,f) separated into multiple point sources

Next, the diffuse sound S”(t,f) and the direct point sound source can be separately analyzed according to the monophonic frequency band extension method.

Perform independent bandwidth expansion.

In this paper, the state space model is used to directly fit the mapping relationship between the narrow-band spectral parameters, and in the actual expansion, the spectral envelope of the high-frequency components is estimated according to a certain error criterion.

S _Y (t,f)＝F[S _X (t,f)]

In the formula, S _X (t, f) and S _Y (t, f) represent the short-term spectrum of the narrowband and wideband signals, respectively, and F[] represents the mapping (or estimation) function.

According to the state space model, the mapping function F[] can be described by two processes, the state evolution function F _state [] and the observation function F _obs [], as shown in the following formula,

S _hidden (t,f)=F _state [S _hidden (t-1,f),S _X (t-1,f),N ₁ (t,f)]

S _Y (t, f) = F _obs [S _hidden (t, f), S _X (t, f), N ₂ (t, f)]

Among them, S _hidden (t, f) is the hidden state vector in the model, and N ₁ (t, f) and N ₂ (t, f) describe the error of the state evolution function F _state and the observation function F _obs , respectively. In the above model, the hidden state vector S _hidden (t, f) at the current moment is composed of the hidden state vector S _hidden (t-1, f) at the previous moment and the short-term spectrum S _X (t-1, f) of the narrowband signal at the previous moment. f), and the short-term spectrum S _Y (t, f) of the broadband signal at the current moment is further determined by the hidden state vector S _hidden (t, f) at the current moment and the short-term spectrum S _X (t, f) of the narrowband signal at the current moment )Decide. Using the hidden state recursive structure contained in the state space model can more accurately fit the complex mapping relationship between narrow-band spectrum parameters. The model can be implemented by the generalized Kalman filter method, or two independent deep neural networks can be used. to fulfill. The basic principle of state space model based on deep neural network is shown in Figure 5. Here, the state evolution function F _state and the observation function F _obs can be implemented by various forward and recursive deep neural networks such as stacked autoencoders, multilayer perceptrons, delayed recurrent networks, and long and short-term memory networks.

4. Stereo signal synthesis

2,…,N进行扩展，从而得到相应的宽带频谱S_Y(t,f)。接下来，可以利用各个点声源方向信息θ_i来重现宽带直达声

The monophonic frequency band extension method can be used for diffuse sound S”(t,f) and direct point sound source respectively.

2,...,N are expanded to obtain the corresponding broadband spectrum S _Y (t, f). Next, the broadband direct sound can be reproduced using the direction information θ _i of each point sound source

为扩展后的点声源宽带频谱。

为扩展后宽带直达声的短时频谱。最终，结合扩展后的宽带扩散声

可以实现宽带立体声信号

的重现，in,

is the expanded wideband spectrum of the point sound source.

is the short-term spectrum of the extended broadband direct sound. Finally, combined with the extended broadband diffuse sound

Wideband stereo signal can be realized

reappearance,

In the embodiment of the present invention, the input stereo signal is firstly decomposed into two components: direct sound and diffused sound by using the spectral correlation between channels, and then the diffused sound component is directly expanded by using the traditional frequency band expansion method; The sparseness of the frequency structure is separated into multiple point sound sources with different orientations, and the bandwidths are expanded respectively. Finally, the expanded point sound sources are remixed according to their orientation information in the original stereo, and combined with the diffused sound after bandwidth expansion. components to reconstruct a wideband stereo audio signal. The invention solves the problem that in the prior art, the expansion of the signal bandwidth is only realized according to the subjective quality of the reconstructed signal of a single channel, and the correlation between the signal energy and the phase in the two channels is not considered, and the reconstructed stereo signal seriously affects the listener. The problem of determining the position and distance of the sound source.

The above description is only an embodiment of the present invention, and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the scope of the claims of the present invention.

Claims (10)

1. A method of bandwidth extension for stereo audio, comprising:

decomposing the stereo signal into direct sound and diffuse sound;

performing bandwidth extension on the diffused sound according to a preset frequency band extension method;

separating the direct sound into a plurality of point sound sources in different directions, and respectively performing bandwidth expansion on the plurality of point sound sources to obtain a plurality of point sound sources with expanded bandwidths;

and remixing the plurality of point sound sources after bandwidth expansion according to the pre-estimated azimuth information to obtain direct sound after bandwidth expansion, and reconstructing a broadband stereo audio signal according to the direct sound after bandwidth expansion and the diffuse sound after bandwidth expansion.

2. The method of bandwidth extension of stereo audio according to claim 1, wherein said decomposing the stereo signal into direct sound and diffuse sound comprises:

decomposing the stereo signal into a left channel and a right channel;

respectively carrying out time-frequency transformation on the left channel and the right channel subjected to the framing processing to obtain a left channel short-time frequency spectrum component and a right channel short-time frequency spectrum component of the stereo signal;

respectively obtaining the sum P between the energy spectrums of the left and right sound channel signals according to the left and right sound channel short-time frequency spectrum components_sumThe difference P between the energy spectra of the left and right channel signals_diffCross-correlation P between the energy spectra of the left and right channel signals_cc；

Using said P_sum、P_diffAnd P_ccBy a minimum of twoMultiplying to obtain a direct sound matrix;

separating direct sound from the stereo signal using the direct sound matrix;

and eliminating the direct sound from the stereo signal to obtain diffuse sound.

3. The method of bandwidth extension for stereo audio according to claim 1,

the separating the direct sound into a plurality of point sound sources of different azimuths comprises:

calculating the direction information of direct sound on each time frequency point, clustering the direction information of all the time frequency points to obtain a clustering center of the direction information, wherein the clustering center respectively corresponds to the direction information of each point sound source;

obtaining a masking matrix according to the direction information of the direct sound at a certain time frequency point and the clustering center of the direction information;

and separating the direct sound by using the masking matrix to obtain a plurality of point sound sources in different directions.

4. The method of bandwidth extension for stereo audio according to claim 1,

the bandwidth extension is respectively carried out on the plurality of point sound sources, and the bandwidth extension comprises the following steps:

respectively inputting a plurality of point sound sources into a preset state space model to fit the mapping relation between the short-time frequency spectrum of the narrow-band signal and the short-time frequency spectrum of the broadband signal, estimating the spectrum envelope of the high-frequency component of the short-time frequency spectrum of the broadband signal according to a preset error criterion, and combining the low-frequency spectrum envelope and the spectrum details after being expanded by adopting a proper spectrum repairing method to obtain the plurality of point sound sources after bandwidth expansion.

5. The method of bandwidth extension for stereo audio according to claim 4,

the method comprises the following steps of fitting a mapping relation between a short-time frequency spectrum of a narrow-band signal and a short-time frequency spectrum of a wide-band signal in the state space model, and estimating a frequency spectrum envelope of a high-frequency component according to a preset error criterion, wherein the method comprises the following steps:

obtaining a hidden state vector in the state space model by using the hidden state vector at the previous moment and the short-time frequency spectrum of the narrowband signal at the previous moment;

and obtaining the short-time frequency spectrum of the broadband signal by using the hidden state vector in the state space model and the short-time frequency spectrum of the narrow-band signal at the current moment.

6. An apparatus for bandwidth extension of stereo audio, comprising: the system comprises a decomposition module, a diffuse sound expansion module, a direct sound separation and expansion module and a reconstruction module;

the decomposition module is used for decomposing the stereo signal into direct sound and diffuse sound;

the diffuse sound expansion module is used for performing bandwidth expansion on diffuse sound according to a preset frequency band expansion method;

the direct sound separation and expansion module is used for separating the direct sound into a plurality of point sound sources in different directions, and performing bandwidth expansion on the plurality of point sound sources respectively to obtain a plurality of point sound sources after bandwidth expansion;

and the reconstruction module is used for remixing the plurality of point sound sources after the bandwidth expansion according to the pre-estimated azimuth information to obtain direct sound after the bandwidth expansion, and reconstructing a broadband stereo audio signal according to the direct sound after the bandwidth expansion and the diffuse sound after the bandwidth expansion.

7. The apparatus of claim 6, wherein the decomposition module is specifically configured to:

decomposing the stereo signal into a left channel and a right channel;

respectively carrying out time-frequency transformation on the left channel and the right channel subjected to the framing processing to obtain a left channel short-time frequency spectrum component and a right channel short-time frequency spectrum component of the stereo signal;

obtaining the energy spectrum of the left and right sound channel signals according to the left and right sound channel short-time frequency spectrum components respectivelyMeta and P_sumThe difference P between the energy spectra of the left and right channel signals_diffCross-correlation P between the energy spectra of the left and right channel signals_cc；

Using said P_sum、P_diffAnd P_ccObtaining a direct sound matrix through a least square method;

separating direct sound from the stereo signal using the direct sound matrix;

and eliminating the direct sound from the stereo signal to obtain diffuse sound.

8. The stereo audio bandwidth extension device of claim 6, wherein the direct sound separation and extension module is specifically configured to:

calculating the direction information of direct sound on each time frequency point, clustering the direction information of all the time frequency points to obtain a clustering center of the direction information, wherein the clustering center respectively corresponds to the direction information of each point sound source;

obtaining a masking matrix according to the direction information of the direct sound at a certain time frequency point and the clustering center of the direction information;

and separating the direct sound by using the masking matrix to obtain a plurality of point sound sources in different directions.

9. The stereo audio bandwidth extension device of claim 6, wherein the direct sound separation and extension module is specifically configured to:

respectively inputting a plurality of point sound sources into a preset state space model to fit the mapping relation between the short-time frequency spectrum of the narrow-band signal and the short-time frequency spectrum of the broadband signal, estimating the spectrum envelope of the high-frequency component of the short-time frequency spectrum of the broadband signal according to a preset error criterion, and combining the low-frequency spectrum envelope and the spectrum details after being expanded by adopting a proper spectrum repairing method to obtain the plurality of point sound sources after bandwidth expansion.

10. The stereo audio bandwidth extension device of claim 9, wherein the direct sound separation and extension module is specifically configured to:

obtaining a hidden state vector in the state space model by using the hidden state vector at the previous moment and the short-time frequency spectrum of the narrowband signal at the previous moment;

and obtaining the short-time frequency spectrum of the broadband signal by using the hidden state vector in the state space model and the short-time frequency spectrum of the narrow-band signal at the current moment.

Images