CN114495971A - A Speech Enhancement Method Using Embedded Hardware to Run Neural Networks - Google Patents

Abstract

The invention discloses a speech enhancement method using embedded hardware to run a neural network, and relates to the field of speech signal processing. ;Construct the R‑CED neural network with the logic unit of FPGA, and obtain the R‑CED neural network digital logic subsystem; denoise the spectrogram data through the R‑CED neural network digital logic subsystem; use the FPGA to denoise the denoised The spectrogram data is restored in time domain to obtain speech enhancement data. The invention is based on an embedded hardware platform, constructs a neural network through FPGA, makes full use of the parallelism of the FPGA digital logic gate array, and greatly improves the processing speed compared with the neural network operation mode based on GPU, CPU and other processors, and guarantees voice Enhance real-time processing.

Description

A Speech Enhancement Method Using Embedded Hardware to Run Neural Networks

technical field

The invention relates to the field of speech signal processing, in particular to a speech enhancement method using embedded hardware to run a neural network.

Background technique

Speech enhancement technology refers to the technology that when the pure target speech signal is disturbed or even submerged by one or more kinds of noise in a complex environment, the influence of noise is suppressed and reduced by a certain noise reduction algorithm, and the pure target speech is extracted as much as possible. It is widely used in mobile communications, human-computer interaction, military communications and other fields to eliminate and reduce the negative effects of various noises.

With the development of Internet of Things technology, voice processing equipment is developing rapidly in the direction of intelligence and terminalization, and voice enhancement technology is widely used in hardware platforms. However, the cloud computing model in the Internet of Things technology is not applicable to terminal devices due to the large use of network bandwidth and the inability to provide real-time feedback. In order to supplement the disadvantages of cloud computing, the edge computing model came into being.

Edge computing chooses to distribute computing tasks to lightweight devices close to the data source, directly collect and operate some data locally, and feed back to users in real time. With the improvement of the semiconductor manufacturing technology level, semi-custom integrated circuit chips such as FPGA (Field Programmable Gate Array, FPGA), and system on chip (system on chip, SoC) FPGA provide application scenarios for edge computing. Although such embedded devices have the advantages of local acquisition and local computing, the limitations of their transmission bandwidth, storage resources and computing resources also hinder the development of their large-scale applications.

Existing speech enhancement algorithms are usually based on machine learning techniques, such as Generative Adversarial Networks (GAN), GAN with autoencoder structure, and network models such as Long Short-Term Memory (LSTM). Most of these algorithms use neural network models with different structures and deep layers, which exchange higher computing costs for some performance improvements, making these complex neural networks difficult to implement on resource-limited hardware platforms.

SUMMARY OF THE INVENTION

In view of the above deficiencies in the prior art, the present invention provides a voice enhancement method using embedded hardware to run a neural network, which solves the problem that the current neural network-based voice enhancement system is difficult to implement on an embedded hardware platform with limited resources.

In order to achieve the above-mentioned purpose of the invention, the technical scheme adopted in the present invention is:

A speech enhancement method using embedded hardware to run a neural network, comprising the following steps:

S1. Collect voice data through a voice sensor, and perform Fourier transform on the voice data through an FPGA to obtain spectrogram data;

S2. Construct the R-CED neural network by using the logic unit of the FPGA, and obtain the R-CED neural network digital logic subsystem;

S3, denoise the spectrogram data through the R-CED neural network digital logic subsystem;

S4, performing time domain restoration on the noise-reduced spectrogram data through the FPGA to obtain speech enhancement data.

Further, in the step S1, Fourier transform is performed on the voice data through the programmable logic PL end of the Zynq7020 type hardware platform FPGA;

In described step S2, adopt the logic unit in the programmable logic PL end of Zynq7020 hardware platform FPGA to construct R-CED neural network;

In the step S4, time domain restoration is performed on the noise-reduced spectrogram data through the PS end of the processor system of the Zynq7020 hardware platform FPGA.

Further, the step S2 includes the following sub-steps:

S21. Construct a neural network convolution module by using the logic unit in the programmable logic PL end of the Zynq7020 hardware platform FPGA;

S22. Build an R-CED neural network digital logic subsystem through a neural network convolution unit;

S23, using the logic unit in the programmable logic PL end of the Zynq7020 hardware platform FPGA to construct a neural network convolution parameter storage module;

S24, store the parameters of each convolution kernel module in the R-CED neural network digital logic subsystem through the neural network convolution parameter storage module.

Further, the neural network convolution module constructed in the step S21 includes: a shift register, at least one multiplication group unit, a convolution control unit and an accumulation unit;

The shift register is used to move the input spectrogram data and the convolution operation weight parameters stored in the neural network convolution parameter storage module to the convolution operation module through a shift operation according to the FPGA machine clock cycle;

The multiplication group unit is used for multiplying the input spectrogram data and the weight parameter of the convolution operation to obtain the result of the convolution operation;

The convolution control unit is used to perform timing control on the shift register, the multiplication group unit and the accumulation unit through the preset convolution control finite state machine, so as to realize the convolution operation;

The accumulating unit is used for accumulating the operation results of each convolution control unit.

Further, the R-CED neural network digital logic subsystem also includes an input spectrogram data filling module, which is used for using a full filling method to perform convolution operations on the input spectrogram data by the neural network convolution module. Padding fills 0 operation.

The beneficial effects of the present invention are:

1) The present invention is based on an embedded hardware platform, constructs a neural network through an FPGA, makes full use of the parallelism of the FPGA digital logic gate array, and greatly improves the processing speed compared to the neural network operation mode based on processors such as GPU and CPU, guaranteeing Real-time speech enhancement processing.

2) Using the Zynq7020 hardware platform FPGA with built-in programmable logic PL side and processor system PS side, making full use of programmable logic PL to easily build digital logic modules for parallel data processing and processor system PS to easily execute complex serial program instructions Differential characteristics, the division of labor performs the operation of neural network noise reduction and frequency domain signal conversion to time domain.

3) According to the characteristics of Zynq7020 hardware platform FPGA, the basic structure of R-CED neural network-neural network convolution module is designed, and the multi-loop serial operation of convolution is converted into a finite state machine unique to digital logic circuits. The controlled pipeline parallel operation effectively improves the processing speed of the R-CED neural network.

Description of drawings

1 is a flowchart of a speech enhancement method using embedded hardware to run a neural network provided by an embodiment of the present invention;

2 is a schematic structural diagram of a Zynq7020 hardware platform FPGA according to an embodiment of the present invention;

3 is a schematic diagram of a software program implementation of a neural network convolution module;

4 is a structural diagram of a neural network convolution module provided by an embodiment of the present invention;

5 is a schematic diagram of the principle of a shift register provided by an embodiment of the present invention;

6 is a structural diagram of a row accumulation of an accumulation unit provided by an embodiment of the present invention;

7 is a structural diagram of a column accumulation of an accumulation unit provided by an embodiment of the present invention;

FIG. 8 is a layer state transition diagram of an embodiment of the present invention;

9 is an operation state transition diagram of an embodiment of the present invention;

FIG. 10 is a data transfer state transition diagram according to an embodiment of the present invention;

11 is a state transition diagram of a filling operation according to an embodiment of the present invention;

FIG. 12 is a schematic diagram of the principle of a neural network convolution parameter storage module provided by an embodiment of the present invention.

Detailed ways

The specific embodiments of the present invention are described below to facilitate those skilled in the art to understand the present invention, but it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, as long as various changes Such changes are obvious within the spirit and scope of the present invention as defined and determined by the appended claims, and all inventions and creations utilizing the inventive concept are within the scope of protection.

As shown in Figure 1, in one embodiment of the present invention, a speech enhancement method using embedded hardware to run a neural network includes the following steps:

S1. Fourier transform is performed on the speech data through the programmable logic PL end of the Zynq7020 hardware platform FPGA to obtain spectrogram data.

S2. The R-CED neural network is constructed by using the logic unit in the programmable logic PL end of the Zynq7020 hardware platform FPGA, and the R-CED neural network digital logic subsystem is obtained.

The structure of the Zynq7020 hardware platform FPGA in this embodiment is shown in Figure 2. The programmable logic PL side provides a logic gate array that can use a hardware description language (such as Verilog HDL) to control the connection relationship, and the processor system PS side provides runnable software program for the ARM processor. In FIG. 2 , the broken line indicates the flow of the control signal, and the solid line indicates the flow of the data.

The core of the R-CED neural network digital logic subsystem is the convolutional CNN implemented by FPGA. Step S2 of this embodiment includes the following sub-steps:

S21, using the logic unit in the programmable logic PL end of the Zynq7020 hardware platform FPGA to construct a neural network convolution module.

Usually, the neural network convolution module is implemented by a software program. The program is shown in Figure 3. The convolution operation can be regarded as the accumulation of multiplication. The operation includes output feature map channel loop, input feature map channel loop, input feature map loop and Convolution kernel loop.

The core idea of the present invention is to realize the parallel processing of the convolution operation by using the hardware-based digital logic gate circuit to achieve the effect of hardware acceleration.

According to the results of the parallelization analysis, in the convolution operation, the operations of each layer are serialized, and the single-layer operation is parallel, so the acceleration design is carried out for the single-layer operation.

Table 1 Parameters of each neural network convolution module

As shown in the pseudocode in Figure 3, the single-layer operation contains four nested loops. According to their different characteristics, different expansion strategies are adopted to speed up the convolution operation:

Convolution kernel cycle: All are expanded, and registers are used to store all feature map data and convolution kernel data required for a single convolution kernel operation; the parameters of each neural network convolution module are shown in Table 1.

Input feature map cycle: do not expand, because the data of the convolution operation in the system comes from the RAM on the FPGA chip, it is impossible to input multiple addresses at the same time to obtain the feature maps of multiple different convolution windows corresponding to the convolution kernel during the sliding process data;

Input feature map channel loop: Partially expand, divide the feature map and weight into 4 operation channels according to the channel, in the operation channel, the reading and operation of the input feature map are serial, while between the operation channels are parallel;

Output feature map channel loop: Partial expansion, the weight is divided into 4 parts according to the output feature map channel, comprehensively according to the input feature map channel loop expansion method, each part contains 4 computing channels, a total of 16 computing channels.

As shown in FIG. 4 , the neural network convolution module constructed in this embodiment includes: a shift register, at least one multiplication group unit, a convolution control unit, and an accumulation unit.

The shift register is used to move the input spectrogram data and the convolution operation weight parameters stored in the neural network convolution parameter storage module to the convolution operation module through the shift operation according to the FPGA machine clock cycle. Its working principle is shown in Figure 5.

If the maximum window of all convolutional layers is I × J, the number of shift registers is J-1, the depth is I, and the weight input is a single register. The shift register does not need to be full of I×J data, and the data can be transferred to the REG register when I×(J-1)+1. Figure 5 takes the maximum convolution window 5×5, the current convolution window 4×4, and the weight data 0-15 as an example. The dashed box is the position occupied by the current convolution window. As shown by the arrow, the weight data is stored in a single register, and each clock shift register will have the right address data moved to the left. The left address not only outputs data, but also stores it in the specified address of the previous shift register.

Each clock of the shift register outputs data to the REG register, and each clock inside the REG register moves the data in the direction of the arrow, instructing to obtain all the weight data sets of the convolution window at the effective position.

The multiplication group unit is used to multiply the input spectrogram data and the weight parameter of the convolution operation to obtain the result of the convolution operation.

The multiplication group unit in this embodiment adopts a pipeline design, and each clock outputs the product result of all elements in the convolution window.

The accumulation unit is used to accumulate the operation results of each convolution control unit. According to the hardware characteristics, the accumulation process is divided into a multi-level accumulation adder tree structure, and the intermediate results of each level of operation are cached in registers to form a pipeline structure.

The accumulation unit of this embodiment includes two parts, Add_col (row accumulation) and Add_row (column accumulation), as shown in FIG. 6 and FIG. 7 . The Add_col module accumulates and outputs the corresponding 4-channel data input in parallel and synchronously, and adopts the pipeline processing calculation without additional parameter configuration. The accumulated data of the Add_row module is serial input, and the first frame data is buffered in RAM. During the accumulation, read out the result of the previous accumulation in the RAM according to the address, and write the original address after the addition. After completing the accumulation of the last frame, add the bias offset value. After the data input is completed, a frame of output feature map is obtained and sent to the subsequent module.

The convolution control unit is used to perform sequential control on the shift register, the multiplication group unit and the accumulation unit through the preset convolution control finite state machine, so as to realize the convolution operation.

The convolution control unit contains two types of inputs: the overall controller command ctrl_cmd and the input flag signal flag_in. The output includes three types: parameter configuration signal config, output flag signal flag_out and read data signal Rd.

The convolution control state machine contains three nested state machines, the outermost layer is the layer state, followed by the operation state, and the innermost layer is the data handling state. The state machines and state descriptions are shown in Table 2.

The layer state is divided according to the convolution layer, the signal initialization is completed before entering the 0th layer operation, and the preparation state IDLE is added. The layer state machine transition diagram is shown in Figure 8.

The convolution operation related signal initialization is completed in the IDLE state, and the convolution operation from the 0th layer to the 15th layer is completed in the LAYER0-LAYER15 state.

Table 2 Composition and description of convolution control state machine

According to the output flag signals of the Conv_ctrl module, up_fm, up_w, get_fm, PS can determine when to input weight data and feature map data to the convolution operation module, and when to read data from the output buffer of the convolution module.

The operating state is shown in Figure 9. In the CONFIG state, the parameter configuration of related modules is mainly completed.

In the CAL state, determine the actual reading of the weight kernel feature map data, and record the number of convolution kernels that have completed operations.

In the DM state, a flag signal for data update is generated, and the number of convolution kernels that have been calculated is recorded.

The data transfer status is shown in Figure 10.

The data transfer state is a sub-state of the layer state DM. According to different data handling requirements, they are: waiting state WAIT, handling feature map state MOVE_M, handling weight and feature map state MOVE_MW, and handling next data state MOVE_NEXT.

S22. Build an R-CED neural network digital logic subsystem through a neural network convolution unit.

The R-CED neural network digital logic subsystem also includes an input spectrogram data padding module, which is used to use the full padding method to perform padding filling 0 operations during the convolution operation of the input spectrogram data by the neural network convolution module.

Populating the module performs three steps:

A1. Before writing the original feature map to OutputRAM, write the corresponding number of 0s into the reserved space;

A2. During the writing of the original feature map, after each line of data is written, write the corresponding number of 0s;

A3. After a feature map is written, continue to write the corresponding number of 0s. ,

The control of the filling module adopts the filling operation state machine, and its transition diagram is shown in Figure 11.

The fill operation state machine contains five states: IDLE, UP_ROW, MAP_ROW, PAD0, DOWN_ROW. The specific meaning of each state is shown in Table 3.

Table 3 Composition and description of filling operation state machine

S23, using the logic unit in the programmable logic PL end of the Zynq7020 hardware platform FPGA to construct a neural network convolution parameter storage module.

S24, store the parameters of each convolution kernel module in the R-CED neural network digital logic subsystem through the neural network convolution parameter storage module.

In this embodiment, the design of the neural network convolution parameter storage module is shown in FIG. 12 . For a 14-channel convolutional layer, FM_K _k (k=0,1,2,3,…) and W_C _c (c=0,1,2,3,…) represent the feature maps of the kth and c channels, respectively and weight data, since there are 16 channels in 4 RAMs, the redundant channels are filled with 0;

In the figure Convx1_x2(x1,x2=0,1,2,3) indicates that the input of the convolution module comes from the x2 channels of the x1 weight RAM and the x2 input feature map RAM, so there are a total of 16 convolution operations The modules work at the same time; the input feature map cache and handling scheme of each convolutional layer are shown in Table 4.

Table 4 The input feature map cache and handling scheme of each convolution module stored in the neural network convolution parameter storage module

S3. Noise reduction is performed on the spectrogram data through the R-CED neural network digital logic subsystem.

S4, performing time domain restoration on the noise-reduced spectrogram data through the processor system PS end of the Zynq7020 hardware platform FPGA to obtain speech enhancement data.

To sum up, the present invention has the following beneficial effects:

1) The present invention is based on an embedded hardware platform, constructs a neural network through an FPGA, makes full use of the parallelism of the FPGA digital logic gate array, and greatly improves the processing speed compared to the neural network operation mode based on processors such as GPU and CPU, guaranteeing Real-time speech enhancement processing.

2) Using the Zynq7020 hardware platform FPGA with built-in programmable logic PL side and processor system PS side, making full use of programmable logic PL to easily build digital logic modules for parallel data processing and processor system PS to easily execute complex serial program instructions Differential characteristics, the division of labor performs the operation of neural network noise reduction and frequency domain signal conversion to time domain.

3) According to the characteristics of Zynq7020 hardware platform FPGA, the basic structure of R-CED neural network-neural network convolution module is designed, and the multi-loop serial operation of convolution is converted into a finite state machine unique to digital logic circuits. The controlled pipeline parallel operation effectively improves the processing speed of the R-CED neural network.

In the present invention, the principles and implementations of the present invention are described by using specific embodiments, and the descriptions of the above embodiments are only used to help understand the method and the core idea of the present invention; The idea of the invention will have changes in the specific implementation and application scope. To sum up, the content of this specification should not be construed as a limitation to the present invention.

Those of ordinary skill in the art will appreciate that the embodiments described herein are intended to assist readers in understanding the principles of the present invention, and it should be understood that the scope of protection of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations without departing from the essence of the present invention according to the technical teaching disclosed in the present invention, and these modifications and combinations still fall within the protection scope of the present invention.

Claims (5)

1. A speech enhancement method for operating a neural network by using embedded hardware is characterized by comprising the following steps:

s1, voice data are collected through a voice sensor, and Fourier transform is carried out on the voice data through an FPGA to obtain spectrogram data;

s2, constructing an R-CED neural network by adopting a logic unit of the FPGA to obtain an R-CED neural network digital logic subsystem;

s3, denoising the spectrogram data through an R-CED neural network digital logic subsystem;

and S4, performing time domain restoration on the denoised spectrogram data through the FPGA to obtain voice enhancement data.

2. The speech enhancement method using embedded hardware to run neural network according to claim 1, wherein in step S1, the speech data is fourier transformed by the programmable logic PL port of the Zynq7020 type hardware platform FPGA;

in the step S2, an R-CED neural network is constructed by adopting logic units in a programmable logic PL end of a Zynq7020 type hardware platform FPGA;

in the step S4, the processor system PS end of the Zynq7020 type hardware platform FPGA performs time domain restoration on the denoised spectrogram data.

3. The speech enhancement method for operating a neural network with embedded hardware according to claim 2, wherein the step S2 comprises the following sub-steps:

s21, constructing a neural network convolution module by adopting a logic unit in a programmable logic PL (programmable logic) end of a Zynq7020 type hardware platform FPGA;

s22, building an R-CED neural network digital logic subsystem through a neural network convolution unit;

s23, constructing a neural network convolution parameter storage module by adopting a logic unit in a programmable logic PL (programmable logic) end of a Zynq7020 type hardware platform FPGA;

and S24, storing the parameters of each convolution kernel module in the R-CED neural network digital logic subsystem through the neural network convolution parameter storage module.

4. The speech enhancement method according to claim 3, wherein the neural network convolution module constructed in step S21 comprises: the device comprises a shift register, at least one multiplication group unit, a convolution control unit and an accumulation unit;

the shift register is used for moving input spectrogram data and convolution operation weight parameters stored in the neural network convolution parameter storage module to the convolution operation module according to the clock period of the FPGA machine through shift operation;

the multiplication group unit is used for carrying out multiplication operation on the input spectrogram data and the convolution operation weight parameters to obtain a convolution operation result;

the convolution control unit is used for controlling the time sequence of the shift register, the multiplication group unit and the accumulation unit through a preset convolution control finite state machine so as to realize convolution operation;

and the accumulation unit is used for accumulating the operation results of all the convolution control units.

5. The speech enhancement method for operating a neural network by using embedded hardware as claimed in claim 3, wherein the R-CED neural network digital logic subsystem further comprises an input spectrogram data padding module for performing padding 0 operation in the convolution operation of the input spectrogram data by the neural network convolution module in a full padding manner.

Images