CN111563584A - A method for splitting a neural network model and related products - Google Patents

Abstract

The present disclosure discloses a method for splitting a neural network model and related products. In this solution, an operator is split into multiple sub-operators with smaller scales, so that the calculation library under the single-core architecture can be directly called, avoiding the need for new Additional work to achieve.

Description

A method for splitting a neural network model and related products

technical field

Embodiments of the present disclosure relate to a method for splitting a neural network model and related products.

Background technique

In recent years, deep learning accelerators have been continuously proposed, and like general-purpose processors, they are expanding from single-core to multi-core. This extended multi-core structure can support data parallelism in the training phase to improve data throughput and speed up training. However, in the inference phase, deep neural networks have higher requirements on end-to-end latency than throughput, which often determines the availability of accelerators in certain scenarios. Traditional data parallel solutions cannot meet the requirements of accelerators for small data and low latency in inference scenarios.

SUMMARY OF THE INVENTION

In order to solve the above-mentioned technical problems, the present disclosure proposes a method for splitting a neural network model and related products.

To achieve the above object, the present disclosure provides a method for splitting a neural network model, wherein the method includes:

According to the operator of the target layer in the neural network model, the split state set of the tensor data associated with the operator of the target layer is determined; wherein the target layer is at least one layer in the neural network model ;

Traverse the split state set according to the directed acyclic graph of the neural network model, and determine the state paths and the weights of the state paths between adjacent split state sets; wherein, the state path represents the value of the operator Splitting method; each state in the split state set represents a sub-tensor data set, and the union result of all sub-tensor data of the state is the tensor data;

Determine the target splitting path of the target layer according to the weight of the state path;

The operator of the target layer of the neural network model is split using the target splitting path.

Preferably, the step of determining the target splitting path of the target layer includes:

Traverse all split state sets of the target layer, traverse each state for the current split state set, and obtain all state paths pointing to the current state and the input tensors from the starting state of the state path to the target layer The split path of the starting state of the data;

The splitting path from the current state to the starting state of the input tensor data of the target layer is determined according to the weight of the state path and the weight of the splitting path; wherein the weight of the splitting path is based on the weight of the splitting path. Determine the weights of all state paths corresponding to the split paths;

After traversing all the split state sets of the target layer, obtain the target split between the split state set of the input tensor data of the target layer and the split state set of the output tensor data of the target layer path.

Preferably, the step of determining the target splitting path of the target layer includes:

Traverse all split state sets of the target layer, traverse each state for the current split state set, and obtain all the state paths starting from the current state and the output sheets from the end state of the state path to the target layer. The split path of the terminal state of the volume data;

The splitting path from the current state to the terminal state of the output tensor data of the target layer is determined according to the weight of the state path and the weight of the splitting path; wherein, the weight of the splitting path is based on the The weights of all state paths corresponding to the split paths are determined;

After traversing all the split state sets of the target layer, obtain the target split between the split state set of the input tensor data of the target layer and the split state set of the output tensor data of the target layer path.

Preferably, the number of sub-operators obtained after the operator of the target layer of the neural network model is split is an integer power of 2.

Preferably, the state in the split state set of the input tensor data of the operator of the target layer of the neural network model is determined according to the calculation logic of the operator and the state in the split state set of the corresponding output tensor data .

Preferably, the state in the split state set of the output tensor data of the operator of the target layer of the neural network model is determined according to the calculation logic of the operator and the state in the split state set of the corresponding input tensor data .

Preferably, it also includes:

In the forward traversal stage, when the output tensor data of the operator is used as input tensor data by at least two operators, or the operator has at least two output tensor data, the output tensor data of the operator One split state remains in the set of split states of the volume data, and the split state is determined via the same state path of the operator.

Preferably, it also includes:

In the reverse traversal stage, when the operator has at least two input tensor data, one split state remains in the split state set of the input tensor data of the operator, and the split state is passed through the The same state path of the predicate operator is determined.

Preferably, the weight of the state path is determined according to the type and scale of the operator and the hardware parameters of the multi-core processor.

In order to achieve the above object, the present disclosure provides a neural network model splitting device, which includes:

A split state set module, configured to determine a split state set of tensor data associated with the operator of the target layer according to the operator of the target layer in the neural network model; wherein, the target layer is the at least one layer in the neural network model;

a state path module, configured to traverse the split state set according to the directed acyclic graph of the neural network model, and determine the state path and the weight of the state path between adjacent split state sets; wherein, the state path Represents the splitting method of the operator; each state in the split state set represents a sub-tensor data set, and the union result of all sub-tensor data of the state is the tensor data;

a target splitting path module, configured to determine the target splitting path of the target layer according to the weight of the state path;

A splitting module, configured to split the operator of the target layer of the neural network model by using the target splitting path.

Preferably, the target splitting path module includes:

The first traversal unit is used to traverse all the split state sets of the target layer, traverse each state for the current split state set, and obtain all the state paths pointing to the current state and the starting state of the state path to all the state paths. Describe the splitting path of the starting state of the input tensor data of the target layer;

a first splitting path determination unit, configured to determine a splitting path from the current state to the starting state of the input tensor data of the target layer according to the weight of the state path and the weight of the splitting path; wherein , the weight of the splitting path is determined according to the weights of all state paths corresponding to the splitting path;

The first selection target splitting path unit is used to obtain the split state set of the input tensor data of the target layer and the output tensor data of the target layer after traversing all the split state sets of the target layer The target split path between the set of split states.

Preferably, the target splitting path module includes:

The second traversal unit is used to traverse all the split state sets of the target layer, traverse each state for the current split state set, and obtain all the state paths starting from the current state and the end states of the state paths to The splitting path of the terminal state of the output tensor data of the target layer;

The second splitting path determination unit is configured to determine the splitting path from the current state to the terminal state of the output tensor data of the target layer according to the weight of the state path and the weight of the splitting path; wherein, The weight of the split path is determined according to the weight of all state paths corresponding to the split path;

The second selection target splitting path unit is used to obtain the split state set of the input tensor data of the target layer and the output tensor data of the target layer after traversing all the split state sets of the target layer The target split path between the set of split states.

Preferably, it also includes:

The first split state set optimization module is used in the forward traversal stage, when the output tensor data of the operator is used as input tensor data by at least two operators, or the operator has at least two output tensor data. When the amount of data is stored, one split state remains in the split state set of the output tensor data of the operator, and the split state is determined through the same state path of the operator.

Preferably, it also includes:

The second split state set optimization module is configured to, in the reverse traversal stage, when the operator has at least two input tensor data, retain one split state set of the input tensor data of the operator split states, and the split states are determined via the same state path of the operator.

In order to achieve the above object, the present disclosure provides a neural network model splitting hardware device, including a memory and a processor, where the memory stores a computer program that can run on the processor, wherein the processor executes the computer program. The program implements the steps of the method described above.

To achieve the above object, the present disclosure provides a computer-readable storage medium on which a computer program is stored, wherein, when the computer program is executed by a processor, the steps of the above-mentioned method are implemented.

The technical solution of the present disclosure can realize the expansion of the deep learning accelerator from a single core to a multi-core structure with a small overhead, and can provide an efficient splitting solution for a given network and underlying accelerator characteristics, and the solution can effectively Reduce the end-to-end latency of various networks on multi-core accelerators.

Description of drawings

In order to explain the technical solutions of the embodiments of the present disclosure more clearly, the accompanying drawings of the embodiments will be briefly introduced below. Obviously, the drawings in the following description only relate to some embodiments of the present disclosure, rather than limit the present disclosure. .

1 is a schematic diagram of a shared storage multi-core structure;

FIG. 2 is one of the flowcharts of a method for splitting a neural network model according to an embodiment of the present disclosure;

Figure 3 is a schematic diagram of the splitting of the serial neural network model;

FIG. 4 is the second flowchart of a method for splitting a neural network model provided by an embodiment of the present disclosure;

Fig. 5 is the split schematic diagram of introducing glue operator between operator and input tensor data;

6 is a schematic diagram of compensation;

FIG. 7 is the third flow chart of a method for splitting a neural network model provided by an embodiment of the present disclosure;

8 is a schematic diagram of pyramid splitting;

FIG. 9 is the fourth flowchart of a method for splitting a neural network model provided by an embodiment of the present disclosure;

FIG. 10 is a schematic diagram of a hardware device for splitting a neural network model according to an embodiment of the present disclosure.

Detailed ways

The technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the non-limiting example embodiments shown in the drawings and detailed in the following description to more fully describe the examples of the present disclosure. Examples and their various features and advantageous details. It should be noted that the features shown in the figures are not necessarily drawn to scale. The present disclosure omits descriptions of known materials, components, and process technologies so as not to obscure the example embodiments of the present disclosure. The given examples are only intended to facilitate the understanding of the implementation of the example embodiments of the present disclosure and to further enable those skilled in the art to practice the example embodiments. Accordingly, these examples should not be construed as limiting the scope of the embodiments of the present disclosure.

Unless otherwise specifically defined, technical or scientific terms used in the present disclosure shall have the ordinary meaning as understood by one of ordinary skill in the art to which this disclosure belongs. As used in this disclosure, "first," "second," and similar terms do not denote any order, quantity, or importance, but are merely used to distinguish the various components. Furthermore, in various embodiments of the present disclosure, the same or similar reference numerals refer to the same or similar components.

Specific implementations of a method for splitting a neural network model and related products provided by the embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

In recent years, thanks to the great success of deep learning itself in many fields, deep learning accelerators have become a rapidly developing field. These emerging accelerators often have a large advantage over GPUs in terms of performance per watt. Like the development of general-purpose processors, deep learning accelerators can also be extended from single-core to multi-core architectures, and this extension is very suitable for the data-parallel training method in deep learning. Data parallelism refers to speeding up training by dividing the training data set into several parts and using multiple processing cores to process a subset of the data sets separately. Adopting this approach in a multi-core architecture, each core processes different datasets in the training data in parallel, thereby increasing the throughput of the overall system and speeding up training. Therefore, the multi-core accelerator architecture can easily expand the computing throughput of the entire system in the training phase while maintaining a good performance-to-power ratio for each core.

For a multi-core processor structure chip, as shown in Figure 1, this shared memory multi-core structure is a classic multi-core structure. This structure is very suitable for data-parallel neural network training methods. Each core can be used as a processor in data parallelism to read different data and complete the forward and reverse computation of the neural network model in parallel. Each core can still maintain its good performance-to-power ratio under the previous single-core architecture in the computing stage, and at the same time, the throughput of the entire system can also increase with the expansion of the number of cores. The problem with data parallelism is that its scalability depends on the size of the data batches processed. Although this is usually not a problem during the training phase, this premise is harder to guarantee for the inference phase. Generally speaking, for neural network models used in real-time service fields (including video surveillance, autonomous driving, etc.), the processed data is usually input serially in a streaming manner, resulting in a small scale of data processed each time or even a single picture. In this case, data parallelism cannot provide any degree of parallelism, and all work tasks will be concentrated on a single core, which makes the computing resources brought by multi-core cannot be converted into the speed of processing tasks.

When the training of the neural network model is completed using the data set offline, the model will be deployed to the server in the cloud to process the data sent from the outside world. At this time, the application scenario changes from offline training to online inference. In the online inference stage, a very important indicator is the latency, that is, the time from the server receiving the data to be processed to returning the processed result, and further, the time to process the data using the neural network model. Low latency ensures that the cloud server can respond to the data sent by the client in the shortest time. In some more sensitive scenarios, it directly determines whether the solution is available. Therefore, the requirements for accelerators in the online inference stage have changed from processing large batches of data and high throughput to processing small batches of data and low latency.

In this case, traditional data parallelism or model parallelism cannot effectively reduce the delay of inference tasks. For data parallelism, large batches of data are the premise, which in itself contradicts the characteristics of online reasoning of small batches of data. For model parallelism, it is usually a method used to solve a large-scale neural network model that exceeds the memory limit of a single device. Distributing operators to different cores cannot reduce network latency. In order to truly reduce the delay of inference tasks on multi-core accelerators, we must find a way to reasonably allocate inference computing tasks for small batches of data or even single data to each core of the multi-core architecture, ensuring that every moment has Only when as many cores as possible participate in the calculation, the resources of the multi-core architecture can be fully utilized. One method is to split the computing task of each operator in the neural network into multiple cores for calculation. This method can ensure that there are multiple cores at each moment even when processing the reasoning task of a single image. Participate in the calculation, so as to achieve the purpose of using multi-core resources to reduce the delay.

However, for multi-core accelerators, there are still a lot of problems to solve. First of all, the deep learning accelerator adapts its own hardware design to adapt to the data parallel characteristics of the deep learning algorithm itself, and improves the computing throughput. The accelerator often needs enough data scale to achieve high computing efficiency, and further in the operator Splitting reduces the size of the computation on each core. When the split reaches a certain granularity, the loss of computational efficiency on each core will exceed the benefit of increased parallelism by splitting. Therefore, it is necessary to provide sufficient parallelism while ensuring sufficient computational efficiency between splitting parallelism and computational efficiency.

On the other hand, a neural network model can be viewed as a complex computational graph consisting of usually hundreds or even thousands of operators. The algorithm logic in different types of operators is different, which leads to different methods of splitting these operators. The splitting of each operator, in addition to balancing its own computing efficiency and parallelism, also needs to consider the collocation with the front and rear operators, and even the overall impact. The rapid development of deep learning has brought more and more large-scale and complex networks. It is unrealistic to find a good parallel method manually. Therefore, an automated method is needed to ensure that different networks can be used. A better split parallel strategy is given.

Another consideration is portability to the underlying accelerator. For accelerators without good enough programmability, the workload of modifying the software stack caused by scaling from a single core to multiple cores and implementing split parallelism inside the operator is very large. The traditional implementation of data parallelism and model parallelism is still based on one processing core to complete the calculation task of one operator, so it does not bring a lot of extra work, and the cross-core parallelism of a single operator needs to modify the implementation of the operator itself , the difficulty of this modification depends on the programmability of the accelerator and the complexity of the original operator implementation logic. How to reduce the extra overhead in the process of implementing low-latency inference on a multi-core architecture, ease the workload’s dependence on the programmability of the accelerator itself, and make the method more versatile for different multi-core accelerators in the future. A question to consider.

Based on the above analysis and description, an end-to-end splitting scheme is automatically provided for large-scale neural network models. This scheme splits an operator into multiple smaller sub-operators, so that the single-core architecture can be directly invoked. Computational library below, avoiding the extra effort of re-implementation. For example, an activation operator can be divided into many smaller activation operators, which means that it only needs to call the original single-core activation function on multiple cores to complete each subtask, without modifying or rewriting Implement a multicore version of the activation function. In this process, it is necessary to take into account not only the computational efficiency and parallelism of each operator after splitting, but also the mutual cooperation of context operators in splitting. The ultimate goal is to obtain a split parallel scheme that can effectively reduce the end-to-end inference delay of the entire neural network model.

Taking autonomous driving applications as an example, the vehicle needs to analyze and process external information such as images, videos, and voices transmitted by on-board sensors during the process of autonomous driving. In order to ensure safety, the vehicle must get the results of the processing in the shortest possible time to make decisions. Vehicles with multi-core processor structure chips can use this solution to distribute the computational load of the neural network model for processing small batches of external information to multiple processor cores in a balanced manner, complete the processing of the information within the specified response time, and return The processing result assists the automatic driving of the vehicle. The technical solution can realize the extension of the deep learning accelerator from a single core to a multi-core structure with a small overhead, and the solution can effectively reduce the end-to-end delay of various networks on the multi-core accelerator.

In the above application scenario, the multi-core processor structure chip is arranged on the vehicle. In practice, the multi-core processor structure chip can be set on the cloud server, and the vehicle can send external information such as images, videos, and voices from the on-board sensors to the cloud server through 3G/4G, WIFI and other networks. The cloud server uses this solution to evenly distribute the computational load of the neural network model for processing small batches of external information to multiple processing cores. Within the specified response time of the vehicle, the cloud server will feed back the processing results to the vehicle through 3G/4G, WIFI and other networks. In practice, the scale of external information collected by on-board sensors varies. Before application, according to external information of different scales, the on-board processor uses this scheme to determine the corresponding operator splitting path. The operator splitting schemes corresponding to external information of different scales are stored in the corresponding areas. After the multi-core processor structure chip obtains the external information, the corresponding operator splitting path is called to split the operators in the neural network model. The computational load of the information is distributed evenly across multiple processor cores.

Usually, the upper-layer framework needs to call the computing library to obtain the instruction implementation of each operator in the neural network model on the processor. Specifically, the framework informs the computing library of the type and parameters of each operator, and the computing library returns the machine instructions required for each operator to execute on the processor. The framework loads the data and the machine instructions onto the processor through the driver, starts the processor and completes the calculation of the operator.

If the computing platform of the operator is changed from a single-core accelerator to a multi-core accelerator with similar or even the same core structure, correspondingly, the computing library needs to be redesigned so that it can generate machine instructions running on multiple cores. Specifically, since multiple cores need to read different parts of the same input tensor data, and also need to write their respective outputs back to different parts of the same output tensor data, the computing library needs to modify the calculation instructions of each operator. All instructions about the read and store parts.

The neural network splitting method provided by the embodiments of the present disclosure can avoid modifying the computing library of the single-core processor as much as possible, and can also realize the parallel execution of the neural network model on the multi-core processor. Specifically, the upper-layer framework divides the operators in the neural network model into several sub-operators that can be executed in parallel, and for each sub-operator, the framework calls the computing library to generate machine instructions for the sub-operators to execute on a single core , and by loading the machine instructions of the sub-operators to different cores, the parallel computing of the operators on the multi-core processor is realized. Specifically, because the framework uses the single-core processor computing library to generate the calculation instructions of the sub-operators, the input and output tensor data of the operator in the neural network model are also split as the operator is split into sub-operators Divide the data into corresponding sub-tensors.

Based on the above description, as shown in FIG. 2 , an embodiment of the present disclosure provides a flowchart of a method for splitting a neural network model.

include:

Step 201): According to the operator of the target layer in the neural network model, determine the split state set of the tensor data associated with the operator of the target layer.

In this embodiment, the neural network model can generally be regarded as a directed acyclic graph composed of operators and multi-dimensional tensor data. The operators and tensor data are connected to each other through directed edges, and the directed The direction of the edge indicates whether the data is the input or output of the operator. Use op to represent operators and tensor to represent tensor data. At the same time, in order to unify the expression of the splitting methods of different operators, the framework uniformly chooses to use the splitting methods of tensor data associated with the operators to illustrate the splitting methods of different operators. It may be assumed that all tensor data in the network are 4-dimensional. For the input data or output data of the last fully connected layer and the normalized exponential regression layer of the image classification network, the actual dimension is less than 4, and it is still expressed as 4-dimensional tensor. These four dimensions are represented by the symbols N, C, H, and W, respectively. Among them, N represents the batch size, C represents the number of feature images, H represents the height of the feature images, and W represents the width of the feature images. This assumption is only for the convenience of illustration, for the framework itself, can support the processing of neural network models containing tensor data of any number of dimensions. Nonetheless, 4 dimensions are sufficient for a sizable portion of neural network structures.

When the present technical solution splits the operators in the neural network model, the types of operators are different, the calculation logics supported by the operators are also different, and there are also different split strategies. In order to uniformly express the splitting strategies of different operators, the technical solution adopts the splitting state of the input tensor data and output tensor data of the operator to represent the splitting of the computing logic of the operator itself.

For this technical solution, all operators in the entire neural network model can be split, and some operators in the neural network model can also be split. Moreover, the network structures and algorithms newly emerging in the field of deep learning have gradually blurred the physical meaning of each data dimension and the boundaries between each other, and this technical solution can be extended and applied to operator splitting in more dimensions.

Any split of the tensor data is called a state s of the tensor data. After the tensor data is split, a sub-tensor data set is obtained. The state s is represented by the corresponding sub-tensor data set. All possible splits {s ₀ , s ₁ , s ₂ ,…} constitute the split state set S of the tensor data. Generally speaking, this is a very large state space, which means that the tensor data consists of The space of possible splitting modes of the operator represented by the splitting state is also huge.

With some reasonable assumptions, the state set of tensor data can be pruned. First of all, the delay for a multi-core accelerator to complete the calculation of an operator depends on the time of the core that takes the longest to execute the subtask, and the cores in the multi-core architecture are equivalent to each other in terms of hardware structure, so each core’s The amount of time spent depends on the workload assigned to the core. Therefore, a reasonable assumption is that the scale of the split sub-operators should be roughly balanced. For this reason, those split states that are not balanced in splitting can be omitted from the state set S of the tensor data. In addition, the number of cores in a multi-core architecture is usually an integer power of 2, such as 1, 2, 4, 8, 16, etc. A task whose parallelism is not an integer power of 2 often results in "fragmentation" in the scheduling of the cores. Therefore, the number of sub-operators after splitting should be guaranteed to be an integer power of 2. With these two assumptions, the search space of operator splitting strategies can be greatly reduced.

It should be noted that not selecting the split state of any tensor data associated with an operator can represent an effective splitting method for the operator. The split dimension of tensor data should be supported by the operator. For example, the input data of the normalized exponential regression operator (Softmax) should not be split on the dimension to be normalized. In addition, the splitting of the input tensor and output tensor of the operator should satisfy the calculation logic of the operator. For example, the start and end points of each sub-block where the output data of the convolution operator is split in the H/W dimension should be It is indeed the sub-block whose corresponding input data is split in the H/W dimension calculated according to the convolution kernel and displacement step size of the convolution operator; the input data of the convolution operator is split in the C dimension. It is exactly the same as the splitting of the weight data in the C dimension, and the splitting of the output data in the C dimension should be exactly the same as the splitting of the weight data in the N dimension. In the framework, the input state of the operator is deduced backwards according to the specific logic of each operator using the output state, or the output state of the operator is deduced forwardly using the input state according to the specific logic of each operator. This ensures that the state of the relevant data can always represent an efficient operator split.

Step 202): Traverse the split state set according to the directed acyclic graph of the neural network model, and determine the state paths and the weights of the state paths between adjacent split state sets.

As shown in Figure 3, the splitting scheme P of the entire neural network model can be regarded as a splitting state in the splitting state set of the input tensor data of each operator to a splitting state in the output tensor jump. The split state of the output tensor of the previous operator is the split state of the input tensor of the latter operator. Every possible jump through an operator corresponds to an efficient split on that operator. Thus, the state path represents how the operator is split.

In this technical solution, the tensor data is decomposed according to a decomposition method to obtain a sub-tensor set, and the sub-tensor set corresponds to a split state, and different decomposition methods can obtain a variety of split states. The split state obtained by the method constitutes a split state set. It can be seen that each split state corresponds to a sub-tensor set, and the sub-tensor set contains all elements in the tensor data. And, in a set of sub-tensors, the elements of each sub-tensor may or may not overlap.

As described above, the state path represents the splitting mode of the operator, and the calculation logic of the operator is split through the splitting mode corresponding to the state path to obtain the corresponding sub-operator set. The state of the input tensor data and the state of the corresponding output tensor data are connected by a state path, and the sub-tensor data set representing a split state of the input tensor data is processed by the sub-operators in the sub-operator set to obtain the The set of sub-tensor data corresponding to the split state of the output tensor data.

In this technical solution, the weight of the state path is represented by the time it takes for the operator to execute in parallel on the multi-core accelerator in a certain split state, and the time for the multi-core accelerator to complete the computation of one operator depends on the time-consuming execution of the subtask The longest time for that nucleus. The weights of the state paths are estimated using parameters:

1) The calculation loads c ₁ , c ₂ , . . . , c _n of the n sub-operators after splitting. Among them, c _i is calculated according to the type and scale of the ith sub-operator after splitting;

2) The fetched data volumes d ₁ , d ₂ , . . . , d _n of n sub-operators. Among them, d _i is calculated according to the type and scale of the ith sub-operator after splitting;

3) The computational throughput rate α of each accelerator core. α is determined by the performance parameters of the accelerator itself;

4) The memory access bandwidth β of each core. In general, multiple cores share limited memory access bandwidth, so β=B/n. where B is the total bandwidth of the multi-core accelerator.

The formula for calculating the weight of the state path is:

t=max _i=1,...,n (max( _{ci/α, d i /} β)) Equation (1)

Among them, the inner maximum value operation is based on the fact that the calculation part and the memory access part implemented by the operator can be hidden from each other, that is, the calculation and the memory access can be performed concurrently as much as possible. For some accelerators, when the size of the sub-operator is too small, the computational throughput of each core will be reduced, and α can be further modified to make the estimation more accurate. The outer maximum value operation is that the time for the multi-core accelerator to complete the calculation of an operator depends on the time of the core that takes the longest time to execute the subtask.

It should be noted that the above method of obtaining the weight of the state path is only an example of some cases, rather than an exhaustive list. Those skilled in the art may understand the essence of the technical solution of the present application. Based on other deformations or transformations, for example, the weight of the state path can be measured not only by the time spent executing subtasks, but also by the throughput of executing subtasks. Alternatively, the weight of the state path can also be determined by actually measuring the time for executing all subtasks in the operator splitting mode corresponding to the state path on the multi-core processor. However, as long as the functions realized and the technical effects achieved are similar to those of the present application, they should all belong to the protection scope of the present application.

Step 203): Determine the target splitting path of the target layer according to the weight of the state path.

In step 203, the weight of the state path is used to determine the split path of the target layer in two ways. The first way is forward traversal to determine the split path. The steps include:

Traverse all split state sets of the target layer, traverse each state for the current split state set, and obtain all state paths pointing to the current state and the input tensors from the starting state of the state path to the target layer The split path of the starting state of the data;

Determine the splitting path from the current state to the starting state of the input tensor data of the target layer according to the state path and the splitting path;

The splitting path from the current state to the starting state of the input tensor data of the target layer is determined according to the weight of the state path and the weight of the splitting path; wherein the weight of the splitting path is based on the weight of the splitting path. Determine the weights of all state paths corresponding to the split paths;

After traversing all the split state sets of the target layer, obtain the target split between the split state set of the input tensor data of the target layer and the split state set of the output tensor data of the target layer path.

The second way is reverse traversal to determine the split path, the steps include:

Traverse all split state sets of the target layer, traverse each state for the current split state set, and obtain all the state paths starting from the current state and the output sheets from the end state of the state path to the target layer. The split path of the terminal state of the volume data;

The splitting path from the current state to the starting state of the input tensor data of the target layer is determined according to the weight of the state path and the weight of the splitting path; wherein the weight of the splitting path is based on the weight of the splitting path. Determine the weights of all state paths corresponding to the split paths;

After traversing all the split state sets of the target layer, obtain the target split between the split state set of the input tensor data of the target layer and the split state set of the output tensor data of the target layer path.

The following example describes in detail how to obtain the relationship between the split state set of the input tensor data of the target layer and the split state set of the output tensor data of the target layer after traversing all the split state sets of the target layer. target split path.

The neural network model shown in FIG. 3 is serial, and the input tensor data and output tensor data of the entire neural network model are in an unsplit state. Divide the operators of the entire neural network model in Figure 3, and describe a serial neural network model containing n operators as a sequence of operators (op ₁ , op ₂ ,..., op _n ), assuming that each Each operator has only one input and one output. The input of the previous operator is the output of the latter operator, so it includes the input tensor data and output tensor data of the entire neural network model and the intermediate results between all operators. Including tensors, all tensor data constitute a set (tensor ₀ , tensor ₁ ,..., tensor _n ), the input of op _i is tensor _i-1 , and the output is tensor _i . For each data tensor tensor _i , there is a corresponding state set S ⁱ . The goal of the search strategy is to find a mapping relationship between the tensor itself and a certain state of its state set tensor _i →s ⁱ , through A specific split state is determined for each tensor data in the neural network model, so that the split mode of all operators can be determined. Therefore, a mapping from all tensor data in a neural network model to its split state The relationship is called a split scheme P of the network model. In the calculation stage, the ith operator op _i calculates the output tensor data in the split state r with the input data in the split state s. The specific parallel calculation method is determined by the state of the input tensor data and the output tensor data. At the same time, the calculation time of this operator is recorded as t _s→r , and the size of its value depends on the corresponding splitting method and the hardware characteristics of the underlying accelerator, then the calculation formula of the delay T of the entire network is:

It also corresponds to the time t _i used to execute the operator in parallel on the multi-core accelerator using the split method. Therefore, t _i can be regarded as the state of the input tensor data of the operator pointing to the output tensor The weights of the directed edges of the state of the data. At the same time, as the input tensor data and output tensor data of the entire neural network model, there is only one state in the corresponding split state space that keeps the entire data block continuous and complete, which makes the splitting of the neural network model. Scheme P starts with complete input data and ends with complete output data, and external users always see a complete input and output. At this time, searching for a good splitting scheme P for a given neural network model is to find a shortest path from the unsplit state of the input tensor data to the unsplit state of the output tensor data. Each intermediate result tensor must select a state to pass through in the valid state space. Equation 3 and Equation 4 give this abstract formula.

Also note that in Figure 3, there is a situation where one split state of the input tensor data points to multiple split states of the output tensor data. This situation exists. Further, it is this situation that causes the neural network The network model splitting space is huge.

In this technical solution, the unsplit state of the input tensor data of the entire neural network model is set as the starting state s _root , and in the initial stage, the unsplit state of the input tensor data of the neural network model is the starting state s The weight of the splitting path corresponding to the _root is 0, and the weight of the corresponding splitting path of all states of the remaining tensor data is ∞. Any state s of any tensor data in the neural network model has a corresponding splitting path weight ls from the _root of s to _s . Visit each split state set from front to back, and in each split state set, traverse each state s in turn. For each state s, there are directed edges e ₁ ,...,e _ks pointing to several split states in the set of subsequent split states. Taking the split state v in the next split state set as an example, use equation (1) to obtain the weight t _sv between state s and state v, and use equation (5) to update the next split state pointed to by the state path The weight l _v of the split path from s _root to the state v corresponding to the state v in the set.

l _v =min(l _v ,l _s +t _sv ) (5)

After completing the access to all split state sets according to the topological relationship of the neural network model, the unsplit state s _root of the input tensor data of the entire neural network model is obtained to the unsplit state s root of the output tensor data of the neural network model. The target split path for state _send .

The above describes a path from the unsplit state s _root to the unsplit state _send through a state in each split state set, and this path is the split path of the neural network model. From the split paths of the neural network model, the one with the smallest weight is selected as the target split path of the neural network model.

It should be noted that the neural network model shown in FIG. 3 is a serial neural network model, and in order to facilitate the description of the technical solution, the input tensor data of the neural network model and the split state sets corresponding to the output tensor data are different. Split state. When the split state set of the output tensor data of the neural network model is not a non-split state _send , but a set composed of multiple split states, each of the split state sets of the output tensor data of the neural network model The minimum value is selected from the weights of the split paths of the split states as the target split between the split state set of the input tensor data of the entire neural network model and the split state set of the output tensor data of the neural network model path.

Note that the whole scheme can also be transformed to search for a split path from the unsplit state _send to the unsplit state s _root , and the two are equivalent. Similarly, when the split state set of the input tensor data of the neural network model is not the _unsplit state send, but a set composed of multiple split states, the split state set of the input tensor data of the neural network model Among the weights of the splitting paths of each splitting state, the minimum value is selected as the difference between the splitting state set of the input tensor data of the entire neural network model and the splitting state set of the output tensor data of the neural network model. Target split path.

The neural network model shown in FIG. 3 is a serial neural network model, and the state in the split state set of the input tensor data and the output tensor data of the model is not split. In practical applications, for the technical solutions shown in FIG. 2 , FIG. 4 , FIG. 7 and FIG. 9 , it is necessary to solve the problem of consistency of different branch splitting methods in the multi-branch neural network model. Operators located at the intersection of branches have more than one input tensor data, such as bitwise addition operator (Add), bitwise multiplication operator (Mult), and concatenation operator (Concat). For an operator A with two inputs, after accessing the operator, that is, enumerating the split state set of the input tensor data according to the split state set of the input tensor data, the two input tensor data tensor _left , tensor _right has corresponding split state sets S _left and S _right respectively. Continue to traverse along the two paths starting from tensor _left and tensor _right respectively. In one case, the two branches will extend directly until the end of the traversal, which means that the entire network has more than one input data, which is usually not used in inference tasks. Uncommon, in another case, two branches join together at some operator. In either case, when the splitting scheme P is determined, on the two input tensor data tensor _left and tensor _right of operator A, the split states that do not match each other may be selected. Specifically, assuming that operator A is a binary pairwise addition operator, the backtracking process may select a state that is split only in the C dimension in the split state set of tensor _left , and the split state of tensor _right may be selected in the backtracking process. The selected state in the sub-state set may be a state that only has splits in the H dimension. The splitting methods of the addition operators themselves represented by these two split states are inconsistent, so the entire splitting scheme P will be invalid. . In order to solve this problem, it is ensured that the split state sets corresponding to tensor _left and tensor _right only contain one split state before the end of traversal operator A, which ensures the certainty of the state selected in the two state sets during the backtracking process. Therefore, in the forward traversal phase, when the output tensor data of the operator is used as input tensor data by at least two operators, or the operator has at least two output tensor data, the operator's output tensor data is One split state remains in the set of split states of the output tensor data, and the split state is determined via the same state path of the operator. In the reverse traversal stage, when the operator has at least two input tensor data, one split state remains in the split state set of the input tensor data of the operator, and the split state is passed through the The same state path of the predicate operator is determined. In this way, before the end of the traversal of the branch operator, the state with the smallest corresponding cumulative weight is selected from the split state sets of multiple input data and retained, and the other split states in the split state set are removed.

It should be noted that the above method of obtaining the target splitting path is similar to the viterbi algorithm, and this time is only a partial example, not an exhaustive list. Those skilled in the art may understand the essence of the technical solution of the present application. Other deformations or transformations are generated on the basis of the technical solutions of the present application, such as: each split between the split state set of the input tensor data of the neural network model and the split state set of the output tensor data of the neural network model The weights of the sub-paths are determined by the sum of the weights of the corresponding state paths. A threshold is set according to experience. If the weight of the split path is less than the set threshold, the neural network model can be split as the target split path. However, as long as the functions realized and the technical effects achieved are similar to those of the present application, they should all belong to the protection scope of the present application.

Step 204): Use the target splitting path to split the operator of the target layer of the neural network model.

As can be seen from the above description, the hardware resources of the multi-core processor structure chip can be fully utilized by dividing the computing logic of the operator in the neural network into smaller subtasks and assigning them to multiple cores for parallel execution.

For the technical solution shown in Figure 2, under the most ideal conditions, it is hoped that the split sub-operators can write their output tensor data to the corresponding position in a storage space where the complete output tensor data is stored , after all sub-operators of such an operator are executed, a complete and continuous data block is always obtained. But for some accelerators, this is not easy to do. First, since the storage location of the output tensor data of the split operator may be discontinuous in the entire output, it is necessary to rewrite the code of the output part of the operator so that the output result can be written back to a sub-tensor The quantities are in corresponding discrete locations in storage. At the same time, the accelerator usually further adjusts the order of data in storage to improve the memory access efficiency during computation, which makes the work of modifying the output logic of the operator more difficult and cumbersome. At the same time, if the calculation logic or splitting logic of the subsequent operator does not require the input data to ensure storage continuity in a certain dimension, the data in the discrete storage state in the dimension output by the previous layer can be directly used for the next layer. calculation without guaranteeing the continuity of the output data.

Therefore, the framework separates the task of adjusting the split form of tensor data from the computing task of the operator, and abstracts it into a new operator, called the glue operator. This separation avoids the logic of the output of each operator. Modification to enhance the portability of the framework to different underlying accelerators. The glue operator is used to adjust sub-blocks of tensors split in one way into sub-blocks formed by another split. As shown in Table 1, the splitting methods allowed by different types of operators are different in their input tensor data and output tensor data. When the splitting method of the output tensor data of the upper layer operator is not allowed by the next layer operator, it is necessary to use the glue operator to adjust the splitting method of the tensor data, so as to separate the front and rear operators " glue". In addition, even if the splitting method of the output of the previous layer is supported by the next layer, the splitting of the tensor data can be adjusted to a form that is more conducive to the calculation of the next layer through the glue operator.

Table 1

Based on the above description and on the basis of FIG. 2 , this embodiment further proposes another method for splitting the neural network model. As shown in Figure 4. On the basis of Figure 2, it also includes:

Step 201'): insert a glue operator between the operator of the target layer and the associated split state set, and adjust the state in the split state set of the tensor data of the operator; wherein, the glue The operator is used to adjust the state obtained by the tensor data according to a splitting method to a state obtained according to any splitting method.

In this step, the behavior of adjusting the split state of the tensor data is represented by the glue operator. The calculation scale of each layer of the neural network model changes continuously with the extension of the network, and with the split trend of the neural network model The change of the operator needs to be adjusted accordingly, that is, the state of the intermediate result needs to be adjusted. As shown in Figure 5, a glue operator is added between Op_2 in Figure 3 and its input Tensor1, which can convert any split state of tensor data into another split state. For the glue operator, its input tensor data and output tensor data have the same shape and the same state space. From any split state of the input tensor data, there is a pointer to all split states of the output tensor data. Directed edges, so a fully connected network is formed between the split state set of the input tensor data and the split state set of the output tensor data. This allows any split state of the input tensor data to be converted into another split state before operator Op_2, and introduces into the search space of the split scheme P that the input of each operator is adjusted before the calculation starts. The split state of the tensor data, that is, the possibility to adjust the split method of the operator itself.

It should be noted that Fig. 5 shows that the glue operator is inserted between the operator and the corresponding input tensor data. The glue operator can also be inserted between the operator and the corresponding output tensor data. The glue operator is inserted between the corresponding input tensor data and output tensor data. This time is only an example of some cases, not an exhaustive list. Those skilled in the art understand the essence of the technical solution of the present application. , other deformations or transformations may occur on the basis of the technical solutions of the present application, but as long as the functions realized and the technical effects achieved are similar to those of the present application, they should all belong to the protection scope of the present application.

A glue operator is inserted between the operator of the target layer of the neural network model and the associated split state set. Although the split method of the operator is adjusted accordingly, this adjustment will bring additional overhead. How to The glue operator is properly inserted in the whole neural network model to improve the performance of the neural network model. In order to solve this problem, a glue operator is inserted between the operator of the target layer of the neural network model and the associated split state set, and a directed acyclic graph of the neural network model including the glue operator is obtained; according to The directed acyclic graph traverses the split state sets corresponding to all tensor data of the target layer, and determines the state paths and the weights of the state paths between adjacent split state sets; according to the weights of the state paths, Determine the split path of the target layer of the neural network model including the glue operator; use the split path of the target layer of the neural network model including the glue operator to perform each inserted glue operator. Select, delete the glue operators that do not need to be inserted, and keep the glue operators that need to be inserted.

For the glue operator, the glue operator adopts one of the four methods of split-splicing, splicing-splitting, splicing, and splitting. In the splicing stage, adjacent ones in any dimension can be In the splitting stage, any sub-data block can be split into two smaller sub-data blocks. Any split can be transformed into another split through such a two-stage process. To illustrate this point, suppose that the data itself is one-dimensional, and the split form before adjustment is expressed as {(0,p ₁ ),(p ₁ ,p ₂ ),…,(p _n-1 ,end)}, Each segment represents a sub-segment after one-dimensional data is split, and the adjusted split form is {(0,q ₁ ),(q ₁ ,q ₂ ),…,(q _m-1 ,end)}, if The two adjacent segments before adjustment (pi _-1 , _pi ), ( _pi ,pi ₊₁ ) are a segment after adjustment ( _qj ,qj ₊₁ ), that is,pi _-1 =q _j , p _i+1 =q _j+1 , when adjusting this part, only need to splicing (pi _-1 , _pi ),( _pi ,pi ₊₁ ) together in the splicing stage, skip the split stage. In another case as well, if a certain sub-segment before adjustment is a set of several sub-segments after adjustment, the splicing stage is skipped, and corresponding splitting is performed in the splitting stage. In the worst case, all the data can be combined into a complete one-dimensional data in the splicing stage, and the corresponding splitting is carried out in the splicing stage.

Take the glue operator using split-splicing or splicing-split as an example, assuming that the total size of the tensor data to be adjusted is M, and neither stage can be skipped, and each stage must be spliced for 4 dimensions or split. In order to facilitate porting, splicing and splitting are usually implemented using the concatenating operator (Concat) and the splitting operator (Slice) that come with the neural network algorithm. Since these two operators can only process one dimension at a time, the entire glue In the worst case, it will bring 8M storage, read and write overhead. Therefore, it is necessary to find an optimal balance between the adjustment of the split state and the extra overhead introduced, and then to introduce as few glue operators as possible, and to comply with the rules of the network structure and to dismantle the operators in a reasonable place. Adjust the way.

In further detail, the glue operator and the ordinary neural network operator do the same processing. Each glue operator has a corresponding time t to adjust the state of tensor data splitting, and this time is used as the weight of the corresponding state path. . Still use formula (5) to obtain the _unsplit state s _root of the input tensor data of the target layer of the neural network model, including the glue operator, to the unsplit state send of the output tensor data of the neural network model. sub-path. When selecting the glue operator, in the split path, check the split state corresponding to the input tensor data of each glue operator and the split state corresponding to the output tensor data. If the two split states are the same, That is, the split state status_1 in the split state set Tensor_1 in FIG. 5 is connected to the split state status_1 in the split state set Tensor_1' through a state path, and the two split states are the same. It shows that the splitting path P of the target layer of the neural network model does not need to adjust the splitting state of the input tensor data of the operator Op_2, and this is a result based on the overall consideration of the front and rear operators and the overall performance. The glue operator inserted between the operator Op_2 and the corresponding input tensor data is removed from the network. Otherwise, the inserted glue operator needs to be reserved.

It should be noted that the implementation of the glue operator uses the original operator in the neural network model. The splicing stage corresponds to the Concat operator in the neural network model, and the splitting stage corresponds to the Slice operator in the neural network model. Any accelerator that already supports these two operators can quickly implement the glue operator. Moreover, in this embodiment, the above-mentioned method of obtaining the target splitting path is similar to the viterbi algorithm, and this time is only an example of some cases, rather than an exhaustive list. Those skilled in the art understand the essence of the technical solution of the present application. , other deformations or transformations may occur on the basis of the technical solutions of the present application, such as: the split state set of the input tensor data of the neural network model to the split state set of the output tensor data of the neural network model. The weight of each split path is determined by the sum of the weights of the corresponding state paths. A threshold is set according to experience. If the weight of the split path is less than the set threshold, the neural network model can be split as the target split path. However, as long as the functions realized and the technical effects achieved are similar to those of the present application, they should all belong to the protection scope of the present application.

It should be emphasized that the content of the technical solution about operator splitting shown in FIG. 2 is applicable to the technical solution shown in FIG. 4 , and details are not repeated here.

For the neural network model, the convolution operator is a special operator, and in some cases additional auxiliary operators are required to complete the splitting task. When the calculation is divided according to the H/W dimension of the input tensor data, if the size of the convolution kernel window exceeds the step size of each movement, that is, kernel>stride, the split convolution operator will be calculated during the calculation. It occurs that the box moves to the boundary beyond the boundary of the sub-tensor data, and the missing part of the data is located on the adjacent sub-tensor data. In order to deal with the overlapping of input tensor data between subtasks and ensure portability, the behavior of accessing boundary data of adjacent sub-tensor data is also stripped out to form a new auxiliary operator , called the compensation operator.

As shown in Figure 6, the compensation operator is used to obtain target data from adjacent sub-tensor data other than one sub-tensor data, and combine the target data and sub-tensor data to form a larger piece of data. The computation phase window does not move beyond the boundaries of this compensated data block. In addition to the convolution operator, the pooling operator and the currently not very common Local Response Normalization (LRN) operator also have such split subtasks that depend on adjacent data blocks. The problem of data, the pooling operator is similar to the convolution operator, mainly because the pooling window is larger than the moving step size of the window itself, while the local response normalization operator is different, its calculation logic is, that is, in order to calculate To output the result of a point in the C dimension of the tensor data, you need to input a point corresponding to the C dimension of the tensor data and the values of the left and right adjacent k/2 points. Therefore, if the calculation of the local response normalization operator is divided into multiple LRN operators according to the C dimension, each new operator also needs the element data in the adjacent sub-tensor data to calculate the C dimension boundary. value.

According to the range of the input tensor data in the dimension required by the operator to calculate a data in the output tensor data in a certain dimension, the operators can be classified into three categories. One is the point-to-point operator, that is, to calculate a data point of the output tensor data, only the value of a corresponding data point on the input tensor is required. This type of operators includes activation operators (Relu, pRelu), batches Normalization operator (BatchNorm), and basic operators (Add, Sub, Mult, Div) for bitwise addition, subtraction, multiplication and division. This type of operator can perform task splitting in any dimension, and the obtained sub-operator only needs the corresponding sub-tensor data as input in the calculation stage. The other type of operator is a fully dependent type of operator, that is, calculating a data point of the output tensor data only requires all the data in that dimension on the input tensor data, such as the convolution operator and the fully connected operator in the calculation. A point on the C dimension of the output tensor data requires all points on the C dimension of the input tensor data, although the splitting of the convolution operator on the input C dimension can be achieved by accumulating partial sums afterwards. When the calculation logic on the dimension is more complicated, such as the normalized exponential regression operator (Softmax), the formula (6) gives the calculation formula on its normalized dimension.

where I is the vector of the input tensor data in the normalized dimension, and O is the vector of the output tensor data in the normalized dimension. Different from the partial sum accumulation of convolution, the calculation logic here is more complicated, and it is difficult to achieve splitting. From this point of view, the compensation operator is actually used to deal with the third case between the point-to-point type operator and the fully dependent type operator, that is, calculating a point on the output tensor data requires input tensor data Data in the area near the corresponding location. The corresponding position attachment area is determined according to the compensation parameters. In this case, the operators can still be split logically, although they will depend on data other than the sub-tensor data, and the compensation operator can be used to solve this problem uniformly.

Based on this, as shown in FIG. 7 , the third flowchart of a method for splitting a neural network model provided by an embodiment of the present disclosure.

On the basis of Figure 2, it also includes:

Step 201"): insert a compensation operator between the operator of the target layer and the associated split state set, and adjust the state in the split state set of the input tensor data of the operator; The compensation operator is used to obtain target data from adjacent sub-tensor data of any sub-tensor data of the state, and combine the target data and the sub-tensor data.

In this technical solution, in order to solve the problem that the window exceeds the boundary of the input sub-tensor data in the case of task splitting along the H/W dimension due to the window of the convolution operator and the pooling operator being smaller than the displacement step size, The framework introduces a compensation operator, for a sub-tensor data set, to supplement the elements located in adjacent sub-tensor data around each sub-tensor data before the calculation starts. This method avoids modifying the calculation logic of the split convolution operator or the pooling operator itself, so that the dependence on adjacent sub-tensor data is invisible to the convolution operator and the pooling operator itself. It is beneficial to the rapid realization of this system, and it is convenient to be consistent on accelerators of different structures. However, the compensation operator itself will bring additional overhead. Assuming that the size of the original data block is M, the compensation operator introduces a memory access cost of 2M without considering the overlapping part of the sub-tensor data after compensation. . The convolution operator and the pooling operator are the main operators that make up the neural network, especially the image classification neural network. In order to reduce the overhead caused by the compensation behavior, the inserted compensation operators are combined in a pyramid-shaped structure. As shown in Figure 8, the neural network model is a serial sequence composed of two convolution operators. The two convolution operators are divided into four smaller ones according to the H/W dimension. Convolution operator, the N and C dimensions of the data are omitted from the figure. Assuming that the convolution kernel sizes of the two convolution operators are k ₁ and k ₂ respectively, in order to simplify the calculation, the displacement step size is both 1. Under normal circumstances, the data width of the convolution operator Conv1 compensated at the periphery of the sub-tensor data before calculation is k ₁ /2, which ensures that the convolution kernel of the split convolution task will not exceed the input sub-tensor during calculation. data boundaries. But here, the data width of the convolution operator Conv1 compensated at the periphery of the sub-tensor data before calculation is k ₁ /2+k ₂ /2, which makes the sub-tensor data of its output data Tensor1 have k ₂ between each other The widths overlap each other, so the convolution operator Conv2 does not need to perform data compensation on its input sub-tensor data before the computation starts.

In this way, multiple compensation operators used in the serial operator sequence can be combined into one at the top. Although this increases the memory access overhead of the first compensation, the compensation width is much smaller than that of the sub-data block itself. In the case of the size of , it can effectively reduce the memory access cost of the compensation operator after model splitting. But on the other hand, this method will bring repeated calculations. In Figure 8, the result of the overlapping part of the sub-tensor data of the output tensor data Tensor1 of the convolution operator Conv1 is in multiple split convolution operators. are all counted. In addition, for the convolutional network with a small input feature image size, since the compensation width is no longer much smaller than the size of the sub-tensor data itself, it is necessary to more carefully evaluate the total memory access cost before and after combining multiple compensation operators. Changes. To solve this problem, the merging of compensation operators is also added to the search space of the splitting scheme. The whole traversal process is changed from forward traversal to reverse traversal. The two inputs are similar in principle, but the latter is more suitable for the search strategy after the compensating operator is merged. Let the non-split state of the output tensor data of the entire neural network model be the _end state send, and any state s of any tensor data in the neural network model has a corresponding split from s to _send . The weight of the path is _ls . Before the _traversal starts, the corresponding weight of send is 0, and the corresponding weight of all states of all other tensor data is ∞. According to the topological relationship of the neural network model, each operator is traversed in reverse. In the process of enumerating the possible splitting states of the input tensor data from the splitting state of the output tensor data, in addition to enumerating the normal sub-tensor data The split states that do not overlap each other and need to introduce a compensation process, also enumerate the input split states that overlap each other and do not need compensation. The weights of the state paths corresponding to the latter are calculated to take into account the time added by the redundant computation part. Still use formula (5) to obtain the target split of the _unsplit state s _root of the input tensor data of the target layer of the neural network model including the compensation operator to the unsplit state send of the output tensor data of the neural network model. sub-path.

For the compensation operator, a pyramid structure is used to merge multiple compensation operators inserted in the neural network model to obtain one merged compensation operator or multiple merged compensation operators. In this case, the number of compensating operators after merging is less than the number of compensating operators before merging.

It should be noted that the above method of obtaining the target splitting path is similar to the viterbi algorithm, and this time is only a partial example, not an exhaustive list. Those skilled in the art may understand the essence of the technical solution of the present application. Other deformations or transformations are generated on the basis of the technical solutions of the present application, such as: each split between the split state set of the input tensor data of the neural network model and the split state set of the output tensor data of the neural network model The weights of the sub-paths are determined by the sum of the weights of the corresponding state paths. A threshold is set according to experience. If the weight of the split path is less than the set threshold, the neural network model can be split as the target split path. However, as long as the functions realized and the technical effects achieved are similar to those of the present application, they should all belong to the protection scope of the present application.

It should be emphasized that the content of the technical solution about operator splitting shown in FIG. 2 is applicable to the technical solution shown in FIG. 7 , and details are not repeated here.

As shown in FIG. 9 , the fourth flowchart of a method for splitting a neural network model provided by an embodiment of the present disclosure. Both the glue operator and the compensation operator are introduced into the operator splitting scheme. At this time, the splitting method includes:

Step a): According to the operator of the target layer in the neural network model, determine the split state set of the tensor data associated with the operator of the target layer; wherein, the target layer is in the neural network model. at least one layer of;

Step b): insert a glue operator between the operator of the target layer and the associated split state set, and adjust the state in the split state set of the tensor data of the operator; wherein, the glue operator The sub is used to adjust the state in the split state set of the tensor data to any split state of the tensor data;

Step c): insert a compensation operator between the operator of the target layer and the associated split state set, and adjust the state in the split state set of the input tensor data of the operator; wherein, the The compensation operator is used to obtain target data from adjacent sub-tensor data of any sub-tensor data of the state, and combine the target data and the sub-tensor data;

Step d): traverse the split state set according to the directed acyclic graph of the neural network model, and determine the state path and the weight of the state path between adjacent split state sets; The splitting method of the operator; each state in the split state set represents a sub-tensor data set, and the union result of all the sub-tensor data of the state is the tensor data;

Step e): determine the target splitting path of the target layer according to the weight of the state path;

Step f): using the target splitting path to split the operator of the target layer of the neural network model.

A glue operator is inserted between each operator of the neural network model and its input tensor data, and a glue operator is inserted between the output tensor data of the neural network model and the operator that generates the output tensor data. The state set S _i is initialized for each tensor data tensor _i in the neural network model, and the state set stores the state itself and the shortest output state s _root from the split state of the data to the final output data of the network Time, the value pair is represented by (s, t). The state set S _root corresponding to the output tensor data of the entire neural network model has the unsplit state of the data and the corresponding shortest time (s _root , 0), and all other sets are empty. For a given neural network model, a topological order λ is given to all operators in the neural network model according to their dependencies. The topological order must satisfy: For any operator A, all operators that depend on A must be in Topologically sorted after A, and all A-dependent operators must topologically sorted before A.

Considering the insertion of the compensation operator, the split state set of each operator of the neural network model is traversed in reverse. In the reverse traversal stage, the operators in the neural network model are traversed one by one in the order of inverse λ. For the operator A with m input and n output, there are input tensor data u ₁ ,..., _um , and output tensor data v ₁ ,...,v _n , the content of the technical solution about operator splitting in the neural network model shown in Fig. 2 is applicable to the technical solution shown in Fig. 9, and the content of the technical solution shown in Fig. 4 is in the neural network model with the glue operator inserted The technical solution content of the glue operator is applicable to the technical solution shown in FIG. 9 , and the technical solution content of the compensation operator in the neural network model in which the compensation operator is inserted as shown in FIG. 7 is applicable to the technical solution shown in FIG. 9 , will not be repeated here. The time complexity of reverse traversal is O(NM ² ), where N is the number of operators in the neural network model, and M represents the number of states in the largest split state set in the split state set of all tensor data .

It should be emphasized that the content of the technical solution about operator splitting shown in FIG. 2 is applicable to the technical solution shown in FIG. The content is applicable to the technical solution shown in FIG. 9 , and the content about the compensation operator in the technical solution of the operator splitting based on the compensation operator shown in FIG. 7 is applicable to the technical solution shown in FIG. 9 , and will not be repeated here.

The technical solutions shown in Fig. 2, Fig. 4, Fig. 7 and Fig. 9 are sufficient by dividing each operator of the target layer in the neural network model into smaller sub-tasks and assigning them to multiple cores for parallel execution. Take advantage of the hardware resources of a multi-core system. In the technical solutions shown in Fig. 4, Fig. 7 and Fig. 9, by introducing a glue operator or a compensation operator, it is ensured that the computation graph of the neural network model after splitting can still use the operator kernel function on a single core. The implementation avoids the need to modify or re-implement the software stack of a large number of operators for the underlying accelerator during the transplantation process, making it more friendly to accelerators that do not have good programmability. The framework can automatically generate a set of efficient splitting schemes for a given neural network and multi-core accelerator. During the generation process of the scheme, operators can be reasonably adjusted according to the type and scale of the operators combined with the computing throughput rate and memory access bandwidth of the underlying hardware. It achieves a good balance between the computing efficiency of the hardware core and the splitting degree of the operator itself, and also takes into account the mutual cooperation between the context operators in the splitting, and plans multiple operators as a whole. split selection.

The present disclosure provides a neural network model splitting device, including:

A split state set module, configured to determine a split state set of tensor data associated with the operator of the target layer according to the operator of the target layer in the neural network model; wherein, the target layer is the at least one layer in the neural network model;

a state path module, configured to traverse the split state set according to the directed acyclic graph of the neural network model, and determine the state path and the weight of the state path between adjacent split state sets; wherein, the state path Represents the splitting method of the operator; each state in the split state set represents a sub-tensor data set, and the union result of all sub-tensor data of the state is the tensor data;

a target splitting path module, configured to determine the target splitting path of the target layer according to the weight of the state path;

A splitting module, configured to split the operator of the target layer of the neural network model by using the target splitting path.

Preferably, the target splitting path module includes:

The first traversal unit is used to traverse all the split state sets of the target layer, traverse each state for the current split state set, and obtain all the state paths pointing to the current state and the starting state of the state path to all the state paths. Describe the splitting path of the starting state of the input tensor data of the target layer;

a first splitting path determination unit, configured to determine a splitting path from the current state to the starting state of the input tensor data of the target layer according to the weight of the state path and the weight of the splitting path; wherein , the weight of the splitting path is determined according to the weights of all state paths corresponding to the splitting path;

The first selection target splitting path unit is used to obtain the split state set of the input tensor data of the target layer and the output tensor data of the target layer after traversing all the split state sets of the target layer The target split path between the set of split states.

Preferably, the target splitting path module includes:

The second traversal unit is used to traverse all the split state sets of the target layer, traverse each state for the current split state set, and obtain all the state paths starting from the current state and the end states of the state paths to The splitting path of the terminal state of the output tensor data of the target layer;

The second splitting path determination unit is configured to determine the splitting path from the current state to the terminal state of the output tensor data of the target layer according to the weight of the state path and the weight of the splitting path; wherein, The weight of the split path is determined according to the weight of all state paths corresponding to the split path;

The second selection target splitting path unit is used to obtain the split state set of the input tensor data of the target layer and the output tensor data of the target layer after traversing all the split state sets of the target layer The target split path between the set of split states.

Preferably, it also includes:

The first split state set optimization module is used in the forward traversal stage, when the output tensor data of the operator is used as input tensor data by at least two operators, or the operator has at least two output tensor data. When the amount of data is stored, one split state remains in the split state set of the output tensor data of the operator, and the split state is determined through the same state path of the operator.

Preferably, it also includes:

The second split state set optimization module is configured to, in the reverse traversal stage, when the operator has at least two input tensor data, retain one split state set of the input tensor data of the operator split states, and the split states are determined via the same state path of the operator.

As shown in FIG. 10 , a schematic diagram of a hardware device for splitting a neural network model proposed in an embodiment of the present application is shown. It includes a memory, a processor, and a computer program stored on the memory and running on the processor. When the processor executes the computer program, the above-mentioned method for splitting a neural network model is implemented.

The specific functions implemented by the memory and the processor of a neural network model splitting hardware device provided by the embodiments of this specification can be explained in comparison with the foregoing embodiments in this specification, and can achieve the technical effects of the foregoing embodiments. Here I won't go into details.

In this embodiment, the memory may include a physical device for storing information, usually after digitizing the information, it is stored in a medium using electrical, magnetic, or optical methods. The memory described in this embodiment may further include: devices that use electrical energy to store information, such as RAM, ROM, etc.; devices that use magnetic energy to store information, such as hard disks, floppy disks, magnetic tapes, magnetic core memory, magnetic bubble memory, U disk ; a device that stores information optically, such as a CD or DVD. Of course, there are other ways of memory, such as quantum memory, graphene memory, and so on.

In this embodiment, the processor may be implemented in any suitable manner. For example, the processor may take the form of, for example, a microprocessor or a processor and a computer readable medium storing computer readable program code (eg software or firmware) executable by the (micro)processor, logic gates, switches, application specific integrated Circuit (Application Specific Integrated Circuit, ASIC), programmable logic controller and embedded microcontroller form and so on.

In this embodiment, the embodiment of the present application further provides a readable storage medium on which a computer program is stored, and when the computer program is executed, the above-mentioned method for splitting a neural network model is implemented.

It can be seen from the above that the technical solution of the present disclosure can realize the expansion of the deep learning accelerator from a single core to a multi-core structure with less overhead, and can provide an efficient splitting solution for a given network and underlying accelerator characteristics. Experimental results show that this scheme can effectively reduce the end-to-end delay of various networks on multi-core accelerators.

Those skilled in the art also know that, in addition to implementing the client and the server in the form of pure computer-readable program codes, the client and the server can be implemented as logic gates, switches, application-specific integrated circuits, programmable logic by logically programming the method steps. The same function can be realized in the form of a controller and an embedded microcontroller, etc. Therefore, the client and the server can be regarded as a kind of hardware components, and the devices included therein for realizing various functions can also be regarded as structures within the hardware components. Or even, the means for implementing various functions can be regarded as both a software module implementing a method and a structure within a hardware component.

From the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solutions of the present application can be embodied in the form of software products in essence or the parts that make contributions to the prior art, and the computer software products can be stored in storage media, such as ROM/RAM, magnetic disks , CD-ROM, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present application or some parts of the embodiments.

Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the various embodiments may be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the implementation of the client and the server, reference may be made to the description of the foregoing method implementation for comparison and explanation.

The application may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The application may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including storage devices.

Although the present application has been described in terms of embodiments, those of ordinary skill in the art will recognize that the present application is subject to many modifications and variations without departing from the spirit of the present application, and it is intended that the appended claims include such modifications and variations without departing from the spirit of the present application.

Claims (16)

1. a neural network model splitting method, is characterized in that, described method comprises: According to the operator of the target layer in the neural network model, the split state set of the tensor data associated with the operator of the target layer is determined; wherein the target layer is at least one layer in the neural network model ; Traverse the split state set according to the directed acyclic graph of the neural network model, and determine the state paths and the weights of the state paths between adjacent split state sets; wherein, the state path represents the value of the operator Splitting method; each state in the split state set represents a sub-tensor data set, and the union result of all sub-tensor data of the state is the tensor data; Determine the target splitting path of the target layer according to the weight of the state path; The operator of the target layer of the neural network model is split using the target splitting path. 2. The method of claim 1, wherein the step of determining the target splitting path of the target layer comprises: Traverse all split state sets of the target layer, traverse each state for the current split state set, and obtain all state paths pointing to the current state and the input tensors from the starting state of the state path to the target layer The split path of the starting state of the data; The splitting path from the current state to the starting state of the input tensor data of the target layer is determined according to the weight of the state path and the weight of the splitting path; wherein the weight of the splitting path is based on the weight of the splitting path. Determine the weights of all state paths corresponding to the split paths; After traversing all the split state sets of the target layer, obtain the target split between the split state set of the input tensor data of the target layer and the split state set of the output tensor data of the target layer path. 3. The method of claim 1, wherein the step of determining the target splitting path of the target layer comprises: Traverse all split state sets of the target layer, traverse each state for the current split state set, and obtain all the state paths starting from the current state and the output sheets from the end state of the state path to the target layer. The split path of the terminal state of the volume data; The splitting path from the current state to the terminal state of the output tensor data of the target layer is determined according to the weight of the state path and the weight of the splitting path; wherein, the weight of the splitting path is based on the The weights of all state paths corresponding to the split paths are determined; After traversing all the split state sets of the target layer, obtain the target split between the split state set of the input tensor data of the target layer and the split state set of the output tensor data of the target layer path. 4 . The method according to claim 1 , wherein the number of sub-operators obtained after the operators of the target layer of the neural network model are split is an integer power of 2. 5 . 5. The method according to claim 1, wherein the state in the split state set of the input tensor data of the operator of the target layer of the neural network model is based on the calculation logic of the operator and the corresponding output State determination in the split state collection of tensor data. 6. The method according to claim 1, wherein the state in the split state set of the output tensor data of the operator of the target layer of the neural network model is based on the calculation logic of the operator and the corresponding input State determination in the split state collection of tensor data. 7. The method of claim 1, further comprising: In the forward traversal stage, when the output tensor data of the operator is used as input tensor data by at least two operators, or the operator has at least two output tensor data, the output tensor data of the operator One split state remains in the set of split states of the volume data, and the split state is determined via the same state path of the operator. 8. The method of claim 1, further comprising: In the reverse traversal stage, when the operator has at least two input tensor data, one split state remains in the split state set of the input tensor data of the operator, and the split state is passed through the The same state path of the predicate operator is determined. 9 . The method of claim 1 , wherein the weight of the state path is determined according to the type and scale of the operator and hardware parameters of the multi-core processor. 10 . 10. A neural network model splitting device, characterized in that, comprising: A split state set module, configured to determine a split state set of tensor data associated with the operator of the target layer according to the operator of the target layer in the neural network model; wherein, the target layer is the at least one layer in the neural network model; a state path module, configured to traverse the split state set according to the directed acyclic graph of the neural network model, and determine the state path and the weight of the state path between adjacent split state sets; wherein, the state path Represents the splitting method of the operator; each state in the split state set represents a sub-tensor data set, and the union result of all sub-tensor data of the state is the tensor data; a target splitting path module, configured to determine the target splitting path of the target layer according to the weight of the state path; A splitting module, configured to split the operator of the target layer of the neural network model by using the target splitting path. 11. The apparatus of claim 10, wherein the target splitting path module comprises: The first traversal unit is used to traverse all the split state sets of the target layer, traverse each state for the current split state set, and obtain all the state paths pointing to the current state and the starting state of the state path to all the state paths. Describe the splitting path of the starting state of the input tensor data of the target layer; a first splitting path determination unit, configured to determine a splitting path from the current state to the starting state of the input tensor data of the target layer according to the weight of the state path and the weight of the splitting path; wherein , the weight of the splitting path is determined according to the weights of all state paths corresponding to the splitting path; The first selection target splitting path unit is used to obtain the split state set of the input tensor data of the target layer and the output tensor data of the target layer after traversing all the split state sets of the target layer The target split path between the set of split states. 12. The apparatus of claim 10, wherein the target splitting path module comprises: The second traversal unit is used to traverse all the split state sets of the target layer, traverse each state for the current split state set, and obtain all the state paths starting from the current state and the end states of the state paths to The splitting path of the terminal state of the output tensor data of the target layer; The second splitting path determination unit is configured to determine the splitting path from the current state to the terminal state of the output tensor data of the target layer according to the weight of the state path and the weight of the splitting path; wherein, The weight of the split path is determined according to the weight of all state paths corresponding to the split path; The second selection target splitting path unit is used to obtain the split state set of the input tensor data of the target layer and the output tensor data of the target layer after traversing all the split state sets of the target layer The target split path between the set of split states. 13. The apparatus of claim 10, further comprising: The first split state set optimization module is used in the forward traversal stage, when the output tensor data of the operator is used as input tensor data by at least two operators, or the operator has at least two output tensor data. When the amount of data is stored, one split state remains in the split state set of the output tensor data of the operator, and the split state is determined through the same state path of the operator. 14. The apparatus of claim 10, further comprising: The second split state set optimization module is configured to, in the reverse traversal stage, when the operator has at least two input tensor data, retain one split state set of the input tensor data of the operator split states, and the split states are determined via the same state path of the operator. 15. A neural network model splitting hardware device, comprising a memory and a processor, and a computer program that can be run on the processor is stored on the memory, wherein the processor implements the right when executing the computer program. The steps of the method of any one of claims 1-9. 16. A computer-readable storage medium on which a computer program is stored, characterized in that, when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images