WO2019184888A1 - Image processing method and apparatus based on convolutional neural network - Google Patents

Abstract

Provided are an image processing method and apparatus based on a convolutional neural network. The method comprises: acquiring FN frames of image feature map; equally dividing each of the FN frames of image feature map into Q sub-images, wherein each of the sub-images has a height of C pixel points and a width of P pixels points, and the sub-images at the same position in the FN frames of image feature map form a three-dimensional image matrix with a height of C pixel points, a width of P pixel points, and a length of FN pixel points, where Q, P, C are all positive integers; acquiring a preset weight matrix, wherein the preset weight matrix is an FN x FN two-dimensional matrix; and sequentially performing, according to the preset weight matrix, a matrix multiplication calculation on each of the three-dimensional image matrices in the FN frames of image feature map to obtain a processed image feature map, so that block processing can be performed during convolutional processing of the image, and the convolutional calculation performance is thus improved.

Description

Method and device for image processing based on convolutional neural network

The present application claims priority to Chinese Patent Application No. 201810266076.7, entitled "Method and Apparatus for Image Processing Based on Convolutional Neural Networks", filed on March 28, 2018, the entire contents of which is incorporated herein by reference. This is incorporated herein by reference.

Technical field

The present application relates to the field of computer technology, and in particular, to a method and apparatus for image processing based on a convolutional neural network.

Background technique

The in-depth development of the Convolutional Neural Network (CNN) has made it difficult to solve many traditional computer methods in the fields of image processing, pattern recognition, intelligent robotics, automatic control, predictive estimation, biology, medicine, and economy. The actual problem solved. In the model of convolutional neural network algorithm, convolution calculation accounts for 90% of the calculation of the whole algorithm model. Therefore, the efficient calculation of convolutional layer is the key to greatly improve the computational efficiency of CNN algorithm model. Convolution calculation is realized by hardware acceleration. It is an effective way.

Figure 1 shows a schematic diagram of a conventional convolutional neural network convolutional layer. The convolution layer is an image feature map (output_fm) obtained by convolving the input image feature map (input_fm) and weights, wherein the number of feature layers of input_fm is N, and the number of feature layers of output_fm is M, weights The convolution kernel size is K*K, R is the height of the input image feature map, and P is the width of the input image feature map. The calculation of the convolutional layer is a three-dimensional input_fm(N*R*P) multiplied by four-dimensional weights (N*M*K*K) to obtain a three-dimensional output_fm(M*R*P), that is, a K*K convolution. The kernel and the corresponding K*K image content on the input image feature map are convoluted to obtain an output point, and it can be seen that when the convolution calculation of the image is performed by using the convolutional neural network convolution layer shown in FIG. Convolution calculation can only be performed on one pixel of the input image feature map, and the calculation performance is low.

Summary of the invention

The embodiment of the present application provides a method and apparatus for image processing based on a convolutional neural network to solve the problem of low computational performance of the existing convolutional neural network convolutional layer.

The specific technical solutions provided by the embodiments of the present application are as follows:

In a first aspect, the embodiment of the present application provides a method for image processing based on a convolutional neural network, comprising: acquiring an FN frame image feature map; and dividing each frame in the FN frame image feature map into Q sub-images, Each of the sub-images has a height of C pixels and a width of P pixels, wherein the sub-images of the same position in the FN frame image feature map form a height of C pixels and a width of P a pixel, a three-dimensional image matrix of FN pixels, wherein Q, C, and P are positive integers; a preset weight matrix is obtained, and the preset weight matrix is a two-dimensional matrix of FNxFN; The weight matrix sequentially performs matrix multiplication calculation on each of the three-dimensional image matrices in the FN frame image feature map to obtain a processed image feature map.

This has the advantage that, in the convolution processing of the input image feature map, the one-time image convolution process can process the data of one sub-image and improve the calculation performance.

In combination with the first aspect, in a possible design, the value of the FN includes: one of 2, 3, and 4.

The beneficial effect is that the value of the design FN is a small value, and the number of frames of the image feature map of the convolutional layer input and output is generally an integer multiple of 2, 3 or 4, and the frame of the image feature map that the FN can basically be input. The number of frames of the number and output image feature maps are divisible, or the remainder is small, which can improve the calculation performance and save hardware resources.

In combination with the first aspect, in a possible design, the value of the P includes: one of 16, 32, 64, 128, and 256.

This has the advantage that it can be processed in such a way that each frame of image feature map is sliced into image blocks. At this time, the sub-image divided at this time has the same width as the original image feature map, and the height is the height of the preset image block, so that the image feature map can be divided into image blocks of the horizontal bar, and the image data of one line can be processed at one time. , thus compatible with existing image processing frameworks, reducing storage resources.

In conjunction with the first aspect, in one possible design, the width of the image feature map is equal to the width of the sub-image.

With reference to the first aspect, in a possible design, the C is 1; correspondingly, the height is C pixels, the width is P pixels, and the three-dimensional image matrix of FN pixels is wide P pixels, a two-dimensional image matrix of FN pixels.

In conjunction with the first aspect, in one possible design, when the method is applied to the first neural network layer of the convolutional neural network, the image feature map is an image to be processed.

In conjunction with the first aspect, in a possible design, when the method is used for a non-first neural network layer of the convolutional neural network, the acquiring an FN frame image feature map includes: going to a previous adjacency The processed image feature map obtained in the neural network layer is used as the FN frame image feature map.

With reference to the first aspect, in a possible design, the acquiring an FN frame image feature map includes: sequentially acquiring an image feature map in an N-frame image feature map, and acquiring the FN frame image feature map each time, wherein, N Is a positive integer.

In practical applications, the number of frames of the image feature map input by the CNN convolutional layer is N. When performing image processing based on the convolutional neural network, the input N-frame image feature maps may be grouped, and each group includes FN. Frame image feature map to improve processing efficiency.

With reference to the first aspect, in a possible design, the obtaining the processed image feature map comprises: sequentially storing the processed image feature map, and storing the processed image feature map of the FN frame each time, wherein The stored image feature map is an M frame, and M is a positive integer.

In a second aspect, an embodiment of the present application provides an apparatus for image processing based on a convolutional neural network, including:

a first acquiring unit, configured to acquire an FN frame image feature map; and a dividing unit, each frame in the FN frame image feature image is divided into Q sub-images, wherein each of the sub-images has a height of C pixels The width is P pixels, wherein the sub-images at the same position in the image map of the FN frame form a three-dimensional image matrix with a height of C pixels, a width of P pixels, and a length of FN pixels. The Q, C, and P are all positive integers; the second acquiring unit is configured to obtain a preset weight matrix, the preset weight matrix is a two-dimensional matrix of FNxFN, and the processing unit is configured to use the preset weight matrix. And performing matrix multiplication calculation on each of the three-dimensional image matrices in the FN frame image feature map to obtain a processed image feature map.

In combination with the second aspect, in a possible design, the value of the FN includes: one of 2, 3, and 4.

In combination with the second aspect, in a possible design, the value of the P includes: one of 16, 32, 64, 128, and 256.

In conjunction with the second aspect, in one possible design, the width of the image feature map is equal to the width of the sub-image.

With reference to the second aspect, in a possible design, the C is 1; correspondingly, the height is C pixels, the width is P pixels, and the length of the three-dimensional image matrix of FN pixels is width P pixels, a two-dimensional image matrix of FN pixels.

In conjunction with the second aspect, in one possible design, when the apparatus is for processing a first neural network layer of the convolutional neural network, the image feature map is an image to be processed.

With reference to the second aspect, in a possible design, when the apparatus is configured to process a non-first neural network layer of the convolutional neural network, the first acquiring unit, when acquiring an FN frame image feature map, Specifically, the processed image feature map obtained in the last adjacent neural network layer is used as the FN frame image feature map.

With reference to the second aspect, in a possible design, when acquiring the FN frame image feature map, the first acquiring unit is specifically configured to: sequentially acquire image feature maps in the N-frame image feature map, and acquire the FN each time. A frame image feature map in which N is a positive integer.

With reference to the second aspect, in a possible design, when the processing unit obtains the processed image feature map, the processing unit is specifically configured to: sequentially store the processed image feature map, and store the FN frame after the processing. The image feature map, wherein the stored image feature map is an M frame, and M is a positive integer.

In a third aspect, an embodiment of the present application provides an image processing device, where the device includes a processor and a memory, wherein the memory stores a computer readable program, and the processor is implemented by running a program in the memory. The first aspect relates to a method of image processing based on convolutional neural networks.

In a fourth aspect, the embodiment of the present application provides a computer storage medium for storing the computer software instructions according to the first aspect, which includes a program designed to perform the above aspects.

In a fifth aspect, an embodiment of the present application further provides a computer program product comprising instructions, when executed on a computer, causing a computer to perform the method described in the first aspect above.

It should be understood that the second to fifth aspects of the embodiments of the present application are consistent with the technical solutions of the first and second aspects of the embodiments of the present application, and the beneficial effects obtained by the aspects and the corresponding implementable design manners are similar, and are not described again.

DRAWINGS

1 is a schematic diagram of a convolutional neural network convolutional layer;

2 is a diagram of a computing engine architecture of a convolutional neural network;

3 is a schematic diagram of a convolutional neural network convolutional layer;

4 is a diagram of a computing engine architecture of a convolutional neural network;

FIG. 5 is a flowchart of a method for image processing based on a convolutional neural network in an embodiment of the present application; FIG.

6 is a schematic diagram of an image processing process of an image processing device in an embodiment of the present application;

7A, 7B, and 7C are structural diagrams of image feature maps, weight data, and output image feature maps input in the embodiment of the present application;

8 is a structural diagram of a computing engine of a processor in an embodiment of the present application;

FIG. 9 is a schematic diagram of an apparatus 900 for image processing based on a convolutional neural network according to an embodiment of the present application;

FIG. 10 is a schematic structural diagram of an image processing apparatus 1000 according to an embodiment of the present application.

detailed description

The technical solutions in the embodiments of the present application will be clearly and completely described in the following with reference to the accompanying drawings in the embodiments.

The computational engine architecture designed using the convolutional layer of the convolutional neural network shown in Figure 1 is shown in Figure 2, consisting of K*K multipliers and (K*K-1) adders.

Since the size of the convolution kernel K of each convolutional layer of CNN is not fixed, it may be 1x1, 3x3, 5x5 or even 11x11. The computing engine of this technology must support convolution kernels of various sizes, that is, the largest convolution kernel needs to be designed in the chip design, such as limiting the maximum support of 7x7, so that a PE needs 49 multipliers on the hardware, for the volume. The accumulation is 1x1 and 3x3 convolutional layers, only one or 9 multiplier units are used. In this case, the utilization rate is only 2% and 18%, which will result in a large waste of resources and inefficient use of hardware. Resources to reduce system performance.

At present, there is an improved convolutional neural network calculation method. According to the idea of block processing, the original image to be convoluted is divided into a plurality of image blocks of Tr*Tc. For each image block, the processing process is as follows. Figure 3 shows. The specific implementation process is: dividing the number of layers N of the input image feature map into units of Tn, and each Tn layer is used as a unit, and similarly, the number M of layers of the output image feature map is divided into units of Tm, each The Tm layer acts as a unit. Specifically, the corresponding computing engine architecture for implementing the convolution operation is as shown in FIG. 4: the calculation engine processes the Tn layer feature map corresponding to one point of the input image feature map, and outputs a Tm layer feature map corresponding to one point. The calculation engine consists of Tn*Tm multipliers and (Tn*Tm-1) adders.

The input image feature map of the CNN layer convolution layer is different from the output image feature map. If the Tm and Tn settings are small, the calculation engine has a small calculation capability and low performance. If Tm, Tn setting is large, it will cause waste of hardware resources. Assuming a possible design, the design Tm = 64, Tn = 7, such a calculation engine has 448 multipliers. In many CNN networks, there are many convolutional layers in which the number of image feature layers is smaller than 7 and 64, and many are not integer multiples of 7 and 64. This does not make full use of hardware resources to waste resources. For example, the input image feature map and the output image feature map have layers of 3 and 6, respectively, and only 18 of the 448 multipliers are utilized, and the utilization rate is only 4%, resulting in waste of hardware resources.

Since the calculation engine designed by using the existing convolutional neural network-based calculation method inevitably wastes hardware resources, the present application provides a new method and device for image processing based on convolutional neural network. , can improve the calculation performance when convolving the image.

As shown in FIG. 5, the present application provides a method for image processing based on a convolutional neural network, which specifically includes the following process:

Step 50: Acquire an FN frame image feature map.

Specifically, acquiring the FN frame image feature map includes: sequentially acquiring an image feature map in the N frame image feature map, and acquiring the FN frame image feature map each time, where N is a positive integer.

In practical applications, the number of frames of the image feature map input by the CNN convolutional layer is N. When performing image processing based on the convolutional neural network, the input N-frame image feature maps may be grouped, and each group includes FN. Frame image feature map to improve processing efficiency.

The value of the FN includes: one of 2, 3, and 4. The reason for designing the FN in this way is that the number of frames of the image feature map of the input and output of the CNN convolution layer is generally an integer multiple of 2, 3 or 4, and the number of frames of the image feature map that the FN can basically be input is N, and the image features of the output image feature. The frame number M of the figure is divisible, or the remainder is small, thereby improving computational performance and saving hardware resources.

Step 51: Divide each frame in the FN frame image feature map into Q sub-images, wherein each of the sub-images has a height of C pixels and a width of P pixels, wherein the FN frame The sub-images at the same position in the image feature map constitute a three-dimensional image matrix of C pixels, a width of P pixels, and a length of FN pixels, wherein Q, C, and P are positive integers.

Optionally, the value of the P includes: one of 16, 32, 64, 128, and 256; the width of the image feature map is equal to the width of the sub-image.

In this way, each frame of the image feature map can be processed into a pattern of stripes. At this time, the sub-image segmented at this time has the same width as the original image feature map, and the height is the height of the preset image block, so that the image feature map can be divided into horizontal bar image blocks, and one line of image data can be processed at one time.

In a possible design, the C is 1; correspondingly, the height is C pixels, the width is P pixels, and the three-dimensional image matrix of FN pixels is P pixels wide. A two-dimensional image matrix of FN pixels.

Step 52: Acquire a preset weight matrix, where the preset weight matrix is a two-dimensional matrix of FNxFN.

Step 53: Perform matrix multiplication calculation on each of the three-dimensional image matrices in the FN frame image feature map according to the preset weight matrix to obtain a processed image feature map.

Further, when the method shown in FIG. 5 is used for the first neural network layer of the convolutional neural network, the image feature map is an image to be processed. When the method shown in FIG. 5 is used for the non-first neural network layer of the convolutional neural network, the acquiring the FN frame image feature map includes: the processing to be obtained in the last adjacent neural network layer The image feature map is used as the FN frame image feature map.

Specifically, the obtaining the processed image feature map in step 53 includes: sequentially storing the processed image feature map, and storing the processed image feature map of the FN frame each time, wherein the stored image feature The image feature map is an M frame, and M is a positive integer.

Based on the method shown in FIG. 5, the embodiment of the present application proposes a new image processing device, where the image processing device includes a memory and a processor, wherein the number of frames of the input image feature map (input_fm) received by the image processing device is N, the number of frames of the output image feature map (output_fm) is M, and the size of the convolution kernel of the weight is K*K. The workflow of the image processing apparatus can be seen in FIG. 6.

S101: Receive an input N frame image feature map, as shown in FIG. 7A.

The input image feature map is the three-dimensional data of N*P*C, that is, the size of each feature map is P*C, there are N image feature maps in total, C is the height of the image feature map, and P is the image feature map. width.

S102: Obtain weight data.

The weight data is a four-dimensional matrix composed of M*N K*K convolution kernels, that is, a four-dimensional matrix of M*N*K*K, as shown in FIG. 7B.

S103: Store the input N-frame image feature map and weight data into a memory.

The image feature map is stored in the memory in the order from the first image feature map to the Nth image feature map, and each image feature map is stored point by point in a left-to-right, top-to-bottom order. The number of frames of the output image feature map is M, which is stored in a similar manner to the input image feature map.

The storage order of the weight data in the memory is to store the two-dimensional data of K*K first, then store the N-dimensional data, and finally store the M-dimensional data.

S104: The processor calls an image feature map stored in the memory, and acquires a basic unit of the input image feature map in the memory.

The basic unit of the image feature map is: a two-dimensional image matrix of FN*P composed of corresponding P pixels on the image map of the adjacent FN frame, wherein P is set to the width of one row of each image block.

Specifically, the acquiring process of the basic unit of the input image feature map is: taking a two-dimensional matrix of the FN*C as a basic unit and acquiring the loop image by using a loop Tiling on the input image feature map. The loop is first cycled in the direction of arrow i1 (i.e., N) in Figure 7A and then in the direction of arrow i2 (i.e., C). First, the two-dimensional matrix FN*P composed of all the pixels in the first row of the image feature map of the first, second, ..., FN frame is composed to form the first input image feature map basic unit; then, according to the arrow i1 in FIG. 7A The direction shifting FN frame image feature map shown, that is, the two-dimensional matrix FN*P composed of all the pixels in the first row of the FN+1, FN+2, ..., 2*FN frame image feature map constitutes the second The basic unit of the input image feature map; after the data of the first line of all the N-frame image feature maps is cycled, move to the second line in the direction of the arrow i2 in FIG. 7A, and then circulate according to the i1 arrow. .

S105: The processor calls the weight data stored in the memory, and acquires the basic unit of the weight data in the memory.

The basic unit of the weight data is: a two-dimensional matrix composed of FN*FN weights.

Specifically, the basic unit obtaining process of the weight data is: acquiring, by using a FN*FN*1*1 as a basic unit, a four-dimensional weight data stored in the memory in a loop Tiling manner. There are two ways to cycle the weight data. One is to cycle in the direction of arrow w1 (ie K*K) in Figure 7B, then cycle in the direction of arrow w2 (ie N), and finally follow arrow w3 (ie M The direction of the cycle; the other is to first cycle in the direction of arrow w1 (ie K*K) in Figure 7B, then cycle in the direction of arrow w3 (ie M), and finally cycle in the direction of arrow w2 (ie N) .

S106: The processor performs multiplication calculation on the basic unit of the image feature map and the basic unit of the weight data to output a basic unit of the image feature map.

Wherein: the basic unit of the output image feature map and the basic unit of the input image feature map have the same data format, and are: two-dimensional FN*P composed of P pixels corresponding to the image map of the adjacent FN frame image. matrix.

S107: Store the basic unit of the output image feature map into the memory.

Specifically, the basic unit of the output image feature map is stored in the following manner: the basic unit of the output image feature map is filled with the FN*P as a basic unit in a cyclic block manner to the output image feature map, and the loop mode is It is first circulated in the direction of the arrow o1 (i.e., M) in Fig. 7C and then in the direction of the arrow o2 (i.e., C).

S108: Determine whether the processor is applied to the last neural network layer of the convolutional neural network, and if yes, execute S109; otherwise, return the output image feature map as an input image feature map of the next neural network layer to execute S104.

S109: Output the output image feature map as an output image of the image processing device.

The processor designed by the image processing method of the present application can construct the basic unit of the input image feature map and the basic unit of the weight data, perform multiplication operation, and obtain the output basic unit of the special function diagram, and the calculation engine architecture of the processor is as shown in the figure. 8 shows:

The computation engine architecture consists of P*FN*FN multipliers and P*FN*(FN-1) adders. The basic unit of the input image feature map: FN*P two-dimensional data and the basic unit of weight data: FN*FN two-dimensional data is allocated to the corresponding multiplier for multiplication, and then the intermediate result is assigned to the corresponding addition. The addition operation is performed on the device to obtain the basic unit of the output image feature map.

In the specific implementation, the FN is set to a smaller value, specifically set to 2, 3 or 4. The number of image feature layers of the input and output of the CNN convolutional layer is generally an integer multiple of 2 or 3. The FN can be substantially divisible by M and N or the remainder is small. Therefore, it is possible to improve the utilization of hardware resources in the processor and improve resources. Utilization, achieving higher computing performance.

In one possible design, P is set to the width of each image block in a specific implementation, so that the calculation engine can process image data of one line at a time. In hardware implementation, it is easier to expand when resources are allowed, and only need to increase P.

Based on the foregoing method embodiment, as shown in FIG. 9, the embodiment of the present application provides an apparatus 900 for image processing based on a convolutional neural network, where the apparatus 900 includes a first obtaining unit 901, a dividing unit 902, and a second acquiring unit 903. Processing unit 904, wherein:

a first acquiring unit 901, configured to acquire an FN frame image feature map;

The dividing unit 902 is divided into Q sub-images for each frame in the FN frame image feature map, wherein each of the sub-images has a height of C pixels and a width of P pixels, wherein the FN The sub-images at the same position in the frame image feature map form a three-dimensional image matrix with C pixels, a P pixel, and a FN pixel, wherein Q, C, and P are positive integers;

The second obtaining unit 903 is configured to acquire a preset weight matrix, where the preset weight matrix is a two-dimensional matrix of FNxFN;

The processing unit 904 is configured to perform matrix multiplication calculation on each of the three-dimensional image matrices in the FN frame image feature map according to the preset weight matrix to obtain a processed image feature map.

Optionally, the value of the FN includes: one of 2, 3, and 4.

Optionally, the value of the P includes: one of 16, 32, 64, 128, and 256.

Optionally, the width of the image feature map is equal to the width of the sub-image.

Optionally, the C is 1; correspondingly, the height is C pixels, the width is P pixels, and the three-dimensional image matrix of FN pixels is P pixels wide and FN long. A two-dimensional image matrix of pixels.

Optionally, when the apparatus is configured to process the first neural network layer of the convolutional neural network, the image feature map is an image to be processed.

Optionally, when the device is used to process the non-first neural network layer of the convolutional neural network, the first acquiring unit 901 is specifically configured to: when acquiring the FN frame image feature map The processed image feature map obtained in an adjacent neural network layer is used as the FN frame image feature map.

Optionally, when acquiring the FN frame image feature map, the first acquiring unit 901 is specifically configured to:

The image feature maps in the N-frame image feature map are sequentially acquired, and the FN frame image feature map is acquired each time, where N is a positive integer.

Optionally, when the processed image feature map is obtained, the processing unit 904 is specifically configured to:

The processed image feature map is sequentially stored, and the processed image feature map of the FN frame is stored each time, wherein the stored image feature map is an M frame, and M is a positive integer.

It should be noted that the function implementation and the interaction mode of the various units of the apparatus 900 in the embodiment of the present application may be further referred to the description of the related method embodiments, and details are not described herein again.

According to the same inventive concept, an embodiment of the present application further provides an image processing apparatus 1000. As shown in FIG. 10, the image processing apparatus 1000 includes a processor 1001 and a memory 1002. The program code for executing the solution of the present invention is stored in the memory 1002. The instruction processor 1001 is used to execute the image processing method shown in FIG.

The application can also perform the design programming of the processor to solidify the code corresponding to the method shown in FIG. 5 into the chip, so that the chip can perform the method shown in FIG. 5 during operation.

It can be understood that the foregoing processor in the embodiment of the present application may be a CPU, a DSP, an ASIC, or one or more integrated circuits for controlling the execution of the program of the present invention. One or more memories included in the computer system, which may be read-only memory (ROM: ROM) or other types of static storage devices that can store static information and instructions, random access memory (English: random accessmemory, Abbreviation: RAM) or other types of dynamic storage devices that can store information and instructions, or disk storage. These memories are connected to the processor via a bus or can also be connected to the processor via a dedicated cable.

Those skilled in the art will appreciate that embodiments of the present application can be provided as a method, system, or computer program product. Therefore, the embodiments of the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Moreover, embodiments of the present application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) including computer usable program code.

Embodiments of the present application are described with reference to flowchart illustrations and/or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present application. It will be understood that each flow and/or block of the flowchart illustrations and/or FIG. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing device to produce a machine for the execution of instructions for execution by a processor of a computer or other programmable data processing device. Means for implementing the functions specified in one or more of the flow or in a block or blocks of the flow chart.

The computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture comprising the instruction device. The apparatus implements the functions specified in one or more blocks of a flow or a flow and/or block diagram of the flowchart.

These computer program instructions can also be loaded onto a computer or other programmable data processing device such that a series of operational steps are performed on a computer or other programmable device to produce computer-implemented processing for execution on a computer or other programmable device. The instructions provide steps for implementing the functions specified in one or more of the flow or in a block or blocks of a flow diagram.

It is apparent that those skilled in the art can make various modifications and variations to the embodiments of the present application without departing from the spirit and scope of the application. Thus, it is intended that the present invention cover the modifications and variations of the embodiments of the present invention.

Claims (18)

A method for image processing based on a convolutional neural network, comprising: Obtaining an image map of the FN frame image; Each frame in the FN frame image feature map is divided into Q sub-images, wherein each of the sub-images has a height of C pixels and a width of P pixels, wherein the FN frame image feature map The sub-images in the same position form a three-dimensional image matrix with C pixels, a P pixel, and a FN pixel, wherein Q, C, and P are positive integers; Obtaining a preset weight matrix, where the preset weight matrix is a two-dimensional matrix of FNxFN; Performing matrix multiplication calculation on each of the three-dimensional image matrices in the FN frame image feature map according to the preset weight matrix to obtain a processed image feature map. The method according to claim 1, wherein the value of the FN comprises one of 2, 3 and 4. The method according to claim 1 or 2, wherein the value of P comprises one of 16, 32, 64, 128 and 256. The method according to any one of claims 1 to 3, wherein the width of the image feature map is equal to the width of the sub-image. The method according to any one of claims 1 to 4, wherein the C is 1; correspondingly, the height is C pixels, the width is P pixels, and the length is FN pixels. The three-dimensional image matrix is a two-dimensional image matrix having a width of P pixels and a length of FN pixels. The method according to any one of claims 1 to 5, wherein when the method is used for the first neural network layer of the convolutional neural network, the image feature map is an image to be processed. The method according to any one of claims 1 to 6, wherein when the method is used for a non-first neural network layer of the convolutional neural network, the acquiring an FN frame image feature map includes : treating the processed image feature map obtained in the last adjacent neural network layer as the FN frame image feature map. The method according to any one of claims 1 to 7, wherein the acquiring an FN frame image feature map comprises: The image feature maps in the N-frame image feature map are sequentially acquired, and the FN frame image feature map is acquired each time, where N is a positive integer. The method according to any one of claims 1 to 8, wherein the obtaining the processed image feature map comprises: The processed image feature map is sequentially stored, and the processed image feature map of the FN frame is stored each time, wherein the stored image feature map is an M frame, and M is a positive integer. An apparatus for image processing based on a convolutional neural network, comprising: a first acquiring unit, configured to acquire an FN frame image feature map; a dividing unit, each frame in the FN frame image feature map is divided into Q sub-images, wherein each of the sub-images has a height of C pixels and a width of P pixels, wherein the FN frame The sub-images at the same position in the image feature map form a three-dimensional image matrix with C pixels, a P pixel, and a FN pixel, wherein Q, C, and P are positive integers; a second acquiring unit, configured to acquire a preset weight matrix, where the preset weight matrix is a two-dimensional matrix of FNxFN; And a processing unit, configured to perform matrix multiplication calculation on each of the three-dimensional image matrices in the FN frame image feature map according to the preset weight matrix to obtain a processed image feature map. The apparatus according to claim 10, wherein the value of the FN comprises one of 2, 3 and 4. The apparatus according to claim 10 or 11, wherein the value of said P comprises one of 16, 32, 64, 128 and 256. The apparatus according to any one of claims 10 to 12, wherein the width of the image feature map is equal to the width of the sub-image. The device according to any one of claims 10 to 13, wherein the C is 1; correspondingly, the height is C pixels, the width is P pixels, and the length is FN pixels. The three-dimensional image matrix is a two-dimensional image matrix having a width of P pixels and a length of FN pixels. Apparatus according to any one of claims 10 to 14, wherein said image feature map is an image to be processed when said apparatus is for processing a first neural network layer of said convolutional neural network. The apparatus according to any one of claims 10 to 15, wherein when said apparatus is for processing a non-first neural network layer of said convolutional neural network, said first acquisition unit is acquiring FN The frame image feature map is specifically used to: use the processed image feature map obtained in the previous adjacent neural network layer as the FN frame image feature map. The device according to any one of claims 10 to 16, wherein the first acquiring unit is configured to: when acquiring an FN frame image feature map, specifically: The image feature maps in the N-frame image feature map are sequentially acquired, and the FN frame image feature map is acquired each time, where N is a positive integer. The device according to any one of claims 10 to 17, wherein the processing unit is configured to: when obtaining the processed image feature map: The processed image feature map is sequentially stored, and the processed image feature map of the FN frame is stored each time, wherein the stored image feature map is an M frame, and M is a positive integer.

Images