CN111263163A - Method for realizing depth video compression framework based on mobile phone platform - Google Patents

Abstract

The invention provides a method for realizing a depth video compression framework based on a mobile phone platform, which belongs to the fields of image classification, target detection, face recognition and the like, and comprises the following steps: s1, building the whole video compression network, training the model by using videos of a plurality of different scenes to obtain a trained large network, and storing the graph model and parameter information of the network; s2, pruning and quantifying the trained model; s3, pruning and quantization are performed for each layer, and the weights in the entire network are huffman coded using huffman coding and then stored. Under the condition of low precision loss, the depth video compression model is compressed by utilizing pruning, quantization and Huffman coding, so that the model is about 1/100 times of the original model, and a video compression framework based on depth learning can be conveniently deployed in mobile phone equipment.

Description

A Implementation Method of Deep Video Compression Framework Based on Mobile Phone Platform

technical field

The invention relates to the fields of image classification, target detection, face recognition and the like, in particular to an implementation method of a deep video compression framework based on a mobile phone platform.

Background technique

Today, video has become the main medium for the public to disseminate information. Especially with the development of self-media, video data is growing explosively. Video compression methods based on deep learning have become the mainstream direction of recent research. The video compression method based on deep learning has become a strong competitor of the current mainstream methods H.264 and H.265.

However, video compression methods based on deep learning often have a large amount of parameters. Because mobile devices are often limited in storage and computing power, they cannot be deployed to mobile devices at all. Therefore, how to compress the deep learning video compression algorithm deployed in mobile phones? , became the key issue.

SUMMARY OF THE INVENTION

The technical task of the present invention is to solve the shortcomings of the existing deep learning video compression framework, which is very large and difficult to deploy in embedded devices such as mobile phones, and provides an implementation method of a deep video compression framework based on a mobile phone platform. The present invention utilizes pruning, quantization and Huffman coding to compress the deep video compression model under the condition of little loss of precision, so that the video compression framework based on deep learning is deployed in mobile phones.

The technical scheme adopted by the present invention to solve its technical problems is:

This patent mainly proposes to use pruning, quantization, and Huffman coding to deploy an excellent deep learning-based video compression framework on a mobile phone platform.

1. An implementation method of a deep video compression framework based on a mobile phone platform, the implementation steps of the method are as follows:

S1. Build the entire video compression network, use multiple videos of different scenes to train the model, and then use more than 5,000 videos of different scenes to train the model, iterate a total of 1 million times to obtain a large trained network, and then put The graph model and parameter information of the network are saved;

S2, and then prune and quantify the trained model;

S3, pruning and quantization are performed separately for each layer. In order to further reduce storage, Huffman encoding is used to Huffman the weights in the entire network, and then stored.

Scheme Preferably, the video compression network built using the tensorflow framework in step 1 includes six networks of opticalFlow net, MV Encoder net, MV Decoder net, Motion Compensation Net, Residualencoder net, and Residual decoder net. The working process is as follows:

S101, split the video into each frame of pictures, input the current frame and the last reconstructed frame to the optical flow network Optical FlowNet, and obtain the motion vector of the current frame;

S102, then encode the motion vector through the motion vector encoding network MV Encoder Net to obtain the encoded result,

S103, quantize the encoded result again to obtain the quantized result, as one of the contents that the current frame needs to store;

S104, the result after passing through the motion vector decoding network MV Decoder Net, that is, the reconstructed motion vector of the current frame, and the picture of the previous reconstructed frame are input into the motion compensation network Motion compensation Net to obtain the predicted frame of the current frame;

S105, using the real frame and the predicted frame to subtract, to obtain residual information r _t that the predicted frame failed to include;

S106, encode Residual encoder net, quantize Q, and entropy encode and store the residual information, then decode the Residual decoder net to obtain the reconstruction result of the residual, and then add it to the predicted frame to obtain the final reconstructed frame;

S107 , the compressed video needs to save the encoding of the motion vector after quantization in step S103 and the residual encoding after quantization in step S106 .

Scheme Preferably, in step 2, step comprises as follows:

S201. The first is pruning. By visualizing the trained weights of each layer, all the data whose absolute value is less than 0.5 are pruned to obtain a sparse matrix, and the obtained sparse matrix is stored.

Change the value stored at the absolute position of the index to use the relative value diff. The diff represents the offset of the current value from the previous value. Set the maximum offset to 8, so that 3 bits will be used to store each value. Offset, and add a number 0 at the position of 12, so that when idx is 15, the offset is 3;

S202. After the pruning is completed, quantify the data after the pruning.

Solution Preferably, in step S201, the CSR is used to store the matrix.

Solution Preferably, in step S202, the traditional K-mean algorithm is used to perform matrix quantization.

Scheme Preferably, in step S202, the K-mean algorithm is specifically as follows:

First, select the initial value in K-means, and then perform sampling. The K used is 11, that is, 11 points are selected;

Then use the K-mean algorithm for training to obtain the final 11 center points, and then cluster the data into the corresponding clusters, assuming that K=4 is used, and then use K-mean clustering to obtain each data. Cluster, get 4 cluster centers respectively, and then only need to store these 4 numbers, we also need to store each data index;

After the quantization is completed, the model needs to be tuned a bit, each parameter is reversely derived, and then the derivatives of each cluster are added, and then the quantized parameters are gradient descent using this summed gradient—— param-lr*gradient.

Scheme Preferably, the selection of the initial value in K-means uses a method based on data density, that is, the frequency of data occurrence is used as the probability of selection, and then sampling is performed.

Compared with the prior art, the beneficial effects of a method for implementing a deep video compression framework based on a mobile phone platform of the present invention are:

The present invention uses pruning, quantization and Huffman coding to compress the deep video compression model under the condition that the loss of precision is not large, so that the model is about 1/100 times of the original, so that the video compression framework based on deep learning can be It is very convenient to deploy to mobile devices.

Description of drawings

In order to more clearly describe the working principle of the automatic spraying combined with the dust-catching net of the present invention, a schematic diagram will be attached for further explanation below.

Accompanying drawing 1 is the schematic diagram of the deep learning video compression framework used in the present invention;

Accompanying drawing 2 is the schematic diagram of index number storage of the present invention;

Accompanying drawing 3 is the schematic diagram that the present invention uses K-mean algorithm to carry out matrix quantization;

FIG. 4 is a storage diagram of the CSR sparse matrix of the present invention.

The symbols in the figure represent:

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

Please refer to Fig. 1-3, a method for implementing a deep video compression framework based on a mobile phone platform of the present invention, the implementation steps of the method are as follows:

S1. Use the tensorflow framework to build the entire video compression network, including optical Flownet, MV Encoder net, MV Decoder net, Motion Compensation Net, Residual encodernet, Residual decoder net as shown in Figure 1. These 6 networks, and then we use more than 5,000 Videos of different scenes are used to train the model, and a total of 1 million iterations are performed to obtain a trained large network. Then save the graph model and parameter information of the network.

The working process of the video compression network in Figure 1 is as follows:

S101, split the video into each frame of pictures, input the current frame and the last reconstructed frame to the optical flow network Optical FlowNet, and obtain the motion vector of the current frame;

S102, then encode the motion vector through the motion vector encoding network MV Encoder Net to obtain the encoded result,

S103, quantize the encoded result again to obtain the quantized result, as one of the contents that the current frame needs to store;

S104, the result after passing through the motion vector decoding network MV Decoder Net, that is, the reconstructed motion vector of the current frame, and the picture of the previous reconstructed frame are input into the motion compensation network Motion compensation Net to obtain the predicted frame of the current frame;

S105, using the real frame and the predicted frame to subtract, to obtain residual information r _t that the predicted frame failed to include;

S106, encode Residual encoder net, quantize Q, and entropy encode and store the residual information, then decode the Residual decoder net to obtain the reconstruction result of the residual, and then add it to the predicted frame to obtain the final reconstructed frame;

S107 , the compressed video needs to save the encoding of the motion vector after quantization in step S103 and the residual encoding after quantization in step S106 .

S2, and then prune and quantify the trained model;

Then, the trained model is pruned and quantized, and the pruning and quantization are gradually performed layer by layer. The specific steps are as follows:

S201. The first is pruning. By visualizing the trained weights of each layer, we found that the absolute value of most of the data in each layer is very small, near 0, so we prune all the data whose absolute value is less than 0.5 In this way, we will get a sparse matrix. We use the common CSR method to store the sparse matrix, and CSR stores the matrix as shown in Figure 4.

As shown in Figure 4, there is a 3*3 sparse matrix, and then the information we need to store is the retained values [1, 2, 3, 4, 5, 6], and the column index corresponding to each data [0] , 2, 2, 0, 1, 2], since we store by row, we can only value those numbers in the list that belong to one row, so we use a list to store [0, 2, 3, 6] . Indicates that 1, 2 belong to a row, 3 belongs to a row, and 4, 5, and 6 belong to a row. Therefore, in order to store this sparse matrix, we need to store a total of 2*a+n+1 data, which is much smaller than n*n. In order to compress the stored data, we store the value at the absolute position of the index [0, 2, 3, 6], and we use the relative value, so that the number of bits required to store the index number will be reduced. As shown in Figure 2, if we store the absolute value of idx, the number of bits required for a number is 4 bits. But if we store the diff, the number of bits required for a number is 3 bits. diff represents the offset of the current value from the previous value. In order to use a smaller number of bits to store the offset, here we set the maximum offset to 8, which will use 3 bits to store each offset In order to achieve this effect, we add a number 0 at the position of 12, so that when the idx is 15, the offset is 3.

Experiments show that by pruning, we can reduce the storage capacity to 1/13 of the original, and there is almost no loss in accuracy. This further proves that there is a lot of redundant information in the weights of deep learning.

S202. After pruning, we start to quantify the pruned data. Here, we use the traditional K-mean algorithm to quantify the matrix.

As shown in Figure 3, the selection of the initial value in K-means is first performed. Here we use the method based on data density, that is, the probability of selection is based on the frequency of data occurrence, and then sampling is performed. The K we use is 11, which is to choose 11 points.

Then we train using the K-mean algorithm to obtain the final 11 center points, and then cluster the data into corresponding clusters. Specifically, as shown in Figure 3, we assume that K=4 is used in the figure, and then use K-mean clustering to obtain the clusters of each data respectively. Different colors in the figure represent different clusters, such as blue 2.09, 2.12, 1.92, and 1.87 are a cluster, and we use the data 2.00 to represent them. 2.00 is the average of 4 values. Similarly for the other 3 clusters, we use the same method to process, and get 4 cluster centers 2.00, 1.50, 0.00, -1.00 respectively.

Then we only need to store these 4 numbers, each data occupies 32 bits, we also need to store each data index, but the index only needs to store 2 bits, so compared to 16 data are stored as 32 bits, Our quantized model storage is only 5/16 of the original.

After we quantize, we need to tune the model. Here we take the reverse derivative of each parameter separately, then add the derivatives of each cluster, and then use this summed gradient to quantify the parameters. Gradient descent - param-lr*gradient, as shown in Figure 3.

S3, pruning and quantization are performed separately for each layer. In order to further reduce storage, we use Huffman encoding to Huffman encode the weights in the entire network, and then store them.

Experiments show that our compressed model storage only accounts for 2/205 of the original model. For a model with more than 1G, we compress it into tens of megabytes for storage. It can be easily deployed to mobile devices.

While the preferred embodiments of the present application have been described, additional changes and modifications to these embodiments may occur to those skilled in the art once the basic inventive concepts are known. Therefore, the appended claims are intended to be construed to include the preferred embodiment and all changes and modifications that fall within the scope of this application.

Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Except for the technical features described in the specification, they are all known technologies by those skilled in the art.

Claims (8)

1. A method for realizing a depth video compression framework based on a mobile phone platform is characterized by comprising the following steps:

s1, building the whole video compression network, training the model by using videos of a plurality of different scenes to obtain a trained large network, and storing the graph model and parameter information of the network;

s2, pruning and quantifying the trained model;

s3, pruning and quantization are performed for each layer, and the weights in the entire network are huffman coded using huffman coding and then stored.

2. The method for implementing a depth video compression framework based on a mobile phone platform according to claim 1, wherein the video compression network built by using a tensoflow framework in step S1 includes 6 networks, namely, optical Flow Net, mvencorder Net, MV Decoder Net, Motion compression Net, Residual encoder Net, and Residual Decoder Net.

3. The method for implementing a depth video compression framework based on a mobile phone platform as claimed in claim 2, wherein the step S1 is as follows:

s101, splitting the video into each frame of picture, inputting the current frame and the previous reconstructed frame to an Optical flow network Optical FlowNet, and obtaining a motion vector of the current frame;

s102, coding the motion vector through a motion vector coding network MV Encoder Net to obtain a coded result,

s103, quantizing the coded result to obtain a quantized result, wherein the quantized result is used as one of contents required to be stored in the current frame;

s104, inputting the result after passing through the Motion vector decoding network MV Decoder Net, namely the reconstructed Motion vector of the current frame and the picture of the previous reconstructed frame into a Motion compensation network Motion compensation Net to obtain a predicted frame of the current frame;

s105, subtracting the real frame and the predicted frame to obtain residual error information r which cannot be included in the predicted frame_t；

S106, encoding Residual error encoder net, quantizing Q, entropy encoding and storing Residual error information, then decoding the Residual error encoder net to obtain a Residual error reconstruction result, and adding the Residual error reconstruction result and a predicted frame to obtain a final reconstruction frame;

s107, the compressed video needs to store the motion vector encoded in step S103 and the residual encoded in step S106.

4. The method for implementing a depth video compression framework based on a mobile phone platform according to claim 1, 2 or 3, wherein the step S2 includes the following steps:

s201, pruning is carried out firstly, all data with absolute value less than 0.5 are pruned by visualizing the trained weight of each layer, so as to obtain a sparse matrix, the obtained sparse matrix is stored,

the value stored by indexing this absolute position is changed to be the relative value diff which indicates the offset of the current value from the previous value, the maximum offset is set to be 8, thus 3 bits are used to store each offset, and in addition, the position of 12 is supplemented with a number of 0, so that when idx is 15, the offset is 3;

s202, after pruning is finished, quantizing the data after pruning is finished.

5. The method of claim 4, wherein in step S201, the CSR is used to store the matrix.

6. The method as claimed in claim 4, wherein in step S202, matrix quantization is performed using a conventional K-mean algorithm.

7. The method for implementing the depth video compression framework based on the mobile phone platform as claimed in claim 6, wherein in step S202, the K-mean algorithm is specifically as follows:

firstly, selecting an initial value in K-means, then sampling, wherein the used K is 11, namely 11 points are selected;

then training by using a K-mean algorithm to obtain final 11 central points, clustering data into corresponding clusters, assuming that K is 4, then using the K-mean clustering to respectively obtain clusters of each data, respectively obtaining 4 cluster centers, and then only storing the 4 data, wherein each data index also needs to be stored;

after the quantization is finished, the model needs to be optimized, each parameter is subjected to inverse derivation, the derivatives of each cluster are added, and then the quantized parameter is subjected to gradient reduction, param-lr gradient, by using the addition gradient.

8. The method of claim 1, 2, 3, 5, 6 or 7, wherein the initial value of K-means is selected by using a method based on data density, that is, the probability of selection is determined according to the frequency of occurrence of data, and then sampling is performed.

Images