CN112288142B - Short video memory prediction method and device - Google Patents

Short video memory prediction method and device Download PDF

Info

Publication number
CN112288142B
CN112288142B CN202011090457.8A CN202011090457A CN112288142B CN 112288142 B CN112288142 B CN 112288142B CN 202011090457 A CN202011090457 A CN 202011090457A CN 112288142 B CN112288142 B CN 112288142B
Authority
CN
China
Prior art keywords
matrix
frame
loss function
video
memory
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202011090457.8A
Other languages
Chinese (zh)
Other versions
CN112288142A (en
Inventor
苏育挺
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Quanzhou Jinda Intelligent Research Institute Co ltd
Original Assignee
Quanzhou Jinda Intelligent Research Institute Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Quanzhou Jinda Intelligent Research Institute Co ltd filed Critical Quanzhou Jinda Intelligent Research Institute Co ltd
Priority to CN202011090457.8A priority Critical patent/CN112288142B/en
Publication of CN112288142A publication Critical patent/CN112288142A/en
Application granted granted Critical
Publication of CN112288142B publication Critical patent/CN112288142B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/04—Forecasting or optimisation specially adapted for administrative or management purposes, e.g. linear programming or "cutting stock problem"
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F30/00—Computer-aided design [CAD]
    • G06F30/20—Design optimisation, verification or simulation
    • G06F30/27—Design optimisation, verification or simulation using machine learning, e.g. artificial intelligence, neural networks, support vector machines [SVM] or training a model
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/044—Recurrent networks, e.g. Hopfield networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • Artificial Intelligence (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Software Systems (AREA)
  • Computing Systems (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • General Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Mathematical Physics (AREA)
  • Business, Economics & Management (AREA)
  • Computational Linguistics (AREA)
  • Health & Medical Sciences (AREA)
  • Economics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Strategic Management (AREA)
  • Human Resources & Organizations (AREA)
  • Game Theory and Decision Science (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Development Economics (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Computer Hardware Design (AREA)
  • Medical Informatics (AREA)
  • Geometry (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Marketing (AREA)
  • Operations Research (AREA)
  • Quality & Reliability (AREA)
  • Tourism & Hospitality (AREA)
  • General Business, Economics & Management (AREA)
  • Image Analysis (AREA)

Abstract

The invention provides a short video memory prediction method and a device, comprising the following steps: A. extracting features; B. calculating a criticality score for each frame and applying a first penalty function
Figure DDA0002721932330000011
Constraining; C. obtain abstract video Gs(x) And calculating the reconstructability loss function in the process of generating the abstract video
Figure DDA0002721932330000012
D. Obtaining the memory fraction of each frame; E. multiplying the memory fraction of each frame with the critical score of each frame respectively and summing to obtain a predicted memory fraction and a predicted loss function
Figure DDA0002721932330000013
F. Constructing a final loss function
Figure DDA0002721932330000015
G. Training is carried out until the final loss function
Figure DDA0002721932330000014
And (6) converging. The invention combines a plurality of ideas together, better combines visual information on time and space dimensions, obtains enhanced feature representation and obtains better prediction effect.

Description

Short video memory prediction method and device
Technical Field
The invention relates to a short video memory prediction method and a short video memory prediction device.
Background
With the rapid development of multimedia information processing technology, media platforms such as social networks, media advertisements, information retrieval and recommendation systems need more powerful functions to process data that is changing day by day. Thus, the ability to understand the content of information plays a key role in such media systems. The understanding of content may be affected by different concepts of visual prominence, aesthetics, emotion, popularity, and memory, which measure how well an image or video is remembered, an emerging and less well-known concept in multimedia and computer vision.
The memory prediction has wide practical applications, such as designing impressive promotional advertisements, making impressive educational demonstration videos, and content retrieval, filtering and summarization.
The memory prediction of video involves two core problems: feature characterization and prediction models. The current research on the memory prediction focuses on exploring visual factors influencing the memory prediction, and the prediction process adopts a mode of separating feature extraction from prediction, so that the prediction performance is greatly limited by the previous feature extraction.
Disclosure of Invention
The invention aims to provide a short video memory prediction method and a short video memory prediction device aiming at the defects of the prior art, combines multiple ideas, better combines visual information on time and space dimensions, obtains enhanced feature representation and obtains better prediction effect.
The invention is realized by the following technical scheme:
a short video memory prediction method comprises the following steps:
A. extracting visual characteristic vectors of each frame from videos in a data set by using a residual error network to form a multi-dimensional visual characteristic matrix v, performing spatial pooling on the matrix v to obtain a matrix x, and equally dividing the matrix v and the matrix x into a training set and a test set;
B. taking the training set corresponding to the matrix x as the input of the bidirectional long-short term memory network to obtain the possibility data of each frame as the key frame, and calculating according to the possibility dataCalculating criticality scores for frames and applying a first penalty function
Figure GDA0003620475890000021
Constraining;
C. selecting the first s key frames with the highest key score to form a prediction abstract video, and reconstructing corresponding feature vectors through convolution operation to obtain an abstract video Gs(x) And calculating the reconstructability loss function in the process of generating the abstract video
Figure GDA0003620475890000022
D. Constructing a space attention graph S by using the matrix v, obtaining a reconstruction characteristic matrix F by using the matrix v and the space attention graph S, and inputting the reconstruction characteristic matrix F into a bidirectional long-short term memory network to obtain the memory fraction of each frame;
E. multiplying the memory fraction of each frame with the critical score of each frame respectively and summing to obtain a predicted memory fraction and a predicted loss function
Figure GDA0003620475890000023
F. Constructing a final loss function
Figure GDA0003620475890000024
Wherein λ is the equilibrium coefficient;
G. training completely end to end by using ADAM optimizer at fixed learning rate, and iteratively updating corresponding parameters until final loss function
Figure GDA0003620475890000025
And (6) converging.
Further, the step a comprises:
a1, respectively sampling each video of the data set, obtaining T frame images after each video is sampled, inputting the T frame images into a trained Resnet50, and obtaining a T frame multidimensional visual feature matrix v corresponding to each video from a conv4_3 layer of Resnet 50: v ═ v1,v2,…，vt,…，vT}；
A2, obtaining a matrix x after spatial pooling of the matrix v: x ═ x1,x2,…，xt,…，xT}；
A3, dividing the matrix v and the matrix x into a training set and a test set.
Further, the step B includes:
b1, taking a matrix x of a training set as an input of a bidirectional long-short term memory network capable of capturing timing information:
Figure GDA0003620475890000031
Figure GDA0003620475890000032
Figure GDA0003620475890000033
Figure GDA0003620475890000034
wherein T is (0, T)]，xtA visual characteristic representing the t-th frame,
Figure GDA0003620475890000035
and
Figure GDA0003620475890000036
function f representing forward and backward information of bidirectional long and short term memory network outputv_sum() Hiding state of bidirectional long-short term memory network
Figure GDA0003620475890000037
And an initial feature xtCritical score, W, mapped to Each framex、Wf、WbB is weight and bias, respectively, and tanh is activation function;
Figure GDA0003620475890000038
indicating the likelihood that the tth frame is a key frame;
b2, calculating the criticality score alpha of each frame by the following formulat：
Figure GDA0003620475890000039
B3, applying a first penalty function in order to encourage the critical scores of different frames to be able to make a gap
Figure GDA00036204758900000310
And (3) constraint:
Figure GDA00036204758900000311
further, the reconstructability loss function in the step C
Figure GDA00036204758900000312
Comprises the following steps:
Figure GDA00036204758900000313
wherein G iss(x)tAnd
Figure GDA00036204758900000314
respectively, abstract video Gs(x) And the tth frame of matrix xtCharacteristics of the frame, ft∈Λs，ΛsIs an index used for indicating s key frames for abstract video selection, | · | calving2Is the norm of L2.
Further, the step D includes:
d1, changing v to { v ═ v1,v2,…，vt,…，vTVisual characteristics v of each frame intDenoted by Z, for a given Z, it is fed into two different convolutional layers respectively to generate two new signatures P and Q, which, after reshaping, perform a matrix multiplication between the transposes of P and Q and apply the softmax function to calculate the spatial attention map S:
Figure GDA0003620475890000041
wherein i and j represent position indexes, s, respectivelyijMeasuring the influence of the ith position on the jth position, wherein N is the space size of the feature map;
d2, feeding Z into the convolutional layer to generate a new characteristic diagram O, performing matrix multiplication between O and S, multiplying by a proportional parameter beta, and performing element-by-element summation operation on the characteristic Z to obtain a reconstructed characteristic matrix F:
Figure GDA0003620475890000042
d3, performing space and channel pooling on the reconstruction characteristic matrix F, inputting the reconstruction characteristic matrix F of different frames into a bidirectional long-short term memory network to obtain the hidden state of each frame, mapping the memory fraction of each frame of image through a multilayer neural network and a single output neuron, and using ytAnd (4) showing.
Further, the step E includes:
e1, multiplying the memory fraction of each frame with the criticality score of each frame respectively and summing to obtain the predicted memory fraction
Figure GDA0003620475890000043
E2 prediction loss function
Figure GDA0003620475890000044
Is a predicted value
Figure GDA0003620475890000045
And true value mkThe error between:
Figure GDA0003620475890000046
further, the method also comprises the step H: and obtaining a final result by using the data of the test set, and evaluating the result according to the Spearman ordering correlation coefficient and the mean square error.
The invention is also realized by the following technical scheme:
a short video memory prediction apparatus comprising:
a feature extraction module: the device comprises a matrix v, a training set and a test set, wherein the matrix v is used for extracting visual characteristic vectors of each frame from videos in a data set by using a residual error network to form a multi-dimensional visual characteristic matrix v, pooling the matrix v in space to obtain a matrix x, and equally dividing the matrix v and the matrix x into the training set and the test set;
a first loss function construction module: the method is used for taking a training set of a matrix x as the input of a bidirectional long-short term memory network, obtaining probability data of each frame as a key frame, calculating a criticality score of each frame according to the probability data, and applying a first loss function
Figure GDA0003620475890000051
Constraining;
a reconstructability loss function construction module: the method is used for selecting the first s key frames with the highest key scores to form a prediction abstract video, and reconstructing corresponding feature vectors through convolution operation to obtain an abstract video Gs(x) And calculating the reconstructability loss function in the process of generating the abstract video
Figure GDA0003620475890000052
The prediction loss function building module: the device is used for constructing a space attention graph S by using the matrix v, obtaining a reconstruction characteristic matrix F by using the matrix v and the space attention graph S, and inputting the reconstruction characteristic matrix F into a bidirectional long-short term memory network to obtain the memory fraction of each frame; multiplying the memory fraction of each frame with the critical score of each frame respectively and summing to obtain a predicted memory fraction and a predicted loss function
Figure GDA0003620475890000053
A final loss function construction module: for constructing the final loss function
Figure GDA0003620475890000054
Figure GDA0003620475890000055
Training completely end-to-end at a fixed learning rate using an ADAM optimizer, iteratively updating corresponding parameters until a final loss function
Figure GDA0003620475890000056
Convergence, where λ is the equilibrium coefficient.
The invention has the following beneficial effects:
1. the invention utilizes a bidirectional long-short term memory network to obtain the key score of each frame, applies a first loss function constraint to encourage the key degrees of different frames to generate a difference, provides a differential attention degree for subsequent memory degree prediction, forms a summary video according to the key scores, obtains a reconstructive loss function according to the summary video, obtains the memory degree score of each frame through a space attention and the long-short term memory network, multiplies and sums the memory degree score of each frame and the key score of each frame respectively to obtain a predicted memory degree score, obtains a predicted loss function, better combines visual information on time and space dimensions, obtains enhanced characteristic representation, obtains a final loss function according to the first loss function, the reconstructive loss function and the predicted loss function, uses an ADAM optimizer to carry out end-to-end training, the method combines multiple ideas, adopts end-to-end learning, can obtain better prediction effect, and is particularly suitable for the memory prediction of short videos.
Drawings
The present invention will be described in further detail with reference to the accompanying drawings.
FIG. 1 is a flow chart of the present invention.
Detailed Description
As shown in fig. 1, the short video memory prediction method includes the following steps:
A. extracting visual characteristic vectors of each frame from videos in a data set by using a residual error network to form a multi-dimensional visual characteristic matrix v, performing spatial pooling on the matrix v to obtain a matrix x, and equally dividing the matrix v and the matrix x into a training set and a test set; the method specifically comprises the following steps:
a1, down-sampling each video of the data set to 2fps, adjusting the size of each frame of image to 224 x 224, obtaining a T frame image after each video is sampled, inputting the T frame image into a trained Resnet50, and obtaining a T frame multidimensional visual feature matrix v corresponding to each video from a conv4_3 layer of Resnet 50: v ═ v1,v2,…，vt,…，vT}，vtThe size is [ 14X 1024 ]]；
A2, obtaining a matrix x after spatial pooling of the matrix v: x ═ x1,x2,…，xt,…，xT}，xtThe size is [ 1X 1024 ]]；
A3, equally dividing the matrix v and the matrix x into a training set and a test set;
B. taking a training set corresponding to the matrix x as the input of a bidirectional long-short term memory network capable of capturing time sequence information to obtain probability data of each frame as a key frame, calculating the key score of each frame according to the probability data, and applying a first loss function
Figure GDA0003620475890000061
Constraining; the method specifically comprises the following steps:
b1, using the training set corresponding to the matrix x as the input of a bidirectional long-short term memory network (BilSTM network) capable of capturing the timing information:
Figure GDA0003620475890000071
Figure GDA0003620475890000072
Figure GDA0003620475890000073
Figure GDA0003620475890000074
wherein T is (0, T)]，xtA visual characteristic representing the t-th frame,
Figure GDA0003620475890000075
and
Figure GDA0003620475890000076
function f representing forward and backward information of bidirectional long and short term memory network outputv_sum() Hiding state of bidirectional long-short term memory network
Figure GDA0003620475890000077
And an initial feature xtCritical score, W, mapped to Each framex、Wf、WbB is weight and bias, respectively, and tanh is activation function;
Figure GDA0003620475890000078
represents a score, i.e., the likelihood that the tth frame is a key frame;
b2, calculating a criticality score (weight vector generated by softmax function) α of each frame by the following formulat：
Figure GDA0003620475890000079
Wherein exp represents an exponential function with a natural constant as a base;
b3, applying a first penalty function in order to encourage the critical scores of different frames to be able to make a gap
Figure GDA00036204758900000710
And (3) constraint:
Figure GDA00036204758900000711
C. selecting the first s key frames with the highest criticality score to form a prediction summary video, and reconstructing the prediction summary video through convolution operationPredicting the corresponding visual feature vector in the abstract video to obtain an abstract video Gs(x) And calculating the reconstructability loss function in the process of generating the abstract video
Figure GDA00036204758900000712
Wherein G iss(x)tAnd
Figure GDA00036204758900000713
respectively, abstract video Gs(x) And the tth frame of matrix xtFeatures of the frame, whereint∈Λs，ΛsIs an index, s key frames for indicating the selection of the summarized video, | · | | luminous flux2Is the norm of L2;
D. constructing a space attention diagram S by using the matrix v, obtaining a reconstruction characteristic matrix F (the reconstruction characteristic matrix F is a new transformed matrix v) by using the matrix v and the space attention diagram S, and inputting the reconstruction characteristic matrix F into a bidirectional long-short term memory network (BilSTM network) to obtain the memory fraction of each frame; the method specifically comprises the following steps:
d1, for convenience of description, will v ═ v1,v2,…，vt,…，vTVisual characteristics v of each frame intDenoted by Z, for a given Z, it is fed into two different convolutional layers, i.e. conv4_3 layer of Resnet50, respectively, to generate two new signatures P and Q, after unfolding the two signatures P and Q into vectors, performing a matrix multiplication between the transposes of P and Q, and applying the softmax function to calculate the spatial attention S:
Figure GDA0003620475890000081
wherein i and j represent position indexes, s, respectivelyijMeasuring the influence of the ith position on the jth position, wherein N is the space size of the feature map, and the space size of the feature map is the length multiplied by the width of the feature map;
d2, feeding Z into a convolutional layer to generate a new feature map O, where the convolutional layer is the conv4_3 layer of Resnet50, performing matrix multiplication between O and S, multiplying by the scaling parameter β, and performing element-by-element summation with the feature Z to obtain a reconstructed feature matrix F:
Figure GDA0003620475890000082
d3, performing space and channel pooling on the reconstruction feature matrix F, inputting the reconstruction feature matrix F of different frames into a bidirectional long-short term memory network (BilSTM network) to obtain the hidden state of each frame, mapping the memory fraction of each frame image through a multilayer neural network and a single output neuron, and using ytRepresents;
E. multiplying the memory fraction of each frame with the critical score of each frame respectively and summing to obtain a predicted memory fraction and a predicted loss function
Figure GDA0003620475890000083
The method specifically comprises the following steps:
e1, multiplying the memory fraction of each frame with the criticality score of each frame respectively and summing to obtain the predicted memory fraction
Figure GDA0003620475890000084
E2 prediction loss function
Figure GDA0003620475890000085
Is a predicted value
Figure GDA0003620475890000086
With the true value mkThe error between:
Figure GDA0003620475890000087
the real value is a known number, when a data set is given, the real value is given, and the data set comprises a video and a label corresponding to the video, wherein the label is the real value;
F. build FinalLoss function
Figure GDA0003620475890000091
Wherein, λ is a balance coefficient and is a self-defined parameter greater than zero;
G. training completely end-to-end at a fixed learning rate using an ADAM optimizer, iteratively updating corresponding parameters until a final loss function
Figure GDA0003620475890000092
Converging; the learning rate and the end-to-end are the existing terms in the field, the learning rate can be regarded as a self-defined parameter, and the specific value needs to be determined by combining a specific task; the corresponding parameters refer to the parameters involved in the method;
H. and obtaining a final result (namely a result predicted by the model) by using the data of the test set, and evaluating the result according to the Spearman ordering correlation coefficient and the mean square error.
A short video memory prediction apparatus comprising:
a feature extraction module: the system comprises a video acquisition module, a data processing module, a training set and a test set, wherein the data acquisition module is used for acquiring a multi-dimensional visual characteristic matrix v by extracting visual characteristic vectors of each frame from videos in a data set by using a residual error network, obtaining a matrix x after spatial pooling the matrix v, and equally dividing the matrix v and the matrix x into a training set and a test set;
a first loss function construction module: the method is used for taking a training set of a matrix x as the input of a bidirectional long-short term memory network, obtaining probability data of each frame as a key frame, calculating a criticality score of each frame according to the probability data, and applying a first loss function
Figure GDA0003620475890000093
Constraining;
a reconstructability loss function construction module: the method is used for selecting the first s key frames with the highest key scores to form a prediction abstract video, and reconstructing corresponding feature vectors through convolution operation to obtain an abstract video Gs(x) And calculating the reconstructability loss function in the process of generating the abstract video
Figure GDA0003620475890000094
The prediction loss function building module: the device is used for constructing a space attention graph S by using the matrix v, obtaining a reconstruction characteristic matrix F by using the matrix v and the space attention graph S, and inputting the reconstruction characteristic matrix F into a bidirectional long-short term memory network to obtain the memory fraction of each frame; multiplying the memory fraction of each frame with the critical score of each frame respectively and summing to obtain a predicted memory fraction and a predicted loss function
Figure GDA0003620475890000095
A final loss function construction module: for constructing the final loss function
Figure GDA0003620475890000101
Figure GDA0003620475890000102
Training completely end-to-end at a fixed learning rate using an ADAM optimizer, iteratively updating corresponding parameters until a final loss function
Figure GDA0003620475890000103
Convergence, where λ is the equilibrium coefficient.
The above description is only a preferred embodiment of the present invention, and therefore should not be taken as limiting the scope of the invention, which is defined by the appended claims and their equivalents and modifications within the scope of the description.

Claims (8)

1. A short video memory prediction method is characterized in that: the method comprises the following steps:
A. extracting visual characteristic vectors of each frame from videos in a data set by using a residual error network to form a multi-dimensional visual characteristic matrix v, performing spatial pooling on the matrix v to obtain a matrix x, and equally dividing the matrix v and the matrix x into a training set and a test set;
B. taking the training set corresponding to the matrix x as the input of the bidirectional long-short term memory network to obtain that each frame is offLikelihood data for key frames, and calculating a criticality score for each frame based on the likelihood data, and applying a first penalty function
Figure FDA0003620475880000011
Constraining;
C. selecting the first s key frames with the highest key score to form a prediction abstract video, and reconstructing corresponding feature vectors through convolution operation to obtain an abstract video Gs(x) And calculating the reconstructability loss function in the process of generating the abstract video
Figure FDA0003620475880000012
D. Constructing a space attention graph S by using the matrix v, obtaining a reconstruction characteristic matrix F by using the matrix v and the space attention graph S, and inputting the reconstruction characteristic matrix F into a bidirectional long-short term memory network to obtain the memory fraction of each frame;
E. multiplying the memory fraction of each frame with the critical score of each frame respectively and summing to obtain a predicted memory fraction and a predicted loss function
Figure FDA0003620475880000013
F. Constructing a final loss function
Figure FDA0003620475880000014
Wherein λ is the equilibrium coefficient;
G. training completely end to end by using ADAM optimizer at fixed learning rate, and iteratively updating corresponding parameters until final loss function
Figure FDA0003620475880000015
And (6) converging.
2. The method of claim 1, wherein the short video memory prediction method comprises: the step A comprises the following steps:
a1, sampling each video of the data set respectivelyAnd after sampling each video, obtaining a T frame image, inputting the T frame image into a trained Resnet50, and obtaining a T frame multidimensional visual feature matrix v corresponding to each video from a conv4_3 layer of Resnet 50: v ═ v1,v2,…，vt,…，vT}；
A2, obtaining a matrix x after spatial pooling of the matrix v: x ═ x1,x2,…，xt,…，xT}；
A3, dividing the matrix v and the matrix x into a training set and a test set.
3. The method of claim 2, wherein the short video memory prediction method comprises: the step B comprises the following steps:
b1, and taking a matrix x of the training set as an input of a bidirectional long-short term memory network capable of capturing timing information:
Figure FDA0003620475880000021
Figure FDA0003620475880000022
Figure FDA0003620475880000023
Figure FDA0003620475880000024
wherein T is (0, T)]，xtA visual characteristic representing the t-th frame,
Figure FDA0003620475880000025
and
Figure FDA0003620475880000026
indicating two-way lengthForward and backward information output by short-term memory network, function fv_sum() Hiding state of bidirectional long-short term memory network
Figure FDA0003620475880000027
And an initial feature xtCritical score, W, mapped to Each framex、Wf、WbB is weight and bias, respectively, and tanh is activation function;
Figure FDA0003620475880000028
indicating the likelihood that the tth frame is a key frame;
b2, calculating the criticality score alpha of each frame by the following formulat：
Figure FDA0003620475880000029
B3, applying a first penalty function in order to encourage the critical scores of different frames to be able to make a gap
Figure FDA00036204758800000210
And (3) constraint:
Figure FDA00036204758800000211
4. the method of claim 3, wherein the short video memory prediction method comprises: the reconstructability loss function in step C
Figure FDA00036204758800000212
Comprises the following steps:
Figure FDA00036204758800000213
wherein G iss(x)tAnd
Figure FDA00036204758800000214
respectively, abstract video Gs(x) And the tth frame of matrix xtCharacteristics of the frame, ft∈Λs，ΛsIs an index, s key frames for indicating the selection of the summarized video, | · | | luminous flux2Is the norm of L2.
5. The method of claim 4, wherein the method further comprises: the step D comprises the following steps:
d1, changing v to { v ═ v1,v2,…，vt,…，vTVisual characteristics v of each frame intDenoted by Z, for a given Z, it is fed into two different convolutional layers respectively to generate two new signatures P and Q, which, after reshaping, perform a matrix multiplication between the transposes of P and Q, and apply the softmax function to calculate the spatial attention map S:
Figure FDA0003620475880000031
wherein i and j represent position indexes, s, respectivelyijThe influence of the ith position on the jth position is measured, and N is the space size of the feature map;
d2, feeding Z into the convolutional layer to generate a new characteristic diagram O, performing matrix multiplication between O and S, multiplying by a proportional parameter beta, and performing element-by-element summation operation on the characteristic Z to obtain a reconstructed characteristic matrix F:
Figure FDA0003620475880000032
d3, performing space and channel pooling on the reconstruction characteristic matrix F, inputting the reconstruction characteristic matrix F of different frames into a bidirectional long-short term memory network to obtain the hidden state of each frame, mapping the memory fraction of each frame of image through a multilayer neural network and a single output neuron, and using ytAnd (4) showing.
6. The method of claim 5, wherein the short video memory prediction method comprises: the step E comprises the following steps:
e1, multiplying the memory score of each frame and the criticality score of each frame respectively and summing to obtain the predicted memory score
Figure FDA0003620475880000033
E2 prediction loss function
Figure FDA0003620475880000034
Is a predicted value
Figure FDA0003620475880000035
With the true value mkThe error between:
Figure FDA0003620475880000036
7. a method for short video memory prediction according to any of claims 1-6, characterized by: further comprising the step H: and obtaining a final result by using the data of the test set, and evaluating the result according to the Spearman ordering correlation coefficient and the mean square error.
8. A short video memory prediction apparatus, characterized by: the method comprises the following steps:
a feature extraction module: the device comprises a matrix v, a training set and a test set, wherein the matrix v is used for extracting visual characteristic vectors of each frame from videos in a data set by using a residual error network to form a multi-dimensional visual characteristic matrix v, pooling the matrix v in space to obtain a matrix x, and equally dividing the matrix v and the matrix x into the training set and the test set;
a first loss function construction module: the method is used for taking a training set of a matrix x as the input of a bidirectional long-short term memory network, obtaining probability data of each frame as a key frame, calculating a criticality score of each frame according to the probability data, and applying a first loss function
Figure FDA0003620475880000041
Constraining;
a reconstructability loss function construction module: the method is used for selecting the first s key frames with the highest key scores to form a prediction abstract video, and reconstructing corresponding feature vectors through convolution operation to obtain an abstract video Gs(x) And calculating the reconstructability loss function in the process of generating the abstract video
Figure FDA0003620475880000042
The prediction loss function building module: the device is used for constructing a space attention graph S by using the matrix v, obtaining a reconstruction characteristic matrix F by using the matrix v and the space attention graph S, and inputting the reconstruction characteristic matrix F into a bidirectional long-short term memory network to obtain the memory fraction of each frame; multiplying the memory fraction of each frame with the critical score of each frame respectively and summing to obtain a predicted memory fraction and a predicted loss function
Figure FDA0003620475880000043
A final loss function construction module: for constructing the final loss function
Figure FDA0003620475880000044
Figure FDA0003620475880000045
Training completely end-to-end at a fixed learning rate using an ADAM optimizer, iteratively updating corresponding parameters until a final loss function
Figure FDA0003620475880000046
Convergence, where λ is the equilibrium coefficient.
CN202011090457.8A 2020-10-13 2020-10-13 Short video memory prediction method and device Active CN112288142B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011090457.8A CN112288142B (en) 2020-10-13 2020-10-13 Short video memory prediction method and device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011090457.8A CN112288142B (en) 2020-10-13 2020-10-13 Short video memory prediction method and device

Publications (2)

Publication Number Publication Date
CN112288142A CN112288142A (en) 2021-01-29
CN112288142B true CN112288142B (en) 2022-06-10

Family

ID=74496682

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011090457.8A Active CN112288142B (en) 2020-10-13 2020-10-13 Short video memory prediction method and device

Country Status (1)

Country Link
CN (1) CN112288142B (en)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114120166B (en) * 2021-10-14 2023-09-22 北京百度网讯科技有限公司 Video question and answer method, device, electronic equipment and storage medium
CN115205745B (en) * 2022-07-15 2026-01-27 中国传媒大学 Video memory prediction method, apparatus, device and storage medium
CN116344061B (en) * 2023-03-29 2026-03-17 天津大学 Amnesia Detection Device Based on Visual Attention Representation

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10311913B1 (en) * 2018-02-22 2019-06-04 Adobe Inc. Summarizing video content based on memorability of the video content
CN111062284A (en) * 2019-12-06 2020-04-24 浙江工业大学 Visual understanding and diagnosing method of interactive video abstract model

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10311913B1 (en) * 2018-02-22 2019-06-04 Adobe Inc. Summarizing video content based on memorability of the video content
CN111062284A (en) * 2019-12-06 2020-04-24 浙江工业大学 Visual understanding and diagnosing method of interactive video abstract model

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
Object Memorability Prediction using Deep Learning: Location and Size;Sathisha Basavaraju 等;《Journal of Visual Communication and Image Representation》;20190108;第117-127页 *
基于多特征表征学习的多媒体数据预测方法研究;井佩光;《中国优秀博硕士学位论文全文数据库(博士)》;20191015;全文 *

Also Published As

Publication number Publication date
CN112288142A (en) 2021-01-29

Similar Documents

Publication Publication Date Title
CN111091045B (en) Sign language identification method based on space-time attention mechanism
CN116580257B (en) Feature fusion model training and sample retrieval method, device and computer equipment
CN111246256B (en) Video recommendation method based on multimodal video content and multi-task learning
CN114419351B (en) Image-text pre-training model training and image-text prediction model training method and device
CN112288142B (en) Short video memory prediction method and device
CN114862844A (en) Infrared small target detection method based on feature fusion
CN116933051A (en) Multi-mode emotion recognition method and system for modal missing scene
CN116756314B (en) GCN-based aspect-level multi-mode emotion analysis method
CN113064968A (en) Social media emotion analysis method and system based on tensor fusion network
CN112132770A (en) Image restoration method and device, computer readable medium and electronic equipment
CN114723787A (en) Optical flow calculation method and system
CN113850182A (en) Action recognition method based on DAMR_3DNet
CN115496989B (en) Generator, generator training method and method for avoiding image coordinate adhesion
CN108985899A (en) Recommended method, system and storage medium based on CNN-LFM model
CN118334549A (en) Short video label prediction method and system for multi-mode collaborative interaction
Gao A two-channel attention mechanism-based MobileNetV2 and bidirectional long short memory network for multi-modal dimension dance emotion recognition
CN117009560A (en) Image processing methods, devices, equipment and computer storage media
WO2023185320A1 (en) Cold start object recommendation method and apparatus, computer device and storage medium
CN111325068A (en) Video description method and device based on convolutional neural network
CN118014860A (en) A multi-source and multi-scale image fusion method and device based on attention mechanism
CN113011334A (en) Video description method based on graph convolution neural network
CN117251622A (en) Object recommended methods, devices, computer equipment and storage media
CN119580034A (en) Training method for generating picture description model, picture description generation method, device, equipment, medium and program product
CN120658789A (en) Information pushing method, device, computer equipment and storage medium
CN117033804B (en) A Click-Induced Detection Method Guided by Subjective and Objective Perspectives

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant