CN110147722B - Video processing method, video processing device and terminal equipment - Google Patents
Video processing method, video processing device and terminal equipment Download PDFInfo
- Publication number
- CN110147722B CN110147722B CN201910288788.3A CN201910288788A CN110147722B CN 110147722 B CN110147722 B CN 110147722B CN 201910288788 A CN201910288788 A CN 201910288788A CN 110147722 B CN110147722 B CN 110147722B
- Authority
- CN
- China
- Prior art keywords
- video
- target
- processed
- frame
- video frame
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/20—Scenes; Scene-specific elements in augmented reality scenes
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/46—Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
- G06V30/10—Character recognition
- G06V30/19—Recognition using electronic means
- G06V30/192—Recognition using electronic means using simultaneous comparisons or correlations of the image signals with a plurality of references
- G06V30/194—References adjustable by an adaptive method, e.g. learning
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Multimedia (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Cheminformatics (AREA)
- General Engineering & Computer Science (AREA)
- Artificial Intelligence (AREA)
- Signal Processing (AREA)
- Databases & Information Systems (AREA)
- Television Signal Processing For Recording (AREA)
- Image Analysis (AREA)
Abstract
The invention is applicable to the technical field of image processing, and provides a video processing method, a video processing device and terminal equipment, wherein the video processing method comprises the steps of obtaining a video to be processed; the method comprises the steps of determining a scene category of each video frame in a video to be processed, carrying out target recognition on each video frame in the video to be processed through a first depth learning model after training according to the scene category of each video frame to obtain a target recognition result, wherein the target recognition result indicates whether each video frame contains a target object or not, the target objects correspond to the scene category, identifying a plurality of target frames of the video to be processed according to the target recognition result, wherein the target frames contain the target objects, and sequentially combining the target frames according to time sequence of each target frame in the video to be processed to obtain the target video.
Description
Technical Field
The present invention relates to the field of image processing technologies, and in particular, to a video processing method, a video processing device, and a terminal device.
Background
Currently, video image technology is widely used in various fields (such as video monitoring, mobile terminals, social platforms, etc.), and a large number of video files are generated during the application process. These video files often contain a large number of redundant or unimportant parts and the duration of these video files is often long, resulting in a long time being required for the user to query the video files for the required key information. For example, a user needs to query a video part of an individual in a monitoring video with a duration of 24 hours for executing a specified activity, and currently, the user may take a way to fast forward play a video file, and may mark a time corresponding to the video part of the observed individual in executing the specified activity, so as to find a required video part from the video file according to the corresponding time in subsequent use. However, due to the existence of a large amount of redundant or unimportant parts in the video file, the method of querying the key information in the video takes a long time, and it is difficult for a user to efficiently extract some key information in the video.
Disclosure of Invention
In view of this, embodiments of the present invention provide a video processing method, a video processing apparatus, and a terminal device, which can identify and extract key information in a video (for example, identify and extract a video portion including a target individual), thereby improving the efficiency of a user obtaining key information in a video.
A first aspect of an embodiment of the present invention provides a video processing method, including:
acquiring a video to be processed;
Determining a scene category of each video frame in the video to be processed;
According to the scene category of each video frame, carrying out target recognition on each video frame in the video to be processed through a trained first depth learning model to obtain a target recognition result, wherein the target recognition result indicates whether each video frame contains a target object or not, and the target object corresponds to the scene category;
Identifying a plurality of target frames of the video to be processed according to the target identification result, wherein the target frames contain the target object;
and sequentially combining a plurality of target frames according to the time sequence of each target frame in the video to be processed to obtain the target video.
A second aspect of an embodiment of the present invention provides a video processing apparatus, including:
the acquisition module is used for acquiring the video to be processed;
the determining module is used for determining the scene category of each video frame in the video to be processed;
the first recognition module is used for carrying out target recognition on each video frame in the video to be processed through a first trained deep learning model according to the scene category of each video frame to obtain a target recognition result, wherein the target recognition result indicates whether each video frame contains a target object or not, and the target object corresponds to the scene category;
the second recognition module is used for recognizing a plurality of target frames of the video to be processed according to the target recognition result, wherein the target frames contain the target object;
and the processing module is used for sequentially combining a plurality of target frames according to the time sequence of each target frame in the video to be processed to obtain the target video.
A third aspect of the embodiments of the present invention provides a terminal device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, the processor implementing the steps of the method as described above when executing the computer program.
A fourth aspect of the embodiments of the present invention provides a computer-readable storage medium storing a computer program which, when executed by a processor, implements the steps of the method as described above.
Compared with the prior art, the method and the device have the advantages that the video to be processed is obtained, the scene category of each video frame in the video to be processed is determined, target recognition is carried out on each video frame in the video to be processed through a trained first depth learning model according to the scene category of each video frame, a target recognition result is obtained, the target recognition result indicates whether each video frame contains a target object or not, the target objects correspond to the scene categories, a plurality of target frames of the video to be processed are recognized according to the target recognition result, the target objects are contained in the target frames, and the target frames are sequentially combined according to the time sequence of each target frame in the video to be processed, so that the target video is obtained. According to the embodiment of the invention, by determining the scene category of each video frame and determining whether the target object corresponding to the scene category exists in each video frame through the trained deep learning model, the targeted target recognition can be carried out according to different scene categories, so that the target recognition result is more targeted and has less interference, a plurality of target frames of the video to be processed are recognized according to the target recognition result, the target video is obtained according to the target frames, key information (such as image parts containing target individuals) in the video to be processed can be accurately and efficiently extracted, the target video composed of the key information is obtained, redundant and unimportant video content in the video to be processed is deleted, and valuable video content is reserved, so that in application scenes such as searching for the key information of the monitoring video, a user can carry out more efficient and targeted processing according to the target video without manually searching, clipping, re-synthesizing and other operations on the key information. The embodiment of the invention greatly improves the efficiency of acquiring the key information in the video by the user, and has strong practicability and usability.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed in the embodiments or the description of the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and that other drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a schematic flow chart of an implementation of a video processing method according to a first embodiment of the present invention;
fig. 2 is a schematic diagram of an implementation flow of a video processing method according to a second embodiment of the present invention;
fig. 3 is a schematic diagram of a video processing apparatus according to a third embodiment of the present invention;
fig. 4 is a schematic diagram of a terminal device according to a fourth embodiment of the present invention.
Detailed Description
In the following description, for purposes of explanation and not limitation, specific details are set forth such as the particular system architecture, techniques, etc., in order to provide a thorough understanding of the embodiments of the present invention. It will be apparent, however, to one skilled in the art that the present invention may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present invention with unnecessary detail.
It should be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It is also to be understood that the terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
It should be further understood that the term "and/or" as used in the present specification and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes such combinations.
As used in this specification and the appended claims, the term "if" may be interpreted as "when..once" or "in response to a determination" or "in response to detection" depending on the context. Similarly, the phrase "if a determination" or "if a [ described condition or event ] is detected" may be interpreted in the context of meaning "upon determination" or "in response to determination" or "upon detection of a [ described condition or event ]" or "in response to detection of a [ described condition or event ]".
In order to illustrate the technical scheme of the invention, the following description is made by specific examples.
Fig. 1 is a schematic flowchart of an implementation of a video processing method according to a first embodiment of the present invention, where the video processing method shown in fig. 1 may include the following steps:
Step S101, obtaining a video to be processed.
The video to be processed may be a file subjected to audio/video encoding, and the encoding mode of the video to be processed is known, and the encoding mode is an encoding mode contained in training data, where the training data is used for training a deep learning model.
Of course, the video to be processed may also be video that is not encoded, where each frame of image contains complete image pixel information.
Step S102, determining a scene category of each video frame in the video to be processed.
In the embodiment of the invention, the scene category of the video frame can be determined in various modes. For example, the scene category of the video frame may be determined through a preset depth learning model, and it should be noted that the preset depth learning model may be the same as or different from the trained first depth learning model in step S103, and the preset depth learning model may detect and determine the scene category of each video frame in the video to be processed one by one, or may determine the subject scene category of the video to be processed according to the scene category of the video frame detected by one or more frames (for example, if the scene categories of the video frames with a ratio greater than a preset ratio are all scene category a, determine the subject scene category of the video to be processed as scene category a), and use the subject scene category as the scene category of each video frame of the video to be processed. In addition, the scene category of each video frame in the video to be processed can be determined according to the name, the theme or the preset label of the video to be processed.
Step S103, carrying out target recognition on each video frame in the video to be processed through a trained first depth learning model according to the scene category of each video frame to obtain a target recognition result, wherein the target recognition result indicates whether each video frame contains a target object or not, and the target object corresponds to the scene category.
In the embodiment of the invention, different scene categories can correspond to different target objects. For example, if the scene category of the video frame is determined to be an airport scene, the target object corresponding to the airport scene may include objects such as an airplane, a person, other mechanical equipment (such as a small-sized transportation device in the airport), other flying objects, and the like, and if the scene category of the video frame is determined to be a zoo scene, the target object corresponding to the zoo scene may include animals, people, and the like. The correspondence between the scene category and the target object may be preset.
By way of example, the first deep learning model may include, for example, resNet models, R-CNN models, fast R-CNN models, and the like, and the first deep learning model may be of various types, not limited herein.
Optionally, the target recognition result may include information such as a type of the target object included in the video frame, a position of the target object in the video frame, and the like.
Optionally, before performing object recognition on each video frame in the video to be processed through the trained first deep learning model, training the first deep learning model is further included.
Specifically, the step of training the first deep learning model may include:
Acquiring training data, wherein the training data comprises at least one training video and identification information corresponding to the training video, and the identification information can comprise accurate target object information (such as the type and the position of a target object) in the training video;
Detecting the training data through the first deep learning model, and obtaining a detection result;
And adjusting parameters of the first deep learning model according to the detection result until the detection result obtained through the adjusted first deep learning model meets a preset condition (such as that the corresponding value of the loss function is smaller than a preset threshold value, etc.), and taking the adjusted first deep learning model as a trained first deep learning model.
Step S104, identifying a plurality of target frames of the video to be processed according to the target identification result, wherein the target frames contain the target object.
The target frame may indicate key information in the video to be processed, and a specific identification manner of the target frame of the video to be processed may be set according to a requirement of an actual application scene. For example, it may be determined that any one frame contains a video frame of the target object as the target frame. In addition, the difference between the video frames containing the target object may be sequentially determined according to the target object information and according to the time sequence (the difference may include the difference of the target object and the difference for the non-target object may be ignored), and the target frame of the video to be processed may be determined according to the difference. Illustratively, the differences between video frames containing the target object may be identified by more conventional methods. For example, for each video frame, if the location of the target object in the video frame, the size of the image area where the target object is located, etc. changes relative to the previous video frame or the next video frame, or if the location of the target object in a certain video frame, the size of the area of the image area, etc. of the target object in the previous video frame or the next video frame meets a specified preset condition (for example, the location offset exceeds a preset offset threshold, the ratio of the difference of the areas of the image areas in two frames of video frames to the area of one of the image areas exceeds a preset ratio threshold, etc.), determining that the video frame is the target frame until all the video frames in the video to be processed are traversed. Of course, it is also possible to determine whether each video frame is a target frame by a preset classifier. Illustratively, the classifier may comprise a decision tree, logistic regression, naive bayes, neural networks, etc. algorithm.
One implementation of step S104 is described below with a specific example.
For example, if it is determined that the scene category of each frame of video frame of the video to be processed is an airport scene, the object corresponding to the airport scene is an airplane, and if the video to be processed includes 20 frames of video frames, the first 10 frames of video frames do not include an airplane, the first 10 frames of video frames are determined to be non-target frames, and in the video frames of frames 11 to 20 including an airplane, it is determined that the positions of the airplane in frames 15 to 20 are respectively shifted relative to the positions of the airplane in the video frames of the previous frame, and then the frames 14 to 20 are determined to be target frames of the video to be processed.
Step S105, sequentially combining a plurality of target frames according to the time sequence of each target frame in the video to be processed, so as to obtain the target video.
In the embodiment of the invention, the target video can only contain the target frames in the video to be processed, so that redundant and unimportant video content in the video to be processed can be considered to be deleted in the target video, and valuable video content is reserved, thereby in application scenes such as searching for key information of a monitoring video, a user can perform more efficient and targeted processing according to the target video without operations such as searching for key information, clipping and resynthesizing and the like manually. In the target video, the time node information corresponding to the target frame in the video to be processed may be reserved, or may not be reserved.
Furthermore, optionally, after obtaining the target video, the method may further include:
And storing the target video into a designated storage space.
In the embodiment of the present invention, the storage space may be preset, or after the target video is obtained, the designated storage space may be determined according to the instruction operation of the user, and then the target video is stored in the designated storage space.
Since the video to be processed often contains a large number of redundant or unimportant parts, and these parts occupy a large amount of space and resources, the problems of resource occupation and waste cannot be well improved even by the existing file compression and other modes. Therefore, in the prior art, the video to be processed is directly stored, which wastes a great deal of storage resources. In the embodiment of the invention, the target video can be stored, so that the space required for storing the video is greatly reduced, and the problem of overlarge storage resource burden caused by overlarge video is solved.
Optionally, if the video frame of the video to be processed includes a target object, the target recognition result further indicates an area and/or a feature point position where the target object is located;
correspondingly, the identifying a plurality of target frames of the video to be processed according to the target identification result includes:
judging whether each video frame in the video to be processed meets a first preset condition according to the region and/or the feature point position of the target object indicated by the target identification result;
and determining the video frame meeting the first preset condition as the target frame of the video to be processed.
In the embodiment of the present invention, the area and/or the feature point position where the target object is located may include at least one of information such as an area of an image area where the target object is located, an upper left corner coordinate, a lower right corner coordinate, a center point coordinate, and a feature point coordinate of the target object. The feature points may be determined according to the target object and the application scene, for example, if the target object is a face, the feature points may include feature points of facial five sense organs. The first preset condition may be preset by a user, and the first preset condition may be various, for example, if the feature point coordinates of the target object in the video frame, the area of the image area where the target object is located, etc. change relative to the previous video frame or the next video frame, or if the degree of difference between the feature point coordinates of the target object in a certain video frame, the area of the image area, etc. and the previous video frame or the next video frame accords with a specified preset condition (for example, the positioning offset exceeds a preset offset threshold, the ratio of the difference of the areas of the image areas in two video frames to the area of one of the image areas exceeds a preset ratio threshold, etc.).
Optionally, the determining, according to the position of the target object indicated by the target recognition result, whether each video frame in the video to be processed meets a first preset condition includes:
For each video frame containing the target object in the video to be processed, judging whether the difference degree of the target object in the video frame relative to the region and/or the characteristic point position of the target object in the previous video frame or the next video frame of the video frame accords with a first preset condition according to the region and/or the characteristic point position of the target object indicated by the target identification result;
Correspondingly, the determining the video frame meeting the first preset condition as the target frame of the video to be processed includes:
And if the difference degree of the target object in the video frame relative to the position of the region and/or the characteristic point of the target object in the previous video frame or the next video frame of the video frame meets a first preset condition, taking the video frame as the target frame of the video to be processed.
In an embodiment of the present invention, the degree of difference between the area where the target object is located and/or the position of the feature point may include one or more of a degree of shift of the feature point, a degree of change of the image area, and the like, where the degree of change of the image area may include a degree of change of an area of the image area, or may include a ratio of a portion where a pixel value changes to the image area in the image area. At this time, the first preset condition may be, for example, that an offset of the feature point coordinates of the target object exceeds a preset offset threshold, that a ratio of a difference in areas of the image areas in the two frames of video frames to an area of one of the image areas exceeds a preset ratio threshold, and so on. The first preset condition may be set according to a specific application scenario, which is not limited herein.
Optionally, the determining a scene category of each video frame in the video to be processed includes:
Performing scene recognition on each video frame in the video to be processed through the trained second deep learning model to determine the scene category of each video frame in the video to be processed;
Or alternatively
And determining the scene category of each video frame in the video to be processed according to the name, the theme or the preset label of the video to be processed.
In the embodiment of the present invention, the second deep learning model may be the same as or different from the first deep learning model. For example, if the second deep learning model is the same as the first deep learning model, the second deep learning model (or the first deep learning model) may include two cascaded deep learning sub-models, and determine a scene category of each video frame in the video to be processed and perform object recognition on each video frame in the video to be processed through the cascaded deep learning sub-models, respectively. Assuming that the deep learning submodels are a first deep learning submodel and a second deep learning submodel respectively, the video to be processed can be input into the first deep learning submodel to obtain a first output result of the first deep learning submodel, the output result indicates to identify a scene of each video frame in the video to be processed, then the first output result of the first deep learning submodel is input into the second deep learning submodel to obtain a second output result of the second deep learning submodel, and a target identification result can be obtained according to the second output result. Or the second deep learning model may include only one convolutional neural network model, and obtain the scene category and the target recognition result of each video frame through the convolutional neural network model.
By way of example, the second deep learning model may include, for example, resNet models, R-CNN models, fast R-CNN models, and the like, and the second deep learning model may be of various types, not limited herein.
Optionally, after identifying the plurality of target frames of the video to be processed, the method further includes:
The target frame is identified in the video to be processed.
For example, one or more of time node information, scene category information, and object information (such as type, feature point, location, image area, etc. of the object) corresponding to the object frame may be identified in the video to be processed. The identification manner may be various, for example, the image area of the target object may be identified by a rectangular frame in a specified form, a time node corresponding to the target frame may be identified in a time progress bar of the video to be processed, and so on.
Optionally, when all the continuous multi-frame video frames of the video to be processed are target frames, a first frame and a last frame in the continuous multi-frame video frames can be identified.
According to the embodiment of the invention, by determining the scene category of each video frame and determining whether the target object corresponding to the scene category exists in each video frame through the trained deep learning model, the targeted target recognition can be carried out according to different scene categories, so that the target recognition result is more targeted and has less interference, a plurality of target frames of the video to be processed are recognized according to the target recognition result, the target video is obtained according to the target frames, key information (such as image parts containing target individuals) in the video to be processed can be accurately and efficiently extracted, the target video composed of the key information is obtained, redundant and unimportant video content in the video to be processed is deleted, and valuable video content is reserved, so that in application scenes such as searching for the key information of the monitoring video, a user can carry out more efficient and targeted processing according to the target video without manually searching, clipping, re-synthesizing and other operations on the key information. The embodiment of the invention greatly improves the efficiency of the user to acquire the key information of the video, and has strong practicability and usability.
On the basis of the foregoing embodiment, fig. 2 is a schematic flowchart of an implementation flow of a video processing method according to a second embodiment of the present invention, where the video processing method shown in fig. 2 may include the following steps:
step S201, obtaining a video to be processed;
step S202, determining scene category of each video frame in the video to be processed;
Step S203, performing target recognition on each video frame in the video to be processed through a trained first depth learning model according to the scene category of each video frame to obtain a target recognition result, wherein the target recognition result indicates whether each video frame contains a target object or not, and the target object corresponds to the scene category;
Step S204, identifying a plurality of target frames of the video to be processed according to the target identification result, wherein the target frames contain the target object;
Step S205, according to the time sequence of each target frame in the video to be processed, combining a plurality of target frames in sequence to obtain the target video;
The steps S201, S202, S203, S204, S205 in this embodiment are the same as the steps S101, S102, S103, S104, S105 described above, and the detailed descriptions of the steps S101, S102, S103, S104, S105 will be omitted herein.
Step S206, according to the target frame, establishing an information index.
In an embodiment of the present invention, the information index may include a list of links or pointers to the target frame and/or to information related to the target frame. The information index may provide a pointer or link to content such as designated data or nodes, so that information related to the target frame is quickly accessed through the information index. The information index may be provided at an interface associated with the video to be processed and/or with the target video, and the information index may point to the video to be processed and/or the target video. In addition, it should be noted that the information index may be embedded in the video to be processed or the target video, or may be an information interface (such as a database, a table, a text, etc.) other than the video to be processed and the target video, and the display manner may be various.
Optionally, the establishing an information index according to the target frame includes:
An information index list about the target frame is established, wherein the information index list comprises at least one of time information of the target frame in the video to be processed, scene category information of the target frame and target object information of the target frame.
According to the embodiment of the invention, the time information can indicate the time node corresponding to the target frame in the video to be processed, and the time position of the target frame in the time axis of the video to be processed can be known through the time information, so that a user can know the condition of the target frame in different time reference frames through the time information, and the original time node information is not lost after the target frame is extracted from the video to be processed. The scene category information may indicate a scene category of the target frame, and the target object information may include information related to the target object, such as a type, a position, an image area, and the like of the target object.
According to the embodiment of the invention, the information index is established according to the target frame, so that the efficient retrieval of the related information of the target frame can be realized, and the processing efficiency of the operations such as searching, editing and analyzing the key information in the video to be processed by a user is greatly improved.
It should be understood that the sequence number of each step in the foregoing embodiment does not mean that the execution sequence of each process should be determined by the function and the internal logic, and should not limit the implementation process of the embodiment of the present invention.
Fig. 3 is a schematic diagram of a video processing apparatus according to a third embodiment of the present invention. For convenience of explanation, only portions relevant to the embodiments of the present invention are shown.
The video processing apparatus 300 includes:
an acquisition module 301, configured to acquire a video to be processed;
a determining module 302, configured to determine a scene category of each video frame in the video to be processed;
the first recognition module 303 is configured to perform target recognition on each video frame in the video to be processed through a trained first depth learning model according to a scene category of each video frame, so as to obtain a target recognition result, where the target recognition result indicates whether each video frame contains a target object, and the target object corresponds to the scene category;
A second identifying module 304, configured to identify a plurality of target frames of the video to be processed according to the target identifying result, where the target frames include the target object;
and the processing module 305 is configured to sequentially combine the plurality of target frames according to the time sequence of each target frame in the video to be processed, so as to obtain the target video.
Optionally, if the video frame of the video to be processed includes a target object, the target recognition result further indicates an area and/or a feature point position where the target object is located;
Correspondingly, the second identifying module 304 specifically includes:
the judging unit is used for judging whether each video frame in the video to be processed accords with a first preset condition according to the region and/or the feature point position of the target object indicated by the target identification result;
And the determining unit is used for determining the video frame meeting the first preset condition as the target frame of the video to be processed.
Optionally, the judging unit is specifically configured to:
For each video frame containing the target object in the video to be processed, judging whether the difference degree of the target object in the video frame relative to the region and/or the characteristic point position of the target object in the previous video frame or the next video frame of the video frame accords with a first preset condition according to the region and/or the characteristic point position of the target object indicated by the target identification result;
Correspondingly, the determining unit is specifically configured to:
And if the difference degree of the target object in the video frame relative to the position of the region and/or the characteristic point of the target object in the previous video frame or the next video frame of the video frame meets a first preset condition, taking the video frame as the target frame of the video to be processed.
Optionally, the determining module 302 is specifically configured to:
Performing scene recognition on each video frame in the video to be processed through the trained second deep learning model to determine the scene category of each video frame in the video to be processed;
Or alternatively
And determining the scene category of each video frame in the video to be processed according to the name, the theme or the preset label of the video to be processed.
Optionally, the video processing apparatus 300 further includes:
and the identification module is used for identifying the target frame in the video to be processed.
Optionally, the video processing apparatus 300 further includes:
And the index module is used for establishing an information index according to the target frame.
Optionally, the indexing module is specifically configured to:
An information index list about the target frame is established, wherein the information index list comprises at least one of time information of the target frame in the video to be processed, scene category information of the target frame and target object information of the target frame.
According to the embodiment of the invention, by determining the scene category of each video frame and determining whether the target object corresponding to the scene category exists in each video frame through the trained deep learning model, the targeted target recognition can be carried out according to different scene categories, so that the target recognition result is more targeted and has less interference, a plurality of target frames of the video to be processed are recognized according to the target recognition result, the target video is obtained according to the target frames, key information (such as image parts containing target individuals) in the video to be processed can be accurately and efficiently extracted, the target video composed of the key information is obtained, redundant and unimportant video content in the video to be processed is deleted, and valuable video content is reserved, so that in application scenes such as searching for the key information of the monitoring video, a user can carry out more efficient and targeted processing according to the target video without manually searching, clipping, re-synthesizing and other operations on the key information. The embodiment of the invention greatly improves the efficiency of acquiring the key information in the video by the user, and has strong practicability and usability.
Fig. 4 is a schematic diagram of a terminal device according to a fourth embodiment of the present invention. As shown in fig. 4, the terminal device 4 of this embodiment comprises a processor 40, a memory 41 and a computer program 42 stored in said memory 41 and executable on said processor 40. The steps of the various video processing method embodiments described above, such as steps 101 through 105 shown in fig. 1, are implemented by the processor 40 when executing the computer program 42. Or the processor 40, when executing the computer program 42, performs the functions of the modules/units of the apparatus embodiments described above, e.g. the functions of the modules 301 to 305 shown in fig. 3.
Illustratively, the computer program 42 may be partitioned into one or more modules/units that are stored in the memory 41 and executed by the processor 40 to complete the present invention. The one or more modules/units may be a series of computer program instruction segments capable of performing specific functions for describing the execution of the computer program 42 in the terminal device 4. For example, the computer program 42 may be divided into an acquisition module, a determination module, a first identification module, a second identification module, and a processing module, where each module specifically functions as follows:
the acquisition module is used for acquiring the video to be processed;
the determining module is used for determining the scene category of each video frame in the video to be processed;
the first recognition module is used for carrying out target recognition on each video frame in the video to be processed through a first trained deep learning model according to the scene category of each video frame to obtain a target recognition result, wherein the target recognition result indicates whether each video frame contains a target object or not, and the target object corresponds to the scene category;
the second recognition module is used for recognizing a plurality of target frames of the video to be processed according to the target recognition result, wherein the target frames contain the target object;
and the processing module is used for sequentially combining a plurality of target frames according to the time sequence of each target frame in the video to be processed to obtain the target video.
The terminal device 4 may be a computing device such as a desktop computer, a notebook computer, a palm computer, a cloud server, etc. The terminal device may include, but is not limited to, a processor 40, a memory 41. It will be appreciated by those skilled in the art that fig. 4 is merely an example of the terminal device 4 and does not constitute a limitation of the terminal device 4, and may include more or less components than illustrated, or may combine certain components, or different components, e.g., the terminal device may further include an input-output device, a network access device, a bus, etc.
The Processor 40 may be a central processing unit (Central Processing Unit, CPU), other general purpose Processor, digital signal Processor (DIGITAL SIGNAL Processor, DSP), application SPECIFIC INTEGRATED Circuit (ASIC), off-the-shelf Programmable gate array (Field-Programmable GATE ARRAY, FPGA) or other Programmable logic device, discrete gate or transistor logic device, discrete hardware components, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The memory 41 may be an internal storage unit of the terminal device 4, such as a hard disk or a memory of the terminal device 4. The memory 41 may also be an external storage device of the terminal device 4, such as a plug-in hard disk, a smart memory card (SMART MEDIA CARD, SMC), a Secure Digital (SD) card, a flash memory card (FLASH CARD) or the like, which are provided on the terminal device 4. Further, the memory 41 may also include both an internal storage unit and an external storage device of the terminal device 4. The memory 41 is used for storing the computer program as well as other programs and data required by the terminal device. The memory 41 may also be used for temporarily storing data that has been output or is to be output.
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-described division of the functional units and modules is illustrated, and in practical application, the above-described functional distribution may be performed by different functional units and modules according to needs, i.e. the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-described functions. The functional units and modules in the embodiment may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit, where the integrated units may be implemented in a form of hardware or a form of a software functional unit. In addition, the specific names of the functional units and modules are only for distinguishing from each other, and are not used for limiting the protection scope of the present application. The specific working process of the units and modules in the above system may refer to the corresponding process in the foregoing method embodiment, which is not described herein again.
In the foregoing embodiments, the descriptions of the embodiments are emphasized, and in part, not described or illustrated in any particular embodiment, reference is made to the related descriptions of other embodiments.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
In the embodiments provided in the present invention, it should be understood that the disclosed apparatus/terminal device and method may be implemented in other manners. For example, the apparatus/terminal device embodiments described above are merely illustrative, e.g., the division of the modules or units is merely a logical function division, and there may be additional divisions in actual implementation, e.g., multiple units or components may be combined or integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection via interfaces, devices or units, which may be in electrical, mechanical or other forms.
The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in the embodiments of the present invention may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.
The integrated modules/units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the present invention may implement all or part of the flow of the method of the above embodiment, or may be implemented by a computer program to instruct related hardware, where the computer program may be stored in a computer readable storage medium, and when the computer program is executed by a processor, the computer program may implement the steps of each of the method embodiments described above. . Wherein the computer program comprises computer program code which may be in source code form, object code form, executable file or some intermediate form etc. The computer readable medium may include any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer Memory, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), an electrical carrier signal, a telecommunications signal, a software distribution medium, and so forth. It should be noted that the computer readable medium contains content that can be appropriately scaled according to the requirements of jurisdictions in which such content is subject to legislation and patent practice, such as in certain jurisdictions in which such content is subject to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.
The foregoing embodiments are merely illustrative of the technical solutions of the present invention, and not restrictive, and although the present invention has been described in detail with reference to the foregoing embodiments, it should be understood by those skilled in the art that modifications may still be made to the technical solutions described in the foregoing embodiments or equivalent substitutions of some technical features thereof, and that such modifications or substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims (8)
1. A video processing method, comprising:
acquiring a video to be processed;
Determining a scene category of each video frame in the video to be processed;
According to the scene category of each video frame, carrying out target recognition on each video frame in the video to be processed through a trained first depth learning model to obtain a target recognition result, wherein the target recognition result indicates whether each video frame contains a target object or not, and the target object corresponds to the scene category;
Identifying a plurality of target frames of the video to be processed according to the target identification result, wherein the target frames contain the target object;
Sequentially combining a plurality of target frames according to the time sequence of each target frame in the video to be processed to obtain a target video;
if the video frame of the video to be processed contains a target object, the target identification result also indicates the region and/or the feature point position of the target object;
correspondingly, the identifying a plurality of target frames of the video to be processed according to the target identification result includes:
For each video frame containing the target object in the video to be processed, judging whether the difference degree of the target object in the video frame relative to the region and/or the characteristic point position of the target object in the previous video frame or the next video frame of the video frame accords with a first preset condition according to the region and/or the characteristic point position of the target object indicated by the target identification result;
And if the difference degree of the target object in the video frame relative to the region where the target object is located and/or the characteristic point position in the previous video frame or the next video frame of the video frame accords with a first preset condition, the video frame is taken as the target frame of the video to be processed, wherein the first preset condition is that the offset of the characteristic point coordinates of the target object exceeds a preset offset threshold value, or the ratio of the difference value of the areas of the image regions in the two frames of video frames to the area of one of the image regions exceeds a preset ratio threshold value.
2. The video processing method of claim 1, wherein the determining a scene category for each video frame in the video to be processed comprises:
Performing scene recognition on each video frame in the video to be processed through the trained second deep learning model to determine the scene category of each video frame in the video to be processed;
Or alternatively
And determining the scene category of each video frame in the video to be processed according to the name, the theme or the preset label of the video to be processed.
3. The video processing method of claim 1, further comprising, after identifying a plurality of target frames of the video to be processed:
The target frame is identified in the video to be processed.
4. A video processing method according to any one of claims 1 to 3, further comprising, after identifying a plurality of target frames of the video to be processed:
and establishing an information index according to the target frame.
5. The video processing method of claim 4, wherein said creating an information index from said target frame comprises:
An information index list about the target frame is established, wherein the information index list comprises at least one of time information of the target frame in the video to be processed, scene category information of the target frame and target object information of the target frame.
6. A video processing apparatus, comprising:
the acquisition module is used for acquiring the video to be processed;
the determining module is used for determining the scene category of each video frame in the video to be processed;
the first recognition module is used for carrying out target recognition on each video frame in the video to be processed through a first trained deep learning model according to the scene category of each video frame to obtain a target recognition result, wherein the target recognition result indicates whether each video frame contains a target object or not, and the target object corresponds to the scene category;
the second recognition module is used for recognizing a plurality of target frames of the video to be processed according to the target recognition result, wherein the target frames contain the target object;
the processing module is used for sequentially combining a plurality of target frames according to the time sequence of each target frame in the video to be processed to obtain a target video;
if the video frame of the video to be processed contains a target object, the target identification result also indicates the region and/or the feature point position of the target object;
Correspondingly, the second identification module specifically includes:
the judging unit is used for judging whether each video frame in the video to be processed accords with a first preset condition according to the region and/or the feature point position of the target object indicated by the target identification result;
A determining unit, configured to determine a video frame that meets the first preset condition as a target frame of the video to be processed;
The judging unit is specifically configured to:
For each video frame containing the target object in the video to be processed, judging whether the difference degree of the target object in the video frame relative to the region and/or the characteristic point position of the target object in the previous video frame or the next video frame of the video frame accords with a first preset condition according to the region and/or the characteristic point position of the target object indicated by the target identification result;
the determining unit is specifically configured to:
And if the difference degree of the target object in the video frame relative to the region where the target object is located and/or the characteristic point position in the previous video frame or the next video frame of the video frame accords with a first preset condition, the video frame is taken as the target frame of the video to be processed, wherein the first preset condition is that the offset of the characteristic point coordinates of the target object exceeds a preset offset threshold value, or the ratio of the difference value of the areas of the image regions in the two frames of video frames to the area of one of the image regions exceeds a preset ratio threshold value.
7. A terminal device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the steps of the video processing method according to any one of claims 1 to 5 when the computer program is executed.
8. A computer readable storage medium storing a computer program, characterized in that the computer program when executed by a processor implements the steps of the video processing method according to any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910288788.3A CN110147722B (en) | 2019-04-11 | 2019-04-11 | Video processing method, video processing device and terminal equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910288788.3A CN110147722B (en) | 2019-04-11 | 2019-04-11 | Video processing method, video processing device and terminal equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110147722A CN110147722A (en) | 2019-08-20 |
| CN110147722B true CN110147722B (en) | 2024-12-24 |
Family
ID=67589739
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910288788.3A Active CN110147722B (en) | 2019-04-11 | 2019-04-11 | Video processing method, video processing device and terminal equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110147722B (en) |
Families Citing this family (32)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110569392B (en) * | 2019-08-28 | 2023-01-10 | 深圳市天视通技术有限公司 | Multi-video processing system and method |
| CN110659581B (en) * | 2019-08-29 | 2024-02-20 | 腾讯科技（深圳）有限公司 | Image processing method, device, equipment and storage medium |
| CN110674781B (en) * | 2019-10-07 | 2023-01-10 | 福建新宇电子有限公司 | Monitoring upgrading method and system |
| CN110856014B (en) * | 2019-11-05 | 2023-03-07 | 北京奇艺世纪科技有限公司 | Moving image generation method, moving image generation device, electronic device, and storage medium |
| CN111048071B (en) * | 2019-11-11 | 2023-05-30 | 京东科技信息技术有限公司 | Voice data processing method, device, computer equipment and storage medium |
| CN110881141B (en) * | 2019-11-19 | 2022-10-18 | 浙江大华技术股份有限公司 | Video display method and device, storage medium and electronic device |
| CN111010599B (en) * | 2019-12-18 | 2022-04-12 | 浙江大华技术股份有限公司 | Method and device for processing multi-scene video stream and computer equipment |
| CN111209436A (en) * | 2020-01-10 | 2020-05-29 | 上海摩象网络科技有限公司 | Method and device for shooting material mark and electronic equipment |
| CN111291785B (en) * | 2020-01-16 | 2024-11-19 | 中国平安人寿保险股份有限公司 | Target detection method, device, equipment and storage medium |
| CN111274441A (en) * | 2020-02-26 | 2020-06-12 | 赛特斯信息科技股份有限公司 | System and method for realizing maritime video data screening processing based on deep learning and target detection |
| CN111445499B (en) * | 2020-03-25 | 2023-07-18 | 北京百度网讯科技有限公司 | Method and device for identifying target information |
| CN111416950B (en) * | 2020-03-26 | 2023-11-28 | 腾讯科技（深圳）有限公司 | Video processing method and device, storage medium and electronic equipment |
| TWI749870B (en) * | 2020-04-08 | 2021-12-11 | 四零四科技股份有限公司 | Device of handling video content analysis |
| CN111552837A (en) * | 2020-05-08 | 2020-08-18 | 深圳市英威诺科技有限公司 | Animal video tag automatic generation method based on deep learning, terminal and medium |
| CN111581433B (en) * | 2020-05-18 | 2023-10-10 | Oppo广东移动通信有限公司 | Video processing method, device, electronic equipment and computer readable medium |
| CN113836981B (en) * | 2020-06-24 | 2025-02-28 | 阿里巴巴集团控股有限公司 | Data processing method, device, storage medium and computer equipment |
| CN112783880A (en) * | 2020-07-28 | 2021-05-11 | 薛杨杨 | Data analysis method based on artificial intelligence and big data and block chain service platform |
| CN111901536B (en) * | 2020-08-04 | 2023-03-24 | 携程计算机技术（上海）有限公司 | Video editing method, system, device and storage medium based on scene recognition |
| CN112137591B (en) * | 2020-10-12 | 2021-07-23 | 平安科技（深圳）有限公司 | Target object position detection method, device, equipment and medium based on video stream |
| CN112380922B (en) * | 2020-10-23 | 2024-03-22 | 岭东核电有限公司 | Method, device, computer equipment and storage medium for determining multiple video frames |
| CN112258513B (en) * | 2020-10-23 | 2024-07-16 | 岭东核电有限公司 | Nuclear power test video segmentation method, device, computer equipment and storage medium |
| CN113139428A (en) * | 2021-03-16 | 2021-07-20 | 西安天和防务技术股份有限公司 | Target identification method, edge device, frontier defense monitoring system and readable storage medium |
| CN112954456B (en) * | 2021-03-29 | 2023-06-20 | 深圳康佳电子科技有限公司 | Video data processing method, terminal and computer readable storage medium |
| CN113095194A (en) * | 2021-04-02 | 2021-07-09 | 北京车和家信息技术有限公司 | Image classification method and device, storage medium and electronic equipment |
| CN113329261B (en) * | 2021-08-02 | 2021-12-07 | 北京达佳互联信息技术有限公司 | Video processing method and device |
| WO2023019510A1 (en) * | 2021-08-19 | 2023-02-23 | 浙江吉利控股集团有限公司 | Data indexing method, apparatus and device, and storage medium |
| CN113949827B (en) * | 2021-09-30 | 2023-04-07 | 安徽尚趣玩网络科技有限公司 | Video content fusion method and device |
| CN114677619A (en) * | 2022-03-02 | 2022-06-28 | 海宁奕斯伟集成电路设计有限公司 | Video processing method, apparatus, electronic device, and computer-readable storage medium |
| TWI831235B (en) * | 2022-06-06 | 2024-02-01 | 仁寶電腦工業股份有限公司 | Dynamic image processing method, electronic device, and terminal device connected thereto |
| CN115171241B (en) * | 2022-06-30 | 2024-02-06 | 南京领行科技股份有限公司 | Video frame positioning method and device, electronic equipment and storage medium |
| CN115334271B (en) * | 2022-08-10 | 2024-05-07 | 平安科技（深圳）有限公司 | High-frame-rate video generation method and device, electronic equipment and storage medium |
| CN116680438B (en) * | 2023-05-13 | 2024-02-27 | 全景智联(武汉)科技有限公司 | A video condensation method, system, storage medium and electronic device |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103810195A (en) * | 2012-11-09 | 2014-05-21 | 中国电信股份有限公司 | Index generating method and system |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109213895A (en) * | 2017-07-05 | 2019-01-15 | 合网络技术（北京）有限公司 | A kind of generation method and device of video frequency abstract |
-
2019
- 2019-04-11 CN CN201910288788.3A patent/CN110147722B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103810195A (en) * | 2012-11-09 | 2014-05-21 | 中国电信股份有限公司 | Index generating method and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110147722A (en) | 2019-08-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110147722B (en) | Video processing method, video processing device and terminal equipment | |
| CN110610127B (en) | Face recognition method and device, storage medium and electronic equipment | |
| CN111858869B (en) | Data matching method and device, electronic equipment and storage medium | |
| CN106446816B (en) | Face recognition method and device | |
| EP3493101B1 (en) | Image recognition method, terminal, and nonvolatile storage medium | |
| CN111145214A (en) | Target tracking method, device, terminal equipment and medium | |
| WO2021051601A1 (en) | Method and system for selecting detection box using mask r-cnn, and electronic device and storage medium | |
| CN110909725A (en) | Method, device and equipment for recognizing text and storage medium | |
| CN110781784A (en) | Face recognition method, device and equipment based on double-path attention mechanism | |
| CN112381092B (en) | Tracking method, device and computer readable storage medium | |
| CN111191591B (en) | Watermark detection and video processing method and related equipment | |
| CN108563651B (en) | Multi-video target searching method, device and equipment | |
| CN110781770B (en) | Living body detection method, device and equipment based on face recognition | |
| CN111553241A (en) | Method, device and equipment for rejecting mismatching points of palm print and storage medium | |
| CN114359572A (en) | Training method and device of multi-task detection model and terminal equipment | |
| CN112052251B (en) | Target data updating method and related device, equipment and storage medium | |
| CN115100739B (en) | Man-machine behavior detection method, system, terminal device and storage medium | |
| CN112989869B (en) | Optimization method, device, equipment and storage medium of face quality detection model | |
| CN111460098A (en) | Text matching method and device and terminal equipment | |
| CN116977692A (en) | A data processing method, equipment and computer-readable storage medium | |
| CN112712005A (en) | Training method of recognition model, target recognition method and terminal equipment | |
| CN114387600B (en) | Text feature recognition method, device, computer equipment and storage medium | |
| CN110287943B (en) | Image object recognition method and device, electronic equipment and storage medium | |
| CN112966687B (en) | Image segmentation model training method and device and communication equipment | |
| CN111881740A (en) | Face recognition method, face recognition device, electronic equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |