CN111311475B

CN111311475B - Detection model training method, device, storage medium and computer equipment

Info

Publication number: CN111311475B
Application number: CN202010108690.8A
Authority: CN
Inventors: 毛懿荣; 李岩; 王汉杰; 陈波
Original assignee: Guangzhou Tencent Technology Co Ltd
Current assignee: Guangzhou Tencent Technology Co Ltd
Priority date: 2020-02-21
Filing date: 2020-02-21
Publication date: 2025-03-14
Anticipated expiration: 2040-02-21
Also published as: CN111311475A

Abstract

The application relates to a training method, device, storage medium and computer equipment of a detection model, wherein the method comprises the steps of obtaining an original image to be processed and more than one type of mark image, randomly selecting a target position from a target area of the original image as an embedding position of the mark image for each type of mark image, embedding at least one part of the mark image into the original image according to the corresponding embedding position for each type of mark image to obtain a corresponding sample image, taking the sample image as a training sample, taking the mark type of the mark image embedded in the sample image as a corresponding training label, and training the detection model to be trained through the training sample and the corresponding training label. The scheme provided by the application can improve the training efficiency.

Description

Detection model training method, device, storage medium and computer equipment

Technical Field

The present application relates to the field of machine learning technologies, and in particular, to a method and apparatus for training a detection model, a computer readable storage medium, and a computer device.

Background

With the development of computer technology, machine learning technology has emerged, through which a computer can be trained to simulate or realize learning behaviors of humans, thereby bringing convenience to life and work of people. For example, in the field of image processing, a model may be trained by training data such that the model learns the ability to classify or locate, which may allow a machine to implement processing of an image instead of human.

In practical applications, for example, when a detection model capable of identifying a target object (such as a watermark or trademark) needs to be trained, a large amount of labeling data is often required and then model training is performed. However, in the conventional method, the class to which the target object belongs and the position of the target object in the image are usually marked with human power, so that the marking speed is slow, and the model training efficiency is low.

Disclosure of Invention

Based on the above, it is necessary to provide a method, an apparatus, a computer readable storage medium and a computer device for training a detection model, aiming at the technical problem of low model training efficiency caused by manual annotation data.

A test model training method, comprising:

acquiring an original image to be processed and more than one type of mark image;

for each type of marked image, randomly selecting a target position from a target area of the original image as an embedded position of the marked image;

For each type of mark image, at least one part of the mark image is embedded into the original image according to the corresponding embedding position to obtain a corresponding sample image;

taking the sample image as a training sample, and taking the mark category of the mark image embedded in the sample image as a corresponding training label;

and training the detection model to be trained through the training sample and the corresponding training label.

A test model training apparatus, the apparatus comprising:

the acquisition module is used for acquiring an original image to be processed and more than one type of mark image;

the selecting module is used for randomly selecting a target position from a target area of the original image as an embedding position of the mark image for each type of mark image;

The embedding module is used for embedding at least one part of the mark images into the original image according to the corresponding embedding positions for each type of mark image to obtain a corresponding sample image;

The determining module takes the sample image as a training sample and takes the mark category of the mark image embedded in the sample image as a corresponding training label;

And the training module is used for training the detection model to be trained through the training sample and the corresponding training label.

In one embodiment, the obtaining module is further configured to obtain an original image to be processed and more than one type of marking templates, randomly select a target size ratio from a preset size ratio range, and scale each type of marking templates according to the size of the original image and the target size ratio to obtain a corresponding marking image.

In one embodiment, the target positions include core positions and non-core positions, the selection module is further configured to determine core positions in a target area of the original image, acquire probability values when the core positions are used as embedding positions, wherein the probability values corresponding to the core positions are maximum values in the probability values corresponding to the target positions in the target area, determine probability values when the non-core positions are used as embedding positions according to distances between the non-core positions and the core positions in the target area, wherein the probability values corresponding to the non-core positions are inversely related to distances between the non-core positions and the core positions, and select corresponding target positions as embedding positions of the marker images according to the probability values corresponding to the target positions in the target area of the original image for each type of marker image.

In one embodiment, the selecting module is further configured to obtain a preset number of platform-specific images, each of the platform-specific images includes a label image corresponding to a corresponding platform, determine an average coordinate corresponding to a target vertex of each of the label images according to coordinates of the target vertex in the platform-specific image, and use the average coordinate as a core position in a target area of the original image.

In one embodiment, the target area includes an upper left corner area and a lower right corner area, the selecting module is further configured to, for each type of the marking templates, select, according to probability values corresponding to respective target positions in the upper left corner area of the original image, a corresponding target position as an embedding position corresponding to an upper left vertex of the marking image, and, for each type of the marking templates, select, according to probability values corresponding to respective target positions in the lower right corner area of the original image, a corresponding target position as an embedding position corresponding to a lower right vertex of the marking image, when the target area is the lower right corner area.

In one embodiment, the sample image comprises a first sample image and a second sample image, the embedding module is further used for determining a first mark image to be completely embedded and a second mark image to be shielded and embedded in each type of mark image, the first mark image is completely embedded into the original image according to the corresponding embedding position to obtain a corresponding first sample image, the second mark image is completely embedded into the original image according to the corresponding embedding position, a target shielding proportion is randomly selected from a preset shielding proportion range, and a part of the second mark image is moved out to the boundary of the original image according to the target shielding proportion to obtain a corresponding second sample image.

In one embodiment, the determining module is further configured to determine a label category of a label image embedded in each sample image and position information of the label image in the original image, and use the sample image as a training sample and use the label category of the label image embedded in the sample image and the corresponding position information together as a training label of the training sample.

In one embodiment, the training module is further configured to cut the sample image according to the target area to obtain corresponding sample image blocks, extract features of each sample image block through a detection model to be trained to obtain corresponding feature graphs, detect and output a prediction result based on the feature graphs, and adjust model parameters of the detection model according to differences between a prediction result corresponding to the sample image block and corresponding training labels until training is stopped when training stopping conditions are met.

In one embodiment, the device further comprises a mark detection module, wherein the mark detection module is used for acquiring videos to be detected and a trained detection model, extracting a preset number of video frames from the videos to be detected, cutting each video frame according to the target area to obtain corresponding target image blocks, respectively inputting each target image block into the trained detection model, outputting detection results corresponding to each target image block, and fusing the detection results of each target image block to obtain detection results corresponding to the videos to be detected.

In one embodiment, the mark detection module is further configured to input each target image block to the trained detection model, sequentially process the input target image blocks through at least three groups of convolution groups in the trained detection model, wherein a downsampling layer in a final group of convolution groups is a cavity convolution with a step length being a preset value so as to keep the size of a feature image output by the final group of convolution groups to be a preset size, perform convolution processing on the feature image output by a middle group of convolution groups to obtain a first feature image to be detected, take the feature image output by the final group of convolution groups as a second feature image to be detected, perform convolution processing on the second feature image to be detected to obtain at least one third feature image to be detected, respectively perform detection processing on the first feature image to be detected, the second feature image to be detected and the third feature image to obtain a candidate detection result corresponding to the candidate result and confidence coefficient corresponding to the candidate result, and screen the candidate detection result corresponding to the feature image to the candidate detection result meeting the high confidence condition of the candidate detection result to be used as the input target image of the candidate detection result of the corresponding high confidence condition.

In one embodiment, the detection result comprises a mark category to which the mark image of the video to be detected belongs, the device further comprises a video pushing module for obtaining a video filtering instruction, the video filtering instruction comprises a first target category, mark categories corresponding to videos in a video library are determined through the trained detection model, videos to be filtered with the mark category being the first target category are searched from the video library, and videos except the videos to be filtered in the video library are pushed to a user terminal which initiates the video filtering instruction in response to the video filtering instruction.

In one embodiment, the video pushing module is further configured to obtain a video search instruction, where the video search instruction includes a second target category, determine, through the trained detection model, a tag category corresponding to each video in a video library, search, from the video library, a target video whose tag category is the second target category, and push, in response to the video search instruction, the target video to a user terminal that initiates the instruction of the video.

A computer readable storage medium storing a computer program which, when executed by a processor, causes the processor to perform the steps of:

A computer device comprising a memory and a processor, the memory storing a computer program which, when executed by the processor, causes the processor to perform the steps of:

According to the method, the device, the computer-readable storage medium and the computer equipment for training the detection model, each type of marked image is randomly embedded into the original image, the situation that the marked image is possibly edited or compressed in a real scene is simulated during embedding, and the marked image is embedded in whole or part of the marked image is blocked, so that the marked training data can be automatically generated for training the detection model. The training labels in the training data are the label categories to which the embedded label images belong. Therefore, the condition that a real marked image appears in an original image is simulated by adopting various random strategies without manually marking training data, the labor cost and the marking efficiency for marking the training data are greatly reduced, and the model training efficiency is further greatly improved.

Drawings

FIG. 1 is a diagram of an application environment for a test model training method in one embodiment;

FIG. 2 is a flow chart of a test model training method in one embodiment;

FIG. 3 is a schematic diagram of marking templates in one embodiment;

FIG. 4 is a flowchart illustrating a step of randomly selecting a target position from a target area of an original image as an embedded position of a marker image for each type of marker image according to an embodiment;

FIG. 5 is a flowchart illustrating steps for performing marker detection on a video to be detected through a trained detection model in one embodiment;

FIG. 6 is a schematic diagram of a network structure of a detection network based on SSD algorithm in one embodiment;

FIG. 7 is a schematic diagram of a network architecture of RETINANET networks in one embodiment;

FIG. 8 is a flowchart of the steps for performing marker detection on a video to be detected through a trained detection model and obtaining a detection result in an embodiment;

FIG. 9 is a block diagram of a test model training apparatus in one embodiment;

FIG. 10 is a block diagram of another embodiment of a test model training apparatus;

FIG. 11 is a block diagram of a computer device in one embodiment.

Detailed Description

The present application will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present application more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the application.

FIG. 1 is a diagram of an application environment for a test model training method in one embodiment. Referring to fig. 1, the test model training method is applied to a test model training system. The detection model training system includes a terminal 110 and a server 120. The terminal 110 and the server 120 are connected through a network. The terminal 110 may be a desktop terminal or a mobile terminal, and the mobile terminal may be at least one of a mobile phone, a tablet computer, a notebook computer, and the like. The server 120 may be implemented as a stand-alone server or as a server cluster composed of a plurality of servers. Both the terminal 110 and the server 120 may be used separately to perform the test model training method provided in the embodiments of the present application. Terminal 110 and server 120 may also cooperate to perform the detection model training method provided in embodiments of the present application.

It should be noted that the test model training method involves machine learning (MACHINE LEARNING, ML), which is a multi-domain interdisciplinary, involving multiple disciplines such as probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, etc. It is specially studied how a computer simulates or implements learning behavior of a human to acquire new knowledge or skills, and reorganizes existing knowledge structures to continuously improve own performance. Machine learning is the core of artificial intelligence, a fundamental approach to letting computers have intelligence, which is applied throughout various areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, confidence networks, reinforcement learning, transfer learning, induction learning, teaching learning, and the like.

As shown in FIG. 2, in one embodiment, a test model training method is provided. The embodiment is mainly exemplified by the application of the method to a computer device, which may specifically be the terminal 110 or the server 120 in fig. 1. Referring to fig. 2, the test model training method specifically includes the following steps:

s202, acquiring an original image to be processed and more than one type of mark image.

Wherein the original image is an image to be processed without marking information. The original image may be a real image collected by a camera, or a video frame of a certain frame or frames divided in a video file, or an image synthesized by a computer device, etc. The marking information is information for performing a special marking, and may specifically be specific text, icon, symbol, audio, or other information having a certain degree of recognition. The marked image is an image including marked information, and may be a watermark image or a trademark image.

In particular, the computer device may obtain one or more original images from a local or other computer device, where the plurality of images is more than one, and the terms "plurality", "multiple", or "multiple types", etc. in the embodiments of the present application, unless otherwise specified, all refer to "more than one", or "more than one type", etc. The computer device may obtain the marking information belonging to different marking categories in advance and generate corresponding marking images according to the corresponding marking information. The marking category may specifically be a category for marking different platforms, that is, a category of a platform to which the marking image belongs. The different platforms may specifically be different media platforms, such as a "tremble" platform, a "micro-vision" platform, a "watermelon video" platform, a "fast-handed" platform, a "volcanic small video" platform, or a "picoshrimp" platform, etc. The marking images corresponding to different platforms have the characteristics of the respective platforms and are mutually different.

In one embodiment, to ensure the training effect on the detection model, the computer device may adjust the format of the acquired original image to obtain an original image of uniform format and size.

In one embodiment, the computer device may further obtain a label template having RGBA (red green blue Alpha) pieces of channel information, and adjust the size of the label template according to a preset size ratio according to the size of the original image to obtain a corresponding label image.

S204, randomly selecting target positions from target areas of the original images as embedding positions of the marker images for each type of marker images.

The target area is an area for embedding the marker image, and may specifically be an entire area of the original image, or may be a partial area in the original image, such as an upper left corner area, a lower right corner area, a lower left corner area, or an upper right corner area with a preset size in the original image. The embedded position is a position point for locating a specific position of the marker image in the original image.

Specifically, the computer device may randomly select a certain type of marker image from various types of marker images, and randomly select a target position from a target area of the original image as an embedding position of the marker image. It can be understood that, in order to make the detection model obtain a good detection effect, different label categories can be accurately obtained, and when training data is constructed, an equal number of sample images are constructed for each type of label image, that is, when the computer device selects the label images, the probability that each type of label image is selected is equal. The training data specifically comprises training samples and training labels.

In one embodiment, the computer device may preset the fixed region as the target region of the original image. Or the computer device may determine the target area based on the area in which the marker image is located in the real image that actually includes the marker image. For example, the computer device may count a number of positions where the marker image appears in the real image, and take an area where the probability of the occurrence of the marker image in the real image is greater than a preset probability threshold or a larger area as the target area. The real image here is an image with marking information of a platform corresponding to each of different platforms in the real scene, and may also be referred to as a platform-specific image.

In one embodiment, when the computer device selects a target location from the target area, the probability value of each location in the target area being selected may be set to the same probability value or may be set to a different probability value. In one embodiment, the computer device may determine a location in the target area as a core location that corresponds to a maximum selected probability value, with selected probability values corresponding to other non-core locations decreasing around the core location.

In one embodiment, the computer device may use the selected target position in the target area as the embedded position of the marker image, specifically as the embedded position corresponding to the top left vertex of the marker image, as the embedded position corresponding to the bottom right vertex of the marker image, or as the embedded position corresponding to the center point of the marker image, which is not limited in this embodiment of the present application.

S206, for each type of mark image, at least one part of the mark image is embedded into the original image according to the corresponding embedding position, so as to obtain a corresponding sample image.

Specifically, after determining an embedding position corresponding to a certain mark image, the computer device may embed at least a part of image content in the mark image into the original image according to the embedding position to obtain a corresponding sample image.

In one embodiment, the computer device may paste the entire marker image onto the original image in accordance with the embedded location to obtain a corresponding sample image.

In one embodiment, the computer device may paste the entire marker image to the original image according to the embedding position, and then move the marker image toward the edge of the original image, so that a portion of the marker image is moved out to the edge of the original image, and the image content of the remaining portion obtains a corresponding sample image on the original image. This better simulates that the edges of the marked image may be cropped in practice.

In one embodiment, the computer device may crop out a portion of the image content from the marked image, and paste the portion of the marked image to the original image according to the embedded position to obtain a corresponding sample image.

For example, the computer device may preset an occlusion probability value, such as 0.25, that is, 25% of the training samples have their marker images occluded and 75% of the training samples have their marker images intact. For training samples in which the marker image is occluded, the computer device may randomly select the target occlusion ratio from the preset occlusion ratio range, so as to adjust the height, width or area of the marker image in the original image according to the target occlusion ratio. For example, the computer device may set the preset shielding proportion range to be 0.3-0.7, so that the computer device may uniformly select the target shielding proportion from 0.3-0.7.

S208, taking the sample image as a training sample, and taking the mark category of the mark image embedded in the sample image as a corresponding training label.

In particular, the computer device may randomly generate a large number of sample images comprising marker images of different marker categories based on the random manner mentioned in the above steps. The embedding positions of the mark images in the generated sample images are randomly distributed, and the coverage areas of the mark images in the sample images are also randomly distributed. The computer device can further take the sample images as training samples, and take the mark types of the mark images embedded in the sample images as training labels corresponding to the corresponding training samples.

In one embodiment, the step S208, namely taking the sample image as a training sample and taking the mark type of the mark image embedded in the sample image as a corresponding training label, specifically comprises determining the mark type of the mark image embedded in each sample image and the position information of the mark image in the original image, taking the sample image as the training sample, and taking the mark type of the mark image embedded in the sample image and the corresponding position information together as the training label of the training sample.

In one embodiment, the computer device may determine the marker class of the marker image embedded in each sample image, and the location information of the marker image in the original image. The position information of the marker image in the original image is used for positioning the marker image, and specifically may be coordinates of an upper left vertex and a lower right vertex of the embedded marker image, or coordinates of the lower left vertex and the upper right vertex, or the like. Furthermore, the computer device may take the sample image as a training sample and take the label category of the label image embedded in the sample image and the corresponding position information together as a training label of the training sample. In this way, the detection model obtained through training of the corresponding training sample and training label not only can predict the mark type of the image to be detected, but also can position the mark image in the image to be detected.

S210, training the detection model to be trained through the training sample and the corresponding training label.

The detection model is a convolutional neural network model and is used for carrying out mark detection on an image to be detected or a video to be detected. When the training labels are only of the label types, the detection model obtained through training can be used for classifying the label images included in the input images to be detected. When the training label comprises the label category and the position information of the label image, the detection model obtained through training can be used for classifying and positioning the label image included in the input image to be detected.

Specifically, the training of the detection model is a supervised training process. The computer equipment inputs the training sample into the detection model, takes the training label corresponding to the training sample as target output, and enables the actual output of the detection model to continuously approach the target output by adjusting the model parameters of the detection model.

In one embodiment, the computer device may input training samples into the test model for training to obtain the predicted results. And constructing a loss function according to the prediction result and the difference of the training label. And taking the model parameters when the loss function is minimized as the model parameters of the detection model, returning to the step of inputting the training sample into the detection model for training to obtain a detection result, and stopping training until the training stopping condition is met.

Wherein the training stop condition is a condition for ending model training. The training stop condition may be that a preset iteration number is reached, or that the performance index of the detection model after the model parameter is adjusted reaches a preset index.

In one embodiment, the model structure of the detection model and the detection algorithm used are not limited in this embodiment of the present application. The detection model can be specifically realized by a two-stage detection algorithm based on a neural network, and can also be realized by a one-stage detection algorithm based on the neural network. For example, the computer device may construct a detection model based on an SSD algorithm (Single Shot MultiBox Detector, single-shot multi-box detection, one target detection algorithm) or a RETINANET algorithm (one single-stage target detection algorithm) via a convolutional neural network, and so on.

In one embodiment, step S210, namely training the detection model to be trained through training samples and corresponding training labels, specifically comprises the steps of cutting sample images according to target areas to obtain corresponding sample image blocks, respectively extracting features of each sample image block through the detection model to be trained to obtain corresponding feature images, detecting and outputting a prediction result based on the feature images, and adjusting model parameters of the detection model according to the difference between the prediction result corresponding to the sample image blocks and the corresponding training labels until training is stopped when training stopping conditions are met.

In one embodiment, the computer device may crop the sample image according to the target region to obtain a corresponding sample image block. And the computer equipment can respectively input each sample image block into a detection model to be trained, respectively perform feature extraction on each sample image block through a convolution layer included in the detection model to obtain a corresponding feature map, and perform mark detection based on each feature map to output a prediction result. When the training label is of the label type, the corresponding prediction result is of the predicted label type. When the training label is the label category and the position information of the label image in the target image block, the corresponding prediction result is the predicted label category and regression frame. The computer equipment can adjust the model parameters of the detection model towards the direction of reducing the difference according to the prediction result corresponding to the sample image block and the difference between the corresponding training labels, and the training is stopped until the training stopping condition is met.

In the above embodiment, the sample image is cut according to the target area to obtain the corresponding sample image block, and then the sample image block is used for training the detection model to be trained, so that the interference and the additional processing amount caused by the background information irrelevant to the mark image in the training process can be greatly reduced, and the model training efficiency and accuracy are improved.

It will be appreciated that in practical applications, to avoid the effective content of the original image being blocked by the marker image, the marker image will typically appear in the upper left corner or lower right corner of the original image, which can not only perform the marking effect, but also not block the effective content of the original image. Based on this, in designing training data, if only the marker image is embedded in a fixed position in the upper left corner region and the lower right corner region of the original image, the detection model may misunderstand that the marker image is most likely to exist for the fixed position, and not actually have the ability to judge that different positions contain the marker image, only the ability to learn the resolution position is obtained. In step S204, some random factors are added in determining the embedding position of the marker image to simulate the real situation. In step S206, at least a portion of the marker image is embedded into the original image, mainly considering that the marker image may be edited or compressed in the real scene, so that only a portion of the marker image appears, and a random occlusion factor is added to make the detection model capable of processing the condition that the marker image is only partially visible. Therefore, the real generation condition of the marked image is simulated through various random strategies, reliable and effective training data with marking information is automatically generated, and the labor cost for obtaining the training data is greatly reduced. And training the detection model by the generated training data attached to the real situation, so that the detection model obtained by training has higher detection precision.

According to the detection model training method, each type of marked image is randomly embedded into the original image, the situation that the marked image is possibly edited or compressed in a real scene is simulated during embedding, and all the marked images are embedded or part of marked images are blocked, so that training data with marks can be automatically generated for training the detection model. The training labels in the training data are the label categories to which the embedded label images belong. Therefore, the condition that a real marked image appears in an original image is simulated by adopting various random strategies without manually marking training data, the labor cost and the marking efficiency for marking the training data are greatly reduced, and the model training efficiency is further greatly improved.

In one embodiment, step S202, that is, the step of acquiring the original image to be processed and the more than one type of mark images, specifically includes acquiring the original image to be processed and the more than one type of mark templates, randomly selecting a target size ratio from a preset size ratio range, and scaling each type of mark templates according to the size of the original image and the target size ratio to obtain a corresponding mark image.

The marking templates are marking samples corresponding to each platform and can be used as templates. The size ratio is a size ratio between the sizes of two objects, and in the embodiment of the present application, the size ratio of an image to an original image is specifically indicated. The preset size ratio range is a range composed of a series of size ratios, such as a height ratio range, a width ratio range, or an area ratio range, which are preset.

Specifically, the computer device may obtain the label templates corresponding to the different platforms, that is, label templates belonging to different label categories. The computer device may specifically obtain a label template with RGBA4 channel data. And the computer equipment can select a target size proportion from a preset size proportion range, and respectively scale various marking templates according to the target size proportion according to the size of the original image to obtain a corresponding marking image.

In one embodiment, the computer device may randomly select one of the candidate mark templates, wherein the probabilities of the selection of the respective mark templates are the same. The computer device may preset a height ratio range of the marker image to the original image, such as a height ratio range of 0.04 to 0.14. The computer device may randomly select a target height ratio from the range of height ratios, and scale the marking template according to the corresponding target height ratio to obtain the marking image. It may be understood that the preset size ratio range may specifically be a preset width ratio range or an area ratio range, and the corresponding selected target size ratio may also be a target width ratio or a target area ratio, which is not limited in the embodiment of the present application.

In one embodiment, for different size ratios in the range of preset size ratios, the computer device may set a selection probability corresponding thereto. And then randomly selecting the target size proportion from the preset size proportion range according to the corresponding selection probability. In one embodiment, the setting of the selection probability may be determined according to a real image that contains the marker image, with a larger selection probability for the size proportion that occurs frequently and a smaller selection probability for the size proportion that occurs rarely. Therefore, the real condition can be simulated more truly for the size condition of the marked image relative to the original image, so that the training effect of the detection model can be improved.

Referring to fig. 3, fig. 3 is a schematic diagram of the marking templates in one embodiment. For example, the marking template can be a marking template corresponding to a platform such as "micro-vision", "time-light small video" or "meal-down video". It will be appreciated that the marking templates and marking images referred to in the embodiments of the present application include, but are not limited to, the several types described above, by way of example only, in fig. 3. For example, the marking template can also be a template corresponding to a "tremble sound" platform, a "watermelon video" platform, a "fast hand" platform, a "volcanic small video" platform, a "picoshrimp" platform, or the like.

In one embodiment, for each type of marking template, the frequency with which the various types of marking templates are used by the computer device in constructing the corresponding training data may be the same or different, and embodiments of the present application are not limited herein. With the appearance of different platforms in the market, corresponding marking templates are more and more, and for newly-appearing marking templates, training data do not need to be additionally marked by manpower, and the training data corresponding to the marking templates can be constructed to retrain the detection model by adopting the mode mentioned by the embodiment of the application so as to improve the detection range and the capability of the detection model.

In the above embodiment, the target size proportion is randomly selected from the preset size proportion range, and then various mark templates are respectively scaled according to the size of the original image and the target size proportion, so that mark images with different sizes can be obtained, and the real situation can be more attached, so that the constructed training sample is more accurate, and the training effect on the detection model can be greatly improved.

In one embodiment, the target positions include a core position and a non-core position, and the step S202 of randomly selecting the target positions from the target areas of the original images as the embedded positions of the marker images for each type of marker images specifically includes the steps of:

s402, determining a core position in a target area of an original image.

The core position is the position with the largest probability value of the embedded position selected in the target area. The computer device may select a location with the largest number of embedded locations as a core location from among a predetermined number of image data of the real scene. Or the computer device can also take the average position of the embedded positions corresponding to the marker images as a core position according to the image data of the plurality of real scenes.

In one embodiment, the step S402, that is, the step of determining the core position in the target area of the original image, specifically includes obtaining a preset number of platform-specific images, each platform-specific image includes a label image corresponding to a corresponding platform, determining an average coordinate corresponding to a target vertex according to the coordinate of the target vertex of each label image in the platform-specific image, and taking the average coordinate as the core position in the target area of the original image.

The target vertex of the marked image may specifically be an upper left vertex, a lower left vertex, an upper right vertex or a lower right vertex. The target vertex of the marker image may be specifically an upper left vertex when the target region is an upper left corner region of the original image, and the target vertex of the marker image may be specifically a lower right vertex when the target region is a lower right corner region of the original image.

In one embodiment, the computer device may acquire a preset number of platform specific images, each platform specific image being an image in a real scene. The computer device may determine vertex coordinates of the target vertices of each of the marker images in the platform-specific image, respectively, calculate average coordinates of the respective vertex coordinates, and then use the average coordinates as a core position in the target region of the original image. It will be appreciated that such a core position determined from the target vertices, when selected as an embedding position in constructing the training data, may correspond to the core position in the original image as the position of the corresponding target vertex of the marker image, thereby transferring the marker image to the original image.

For example, a computer device may acquire 50 images of a real scene, i.e., platform-specific images. The average coordinates of the corresponding marked image (such as watermark image) in the 50 platform-specific images in the upper left corner area are calculated, and the position corresponding to the average coordinates is taken as the core position in the upper left corner area. And establishing a rectangular coordinate system downwards by taking the upper left vertex of the original image as the origin of coordinates, wherein the height of the original image is h, the width of the original image is w. The coordinates of the core position may be (w 0.03, h 0.015) by way of example. When the target region is the lower right corner region, the coordinates of the core position of the lower right corner region may be (w-w 0.03, h-h 0.035) by way of example.

In the above embodiment, the average positions corresponding to the target vertices of the target images in the preset number of platform-specific images in the real scene are used as the core positions, so as to allocate the maximum probability value, and the real situation is more attached.

And S404, acquiring a probability value when the core position is taken as the embedded position, wherein the probability value corresponding to the core position is the maximum value in the probability values corresponding to the target positions in the target area.

Specifically, the computer device may set a core position in advance as a probability value when embedding the position, where the probability value corresponding to the core position is a maximum value among probability values corresponding to respective target positions in the target area.

S406, determining probability values when the non-core positions are respectively used as embedded positions according to the distances between the non-core positions and the core positions in the target area, wherein the probability values corresponding to the non-core positions are inversely related to the distances between the non-core positions and the core positions.

Specifically, other locations in the target area than the core location may be referred to as non-core locations. The computer device may calculate a distance from each non-core location in the target area to the core location, which may be specifically represented by a range distance. And determining probability values when the non-core positions are respectively used as embedded positions according to the distances between the non-core positions and the core positions in the target area, wherein the magnitude of the probability values is inversely proportional to a range distance from the core point. That is, the probability distribution of each position decreases around the core position. It is understood that the sum of probability values corresponding to the respective target positions in the target area is 1.

S408, for each type of marked image, selecting the corresponding target position as the embedded position of the marked image according to the probability value corresponding to each target position in the target area of the original image.

Specifically, for a marker image selected from various types of marker images, the computer device may select a corresponding target position as an embedding position of the marker image according to a probability value corresponding to each target position in the target region of the original image, respectively.

In one embodiment, the target area comprises an upper left corner area and a lower right corner area, for each type of marked image, the corresponding target position is selected as an embedded position of the marked image according to the probability value corresponding to each target position in the target area of the original image, and when the target area is the upper left corner area, the corresponding target position is selected as the embedded position corresponding to the upper left vertex of the marked image according to the probability value corresponding to each target position in the upper left corner area of the original image for each type of marked template, and when the target area is the lower right corner area, the corresponding target position is selected as the embedded position corresponding to the lower right vertex of the marked image according to the probability value corresponding to each target position in the lower right corner area of the original image for each type of marked template.

In one embodiment, when the target area is an upper left corner area, the computer device may select, according to the probability value corresponding to each target position in the upper left corner area of the original image, the corresponding target position as an embedding position corresponding to the upper left vertex of the marker image, and further embed the upper left corner of the marker image at the position when the marker image is embedded. When the target area is the lower right corner area, the computer equipment can select the corresponding target position as an embedding position corresponding to the lower right vertex of the mark image according to the probability value corresponding to each target position in the lower right corner area of the original image, and then the lower right corner of the mark image is embedded into the position when the mark image is embedded. Thus, the upper left vertex and the lower right vertex of the marked image are respectively used as reference points for embedding, so that the embedding is more convenient and accurate.

In the above embodiment, the maximum probability value corresponding to the core position in the target area is set, and the probability distribution of the non-core position is decreased from the core position to the periphery, so that the target position is selected as the embedded position of the marker image according to the probability value corresponding to each position, the actual situation is more attached, and the random distribution is satisfied, so that the construction of the sample image is quicker and more accurate.

In one embodiment, the sample image comprises a first sample image and a second sample image, and the step S208 of embedding at least one part of the mark images into the original image according to the corresponding embedding positions for each type of mark images to obtain the corresponding sample image specifically comprises the steps of determining a first mark image to be completely embedded and a second mark image to be shielded and embedded in the mark images for each type of mark images, completely embedding the first mark image into the original image according to the corresponding embedding positions to obtain the corresponding first sample image, completely embedding the second mark image into the original image according to the corresponding embedding positions, randomly selecting a target shielding ratio from a preset shielding ratio range, and moving one part of the second mark image out to the boundary of the original image according to the target shielding ratio to obtain the corresponding second sample image.

Wherein the first marker image is a marker image selected to be fully embedded in the original image and the second marker image is a marker image selected to be occluded from being embedded in the original image. And, correspondingly, the first sample image is a sample image corresponding to the first marker image, and the second sample image is a sample image corresponding to the second marker image. It will be appreciated that the original image may be the same original image or different original images, which is not limited by the embodiment of the present application.

In one embodiment, the computer device may preset the ratio of the number of the first marker images to be completely embedded and the number of the second marker images to be embedded, i.e. determine with a probability of 0.25 whether the marker images are blocked. That is, for each type of marker image, 25% of the marker images are occluded and embedded in the original image, and 75% of the marker images are fully embedded in the original image, among a certain number of marker images. Of course, this ratio may be other values, and the embodiment of the present application is not limited thereto.

Further, the computer device may determine a first marker image to be fully embedded and a second marker image to be occlusion embedded from all marker images. The computer device can completely embed the first mark image into the original image according to the corresponding embedding position, and a corresponding first sample image is obtained. When the target area is the upper left corner area, the computer device may embed the upper left vertex of the first marker image into the embedded position to paste the complete first marker image into the original image, resulting in a corresponding first sample image. When the target area is a lower right corner area, the computer device may embed a lower right vertex of the first marker image into the embedded position to paste the complete first marker image into the original image, and obtain a corresponding first sample image.

And for the second marked image, the computer equipment can randomly select the target shielding proportion from the range of the preset shielding proportion, and after the second marked image is completely embedded into the original image according to the corresponding embedding position, part of the second marked image is moved out to the boundary of the original image according to the selected target shielding proportion, so that the corresponding second sample image is obtained. The position information of the embedded marker image in the second sample image is updated again according to the moved position.

The shielding proportion is the proportion between the height, width or area of the part of the second mark image outside the original image and the original image, such as the height shielding proportion, the width shielding proportion or the area shielding proportion, and the like. The preset shielding proportion range is a range which is preset and is composed of a series of shielding proportions, such as a height shielding proportion range, a width shielding proportion range, an area shielding proportion range and the like. That is, the computer device may randomly select the target occlusion ratio from the range, and then move the second marker image upward, downward, leftward or rightward toward the boundary of the original image according to the target occlusion ratio, so that the ratio of the height, width or area of the portion of the second marker image outside the original image to the height, width or area of the second marker image is the target occlusion ratio.

For example, when the preset occlusion ratio range is a height occlusion ratio range, such as 0.3 to 0.7, the computer device may randomly select the target height occlusion ratio from the corresponding range, such as 0.4, and further translate the second marker image upward or downward when moving the second marker image, so that a partial region of the second marker image is outside the original image, and a height of a partial marker image outside the original image is 0.4 times the height of the marker image. When the preset occlusion ratio range is a width occlusion ratio range, such as 0.3 to 0.7, the computer device may randomly select a target width occlusion ratio, such as 0.4, from the corresponding range, and further translate the second marker image to the left or right when moving the second marker image, so that a partial region of the second marker image is outside the original image, and a width of a partial marker image outside the original image is 0.4 times the width of the marker image. When the preset occlusion ratio range is an area occlusion ratio range, for example, 0.3 to 0.7, the computer device may randomly select a target area occlusion ratio, for example, 0.4, from the corresponding range, and further, when moving the second marker image, move the boundary of the original image of the second marker image, so that a partial area of the second marker image is outside the original image, and an area of the partial marker image outside the original image is 0.4 times an area of the marker image. It will be appreciated that the above ranges of occlusion ratios and target occlusion ratios are exemplary values only and are not intended to limit the present application.

In one embodiment, for different occlusion ratios in a preset occlusion ratio range, the computer device may set a selection probability corresponding thereto. The probability values selected by different shielding proportions may be the same or different, and the embodiment of the present application is not limited to this.

In the above embodiment, when the marker image is embedded into the original image, some marker images are completely embedded into the original image, and some marker images are blocked and embedded into the original image, so that the situation that the marker image in the real scene is possibly edited or compressed can be well simulated, and the accuracy of training data construction is greatly improved.

In one embodiment, the method for training a detection model further includes a step of performing marker detection on the video to be detected through the trained detection model, and the step specifically includes:

S502, obtaining a video to be detected and a trained detection model.

In particular, the computer device may obtain videos to be detected and already trained detection models in the video library. In a specific application scenario, the video to be detected is a small video to be detected. The small video to be detected is a video with a video duration smaller than a preset duration or a video size smaller than a preset size.

S504, extracting a preset number of video frames from the video to be detected, and cutting each video frame according to the target area to obtain a corresponding target image block.

Specifically, the computer device may convert the video to be detected into video frames of one frame by one frame at a preset frequency, for example, the computer device may extract one frame of image every second as a video frame. The computer device may then screen out a predetermined number of video frames from the plurality of frames of video, such as screening out video frames of the first frame and the intermediate frame as video frames to be detected. Based on practical experience, it is known that, for a small video, a watermark image (i.e., a mark image) appears in a large probability in an upper left corner region of a first frame, and a watermark image appears in a large probability in a lower right corner region of an intermediate frame, so that the first frame and the intermediate frame are selected as video frames to be detected, and the accuracy of mark detection can be improved.

Furthermore, the computer device may crop out the extracted image data of the target area in each frame of video frame, so as to obtain a corresponding target image block. Specifically, the upper left corner area and the lower right corner area in each frame of video frame are cut out, so that the corresponding target image block is obtained.

S506, each target image block is respectively input into the trained detection model, and detection results corresponding to each target image block are output.

Specifically, the computer device may input each target image block as input data into the trained detection model, process the input data through the model structure and the model parameters of the detection model, and output the detection results corresponding to each target image block.

It can be understood that, when the training label of the detection model in the training process is only the label category of the label image, the detection result output after the processing of the trained detection model is the label category corresponding to each target image block. When the training label of the detection model in the training process is the label type and position information of the label image, the detection result output after the processing of the trained detection model is the label type and position information corresponding to each target image block.

In one embodiment, step S506, namely, inputting each target image block into a trained detection model respectively, and outputting a detection result corresponding to each target image block specifically includes inputting each target image block into the trained detection model respectively, sequentially processing the input target image block through at least three groups of convolution groups in the trained detection model, wherein a downsampling layer in a final group of convolution groups is a cavity convolution with a step length being a preset value so as to keep the size of a feature map output by the final group of convolution groups to be a preset size, performing convolution processing on the feature map output by a middle group of convolution groups to obtain a first feature map to be detected, taking the feature map output by the final group of convolution groups as a second feature map to be detected, performing convolution processing on the second feature map to be detected to obtain at least one third feature map to be detected, respectively performing detection processing on the first feature map to be detected, the second feature map to be detected and the third feature map to obtain a corresponding candidate detection result and a confidence level corresponding to the candidate result, and filtering the feature map to obtain a candidate result with a high confidence that the confidence level of the candidate result is met as a corresponding candidate image detection result.

Wherein a convolution set (block) is a network structure comprising a plurality of convolution layers. Specifically, the trained detection model comprises at least three groups of convolution groups, the computer equipment inputs the target image block into the trained convolution network, and the target image block is processed through each group of convolution groups in the trained convolution network. After each convolution group of the target image, the width and height of the corresponding feature map are reduced by half. The computer device may set the downsampling layer in the last convolution group to use hole convolution with a step size of a preset value (e.g., 2) such that the size of the last convolution group is a preset size. The computer device may further perform convolution processing on the feature images output by the middle group (for example, the second group) of convolution groups, to obtain a first feature image to be detected, and use the feature image output by the last group of convolution groups as a second feature image to be detected. In addition, the computer equipment can also perform different convolution processing on the second feature map to be detected to obtain at least one third feature map to be detected, wherein the sizes of the third feature maps to be detected are different.

The computer device can further detect the multiple layers of feature images to be detected respectively, wherein each layer is provided with a respective detector (the detectors do not share model parameters), and each detector outputs candidate detection results (specifically, regression boxes and classification results) of each position of the corresponding feature image to be detected. For each layer of feature map to be detected, the computer equipment can use the candidate detection result with the highest confidence coefficient in the candidate detection results of each position in the feature map to be detected as the candidate detection result corresponding to the feature map to be detected. And the computer equipment can screen out candidate detection results with corresponding confidence degrees meeting the high confidence degree condition from candidate detection results corresponding to each feature map to be detected as detection results corresponding to the input target image blocks. The confidence level meeting the high confidence level condition may specifically be the maximum confidence level or the top N (N is a positive integer greater than 1) name after the confidence level is ranked from high to low.

In a particular embodiment, the computer device may employ a neural network-based detection algorithm. The detection algorithm based on the neural network can be roughly divided into a one-stage detection algorithm and a two-stage detection algorithm, and the SSD detection algorithm in the one-stage detection algorithm can be adopted as the detection model in the application in particular in consideration of the fact that the speed of the one-stage detection algorithm is higher. Referring to fig. 6, fig. 6 is a schematic diagram of a network structure of a detection network based on an SSD algorithm in one embodiment. The backbone network of the detection network can adopt ResNet-34 layers of network, a target image block with the size of 300x300 is input, then detection is carried out on a plurality of layers of characteristic images to be detected of the network, each layer is provided with a respective detector (the detectors do not share parameters), and each detector outputs a regression frame and a classification result of each position of the characteristic images to be detected.

The detection network in the embodiment of the application sequentially comprises a convolution layer and 4 Block layers, and the width and the height of the characteristic diagram after each Block are doubled. The size of a feature map output by a target image Block through a second Block layer is 38x38, the size of the feature map after the target image Block passes through a third Block layer is 19x19, and in order to make the feature map output by a fourth Block layer bigger, a computer device realizes the downsampling layer of the fourth Block by adopting hole convolution with the step length of 2, and the size of the output feature map is still 19x19, but the feature expression capability is stronger. The 38x38 feature map output by the second Block layer outputs a first feature map to be detected (the size is 38x 38) after passing through the two layers of convolution layers, the 19x19 feature map corresponding to the fourth Block layer is a second feature map to be detected, and the second feature map to be detected sequentially outputs a third feature map to be detected of 10x10, 5x5 and 3x3 after passing through the 3 two layers of convolution structures (the second convolution step length is 2). The first feature map to be detected, the second feature map to be detected and the third feature map to be detected are 5 feature maps to be detected in total, and each feature map is provided with a corresponding detector header. For each position of the feature map to be detected, a preset number of anchor boxes are designed, and the detector predicts the probability that the preset number of anchor boxes respectively belong to each marking category and the rectangular bounding box of the marking image. The probability that each anchor frame belongs to each marking category and the rectangular bounding box of the marking image are candidate detection results corresponding to each position. The computer device may use the candidate detection result with the highest confidence coefficient of the candidate detection results at each position in the 5 feature maps to be detected as the candidate detection result corresponding to the target image block.

In the above embodiment, the plurality of to-be-detected images corresponding to the target image block are respectively detected by marking, so that the candidate detection result with higher confidence is screened out from the candidate detection results corresponding to the to-be-detected feature images as the detection result corresponding to the target image block, and the accuracy of the detection result can be ensured.

In one embodiment, the detection model may also be implemented by RETINANET networks, wherein RETINANET networks employ a pyramid network structure that adds a top-down side branch that amplifies and phase-adds higher-level features to lower-level features, enhancing the expressive power of lower-level features. Referring to fig. 7, fig. 7 is a schematic diagram of a network structure of RETINANET networks in one embodiment. As shown in fig. 7, the network structure of RETINANET network is mainly composed of (a) ResNet, (b) FPN and 2 FCN subnetworks. Wherein ResNet, english full-scale Residual Network, also called Residual Network. FPN, english full name Feature Pyramid Network, also known as feature pyramid network. FCN, english full-scale Fully Convolutional Networks, also known as full convolutional network. The 2 full convolution networks in the embodiment of the present application may specifically be (c) a classification sub-network (class sub-network) and (d) a detection frame position regression sub-network (box sub-network) for predicting the label class to which the label image belongs and the position information of the label image. The Backbone network (Backbone) of RETINANET network can be composed of ResNet +FPN, and the feature pyramid can be obtained after the feature extraction of the Backbone of the input image. After obtaining the feature pyramids, with continued reference to fig. 7, two sub-networks, namely a class sub-network (class sub-network) and a box position regression sub-network (box sub-network), are used for each layer of feature pyramids respectively, so as to process and output the final detection result.

In one embodiment, RETINANET the backbone network employs a ResNet network, with an input target image block size of 600x600. In practical applications, for image data, there is often a far smaller foreground object than background, while the detector predicts class probabilities at each location of the feature map, which will result in a far greater number of background classes than foreground classes. To address this problem of the large imbalance of the front background, the Loss function may be weighted by the Focal Loss (a Loss function algorithm) of the RETINANET network. The benefit of Focal Loss is that it can automatically mine out difficult samples according to the size of the Loss function and filter out simple samples, so that the network can learn from difficult samples efficiently, i.e. can learn more about the classification of the marker images.

It will be appreciated that the detection model may be implemented using other neural network algorithms, and the SSD algorithms and RETINANET mentioned in the above embodiments are only for illustrative purposes, and are not intended to limit the structure and algorithm of the detection network in the embodiments of the present application.

S508, fusing the detection results of the target image blocks to obtain detection results corresponding to the video to be detected.

Specifically, the computer device may determine the confidence coefficient of the detection result corresponding to each target image block, and use the detection result of the target image block corresponding to the maximum confidence coefficient as the detection result corresponding to the video to be detected. The confidence corresponding to the detection result represents the credibility of the detection result.

In the above embodiment, the image data of the target area is cut out from the video frame in the video to be detected, so as to obtain the corresponding target image blocks, and then, the trained detection model is used for carrying out mark detection on each target image block, and the detection results corresponding to each target image block are fused to obtain the detection results corresponding to the video to be detected, so that the detection speed is high and the detection precision is high when the video to be detected is marked and detected.

In a specific embodiment, referring to fig. 8, fig. 8 is a flowchart illustrating a step of performing marking detection on a video to be detected through a trained detection model and obtaining a detection result in a specific embodiment. As shown in fig. 8, the computer device obtains a video to be detected, such as a small mobile phone video, extracts a video frame from the small mobile phone video, and intercepts target image blocks corresponding to an upper left corner region and a lower right corner region from the extracted video frame. The computer device may then perform marker detection for each target image block. In this embodiment, the marked image is in particular a watermark image and may thus also be referred to as watermark detection. The computer equipment can fuse the detection results of watermark detection corresponding to each image block to obtain the detection result corresponding to the video to be detected.

In practical application, when the marked image is a watermark image, the detection model obtained by training the detection model training method provided by the embodiment of the application can well identify and position the watermark of the small video with the watermark of each platform in practical use, and has high detection precision. In addition, under the condition that various new platforms are endless and the related watermarks of the small videos derived by the new platforms are possibly more and more, new training data are automatically generated and the detection model is retrained by the detection model training method, so that the new watermarks can be well identified, the waste of human resources is greatly reduced, and the training efficiency and accuracy are greatly improved.

In a specific application scene, the detection result comprises a mark category to which a mark image in a video to be detected belongs, the detection model training method further comprises the step of filtering specific videos, the step specifically comprises the steps of obtaining a video filtering instruction, the video filtering instruction comprises a first target category, determining mark categories corresponding to videos in a video library respectively through a trained detection model, searching the video library for the video to be filtered, the mark category of which is the first target category, and pushing the videos in the video library except the video to be filtered to a user terminal initiating the video filtering instruction in response to the video filtering instruction.

In a specific application scenario, when a user initiates a video filtering instruction to a computer device through a user terminal, the computer device may extract a first target class carried in the video filtering instruction. The computer equipment can carry out marking detection on each video in the video library through the trained detection model so as to obtain detection results corresponding to each video respectively, wherein the detection results comprise marking categories corresponding to each video respectively. The computer equipment can search the video library for the video to be filtered with the mark type being the first target type, and push the video with the video to be filtered in the video library to the corresponding user terminal.

In one embodiment, the marked image is a watermark image, and the computer device may filter the video containing the particular watermark image according to the video filtering instructions, so that the user may filter out video from the particular video application according to his/her own likes and dislikes.

In the embodiment, the mark type corresponding to each video in the video library can be determined through the trained detection model, so that the video of a specific mark type disliked by the user is filtered, and the video service can be conveniently and intelligently provided for the user.

In a specific application scene, the detection result comprises a mark category to which a mark image in a video to be detected belongs, the detection model training method further comprises the step of searching a specific video, the step specifically comprises the steps of obtaining a video searching instruction, the video searching instruction comprises a second target category, determining mark categories corresponding to videos in a video library respectively through a trained detection model, searching target videos with the mark category being the second target category from the video library, and pushing the target videos to a user terminal initiating the video instruction in response to the video searching instruction.

In one embodiment, when a user initiates a video search instruction to the computer device through the user terminal, the computer device may extract a second target category carried in the video search instruction. The computer equipment can carry out marking detection on each video in the video library through the trained detection model so as to obtain detection results corresponding to each video respectively, wherein the detection results comprise marking categories corresponding to each video respectively. The computer device may then search the video library for target videos labeled as the second target category and push the target videos to the user terminal initiating the video instruction.

In one embodiment, the tagged image is a watermark image, and the user may input a watermark category associated with the video application, so that the computer device may quickly retrieve the corresponding video, and thus the user may search for video from the particular video application according to his/her own preferences.

In the embodiment, the mark type corresponding to each video in the video library can be determined through the trained detection model, so that the video of a specific mark type liked by the user can be searched, and the video service can be conveniently and intelligently provided for the user.

In one embodiment, the detection result includes a marker category and position information to which the marker image in the video to be detected belongs; the method for training the detection model further comprises the step of editing the marked image in the video, and the step specifically comprises the step that the computer equipment can determine the mark category corresponding to each video in the video library and the position information of the marked image through the trained detection model, and then locate the marked image so as to realize editing processing of the marked image.

FIG. 2 is a flow chart of a test model training method in one embodiment. It should be understood that, although the steps in the flowchart of fig. 2 are shown in sequence as indicated by the arrows, the steps are not necessarily performed in sequence as indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in fig. 2 may include multiple sub-steps or stages that are not necessarily performed at the same time, but may be performed at different times, nor do the order in which the sub-steps or stages are performed necessarily performed in sequence, but may be performed alternately or alternately with at least a portion of the sub-steps or stages of other steps or other steps.

In one embodiment, as shown in FIG. 9, a test model training apparatus 900 is provided, which may employ software modules or hardware modules, or a combination of both, as part of a computer device. The apparatus includes an acquisition module 901, a selection module 902, an embedding module 903, a determination module 904, and a training module 905, wherein:

An acquisition module 901, configured to acquire an original image to be processed and more than one type of marker image.

A selecting module 902, configured to randomly select, for each type of marker image, a target position from a target area of the original image as an embedding position of the marker image.

The embedding module 903 is configured to, for each type of marker image, embed at least a portion of the marker image into the original image according to the corresponding embedding position, so as to obtain a corresponding sample image.

The determining module 904 takes the sample image as a training sample, and takes the label category of the label image embedded in the sample image as a corresponding training label.

The training module 905 is configured to train the detection model to be trained through the training samples and the corresponding training labels.

In one embodiment, the obtaining module 901 is further configured to obtain an original image to be processed and more than one type of marking templates, randomly select a target size ratio from a preset size ratio range, and scale each type of marking templates according to the size of the original image and the target size ratio, so as to obtain a corresponding marking image.

In one embodiment, the target positions include core positions and non-core positions, the selection module 902 is further configured to determine core positions in a target area of the original image, obtain probability values when the core positions are used as embedding positions, wherein the probability value corresponding to the core positions is a maximum value among the probability values corresponding to the target positions in the target area, determine probability values when the non-core positions are used as embedding positions according to distances between the non-core positions and the core positions in the target area, wherein the probability values corresponding to the non-core positions are inversely related to distances between the non-core positions and the core positions, and select corresponding target positions as embedding positions of the marker images according to the probability values corresponding to the target positions in the target area of the original image for each type of marker image.

In one embodiment, the selecting module 902 is further configured to obtain a preset number of platform-specific images, where each platform-specific image includes a label image corresponding to a corresponding platform, determine an average coordinate corresponding to a target vertex according to a coordinate of the target vertex of each label image in the platform-specific image, and use the average coordinate as a core position in a target area of the original image.

In one embodiment, the target area includes an upper left corner area and a lower right corner area, the selecting module 902 is further configured to, for each type of marking template, select, according to probability values corresponding to respective target positions in the upper left corner area of the original image, a corresponding target position as an embedding position corresponding to an upper left vertex of the marking image, and, for each type of marking template, select, for each type of marking template, a corresponding target position as an embedding position corresponding to a lower right vertex of the marking image, according to respective probability values corresponding to respective target positions in the lower right corner area of the original image, respectively.

In one embodiment, the sample image includes a first sample image and a second sample image, the embedding module 903 is further configured to determine, for each type of marker image, a first marker image to be completely embedded and a second marker image to be blocked and embedded in the marker image, completely embed the first marker image into the original image according to a corresponding embedding position to obtain a corresponding first sample image, embed the second marker image into the original image according to a corresponding embedding position, randomly select a target blocking ratio from a preset blocking ratio range, and remove a portion of the second marker image to a boundary of the original image according to the target blocking ratio to obtain a corresponding second sample image.

In one embodiment, the determining module 904 is further configured to determine a label category of the label image embedded in each sample image and position information of the label image in the original image, and take the sample image as a training sample and take the label category of the label image embedded in the sample image and the corresponding position information together as a training label of the training sample.

In one embodiment, the training module 905 is further configured to cut the sample image according to the target area to obtain a corresponding sample image block, extract features of each sample image block through a detection model to be trained to obtain a corresponding feature map, detect and output a prediction result based on the feature map, and adjust model parameters of the detection model according to a difference between a prediction result corresponding to the sample image block and a corresponding training label until a training stop condition is met.

In one embodiment, the test model training device 900 further includes a mark detection module 906, configured to obtain a video to be tested and a trained test model, extract a preset number of video frames from the video to be tested, crop each video frame according to a target area to obtain a corresponding target image block, input each target image block into the trained test model, output a test result corresponding to each target image block, and fuse the test results of each target image block to obtain a test result corresponding to the video to be tested.

In one embodiment, the mark detection module 906 is further configured to input each target image block to a trained detection model, sequentially process the input target image blocks through at least three groups of convolution groups in the trained detection model, wherein a downsampling layer in a final group of convolution groups is a cavity convolution with a step length being a preset value, so as to keep the size of a feature image output by the final group of convolution groups to be a preset size, perform convolution processing on the feature image output by an intermediate group of convolution groups to obtain a first feature image to be detected, take the feature image output by the final group of convolution groups as a second feature image to be detected, perform convolution processing on the second feature image to be detected to obtain at least one third feature image to be detected, perform detection processing on the first feature image to be detected, the second feature image to be detected and the third feature image to be detected to obtain a candidate detection result corresponding to each and a confidence level corresponding to the candidate detection result, and screen the candidate detection result meeting a high confidence condition from the candidate detection results corresponding to the respective candidate detection result to be used as the detection result corresponding to the input target image block.

Referring to fig. 10, in one embodiment, the detection result includes a label category to which a label image in a video to be detected belongs, the detection model training device 900 further includes a video pushing module 907 for acquiring a video filtering instruction, the video filtering instruction includes a first target category, the label categories corresponding to each video in the video library are determined through a trained detection model, the video library is searched for a video to be filtered with the label category being the first target category, and the video except for the video to be filtered in the video library is pushed to a user terminal initiating the video filtering instruction in response to the video filtering instruction.

In one embodiment, the video pushing module 907 is further configured to obtain a video search instruction, where the video search instruction includes a second target category, determine, through a trained detection model, a tag category corresponding to each video in the video library, search the video library for a target video with the tag category being the second target category, and push the target video to the user terminal that initiated the video instruction in response to the video search instruction.

According to the detection model training device, each type of mark image is randomly embedded into the original image, the situation that the mark image is possibly edited or compressed in a real scene is simulated during embedding, and all the mark images are embedded or part of the mark images are blocked, so that training data with marks can be automatically generated for training the detection model. The training labels in the training data are the label categories to which the embedded label images belong. Therefore, the condition that a real marked image appears in an original image is simulated by adopting various random strategies without manually marking training data, the labor cost and the marking efficiency for marking the training data are greatly reduced, and the model training efficiency is further greatly improved.

For specific limitations of the test model training device, reference may be made to the above limitations of the test model training method, and no further description is given here. The modules in the detection model training device can be implemented in whole or in part by software, hardware and a combination thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.

FIG. 11 illustrates an internal block diagram of a computer device in one embodiment. The computer device may be specifically the terminal 110 or the server 120 in fig. 1. As shown in fig. 11, the computer device includes a processor, a memory, and a network interface connected by a system bus. The memory includes a nonvolatile storage medium and an internal memory. The non-volatile storage medium of the computer device stores an operating system, and may also store a computer program that, when executed by the processor, causes the processor to implement a test model training method. The internal memory may also have stored therein a computer program which, when executed by the processor, causes the processor to perform the test model training method.

It will be appreciated by those skilled in the art that the structure shown in FIG. 11 is merely a block diagram of some of the structures associated with the present inventive arrangements and is not limiting of the computer device to which the present inventive arrangements may be applied, and that a particular computer device may include more or fewer components than shown, or may combine some of the components, or have a different arrangement of components.

In an embodiment, there is also provided a computer device comprising a memory and a processor, the memory having stored therein a computer program, the processor implementing the steps of the method embodiments described above when the computer program is executed.

In one embodiment, a computer-readable storage medium is provided, storing a computer program which, when executed by a processor, implements the steps of the method embodiments described above.

Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, or the like. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory. By way of illustration, and not limitation, RAM can be in various forms such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc.

The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.

The above examples illustrate only a few embodiments of the application, which are described in detail and are not to be construed as limiting the scope of the application. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the application, which are all within the scope of the application. Accordingly, the scope of protection of the present application is to be determined by the appended claims.

Claims

1. A detection model training method, characterized in that the method comprises:

Obtaining an original image to be processed and labeled images of more than one category;

The average position corresponding to the target vertex of the marked image in a preset number of platform-specific images is used as the core position in the target area of the original image, and the other positions of the target position in the target area except the core position are used as non-core positions; the probability value corresponding to the core position is the maximum value among the probability values corresponding to the target positions in the target area, and the probability value corresponding to the non-core position is negatively correlated with the distance from the non-core position to the core position;

For each type of labeled image, according to the probability values corresponding to the target positions in the target area of the original image, the corresponding target positions are selected as the embedding positions of the labeled image;

For each type of labeled image, determining a first labeled image to be completely embedded and a second labeled image to be occluded and embedded in the labeled image;

Completely embedding the first marked image into the original image according to the corresponding embedding position to obtain a corresponding first sample image;

The second marked image is completely embedded into the original image according to the corresponding embedding position, and a target occlusion ratio is randomly selected from a preset occlusion ratio range, and a part of the second marked image is moved out to the boundary of the original image according to the target occlusion ratio to obtain a corresponding second sample image;

Using a sample image as a training sample, and using the labeled category of the labeled image embedded in the sample image as a corresponding training label; the sample image includes the first sample image and the second sample image;

The detection model to be trained is trained using the training samples and corresponding training labels.

2. The method according to claim 1, characterized in that the obtaining of the original image to be processed and more than one type of labeled images comprises:

Obtaining an original image to be processed and more than one type of labeled templates;

Randomly select a target size ratio from a preset size ratio range;

According to the size of the original image, each type of marking template is scaled according to the target size ratio to obtain a corresponding marking image.

3. The method according to claim 1, characterized in that after determining the core position and the non-core position of each target position in the target area of the original image, the method further comprises:

Obtaining a probability value when the core position is used as an embedding position;

According to the distances between each non-core position in the target area and the core position, the probability values of taking each non-core position as the embedding position are determined.

4. The method according to claim 1, wherein the step of using the average position corresponding to the target vertices of the marked images in a preset number of platform-specific images as the core position in the target area of the original image comprises:

Acquire a preset number of platform-specific images; each of the platform-specific images includes a marked image corresponding to the corresponding platform;

According to the coordinates of the target vertices of each marked image in the platform-specific image, respectively, determining the average coordinates corresponding to the target vertices;

The average coordinates are used as the core position in the target area of the original image.

5. The method according to claim 1, characterized in that the target area includes an upper left corner area and a lower right corner area; for each type of labeled image, selecting the corresponding target position as the embedding position of the labeled image according to the probability value corresponding to each target position in the target area of the original image, comprises:

When the target area is the upper left corner area, for each type of marked image, according to the probability values corresponding to each target position in the upper left corner area of the original image, the corresponding target position is selected as the embedding position corresponding to the upper left vertex of the marked image;

When the target area is the lower right corner area, for each type of marked image, the corresponding target position is selected as the embedding position corresponding to the lower right vertex of the marked image according to the probability value corresponding to each target position in the lower right corner area of the original image.

6. The method according to claim 1, characterized in that the step of using the sample image as a training sample and using the label category of the label image embedded in the sample image as the corresponding training label comprises:

Determining the tag category of the tag image embedded in each of the sample images and the position information of the tag image in the original image;

The sample image is used as a training sample, and the tag category and corresponding position information of the tag image embedded in the sample image are used together as training labels of the training sample.

7. The method according to claim 1, characterized in that the step of training the detection model to be trained by using the training samples and the corresponding training labels comprises:

Cropping the sample image according to the target area to obtain a corresponding sample image block;

Extract features from each of the sample image blocks using the detection model to be trained to obtain a corresponding feature map, and perform detection based on the feature map to output a prediction result;

According to the difference between the prediction result corresponding to the sample image block and the corresponding training label, the model parameters of the detection model are adjusted until the training is stopped when the training stop condition is met.

8. The method according to any one of claims 1 to 7, characterized in that the method further comprises:

Get the video to be detected and the trained detection model;

Extracting a preset number of video frames from the video to be detected, and cropping each of the video frames according to the target area to obtain a corresponding target image block;

Input each of the target image blocks into the trained detection model, and output the detection results corresponding to each of the target image blocks;

The detection results of each target image block are fused to obtain a detection result corresponding to the video to be detected.

9. The method according to claim 8, characterized in that the step of inputting each of the target image blocks into the trained detection model and outputting the detection results corresponding to each of the target image blocks comprises:

Inputting each of the target image blocks into the trained detection model respectively;

The input target image block is processed in sequence by at least three convolution groups in the trained detection model; wherein the downsampling layer in the last convolution group is a hole convolution with a step size of a preset value, so as to keep the size of the feature map output by the last convolution group at a preset size;

Convolution processing is performed on the feature map output by the convolution group of the middle group to obtain a first feature map to be detected;

Using the feature map output by the last convolution group as the second feature map to be detected;

Performing convolution processing on the second feature map to be detected to obtain at least one third feature map to be detected;

Performing detection processing on the first feature map to be detected, the second feature map to be detected, and the third feature map to be detected respectively, to obtain corresponding candidate detection results and confidence levels corresponding to the candidate results;

From the candidate detection results corresponding to each feature map to be detected, the candidate detection results whose corresponding confidences meet the high confidence condition are screened out as the detection results corresponding to the input target image block.

10. The method according to claim 8, characterized in that the detection result includes the tag category to which the marked image in the video to be detected belongs; the method further comprises:

Obtaining a video filtering instruction; the video filtering instruction includes a first target category;

Determine the label category corresponding to each video in the video library by using the trained detection model;

Searching the video library for videos to be filtered whose marked categories are the first target categories;

In response to the video filtering instruction, the videos in the video library except the video to be filtered are pushed to the user terminal that initiates the video filtering instruction.

11. The method according to claim 8, characterized in that the detection result includes the tag category to which the marked image in the video to be detected belongs; the method further comprises:

Obtaining a video search instruction; the video search instruction includes a second target category;

Searching the video library for a target video whose marked category is a second target category;

In response to the video search instruction, the target video is pushed to a user terminal that initiates the video instruction.

12. A detection model training device, characterized in that the device comprises:

An acquisition module, used for acquiring an original image to be processed and more than one type of labeled images;

A selection module is used to use the average position corresponding to the target vertices of the marked images in a preset number of platform-specific images as the core position in the target area of the original image, and use other positions in the target area except the core position as non-core positions; the probability value corresponding to the core position is the maximum value among the probability values corresponding to the target positions in the target area, and the probability value corresponding to the non-core position is negatively correlated with the distance from the non-core position to the core position; for each type of marked image, select the corresponding target position as the embedding position of the marked image according to the probability value corresponding to each target position in the target area of the original image;

The embedding module is used to determine, for each type of labeled image, a first labeled image to be completely embedded and a second labeled image to be embedded with occlusion in the labeled image; completely embed the first labeled image into the original image according to the corresponding embedding position to obtain a corresponding first sample image; completely embed the second labeled image into the original image according to the corresponding embedding position, and randomly select a target occlusion ratio from a preset occlusion ratio range, and move a part of the second labeled image out to the boundary of the original image according to the target occlusion ratio to obtain a corresponding second sample image;

A determination module, taking a sample image as a training sample, and taking a label category of a label image embedded in the sample image as a corresponding training label; the sample image includes the first sample image and the second sample image;

The training module is used to train the detection model to be trained through the training samples and corresponding training labels.

13. The device according to claim 12, characterized in that the acquisition module is further used for:

Randomly select a target size ratio from a preset size ratio range;

14. The device according to claim 12, characterized in that the acquisition module is further used for:

15. The device according to claim 12, characterized in that the acquisition module is further used for:

16. The device according to claim 12, wherein the target area includes an upper left corner area and a lower right corner area; and the selection module is further used to:

17. The device according to claim 12, wherein the determining module is further used to:

18. The device according to claim 12, characterized in that the training module is further used for:

19. The device according to any one of claims 12 to 18, characterized in that the device further comprises a mark detection module, wherein the mark detection module is used to:

Get the video to be detected and the trained detection model;

20. The device according to claim 19, characterized in that the mark detection module is further used for:

21. The device according to claim 19, characterized in that the detection result includes the tag category to which the marked image in the video to be detected belongs; the device also includes a video push module for:

22. The device according to claim 19, wherein the detection result includes a tag category to which the tagged image in the video to be detected belongs; and the device further comprises a video push module for:

23. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the steps of the method according to any one of claims 1 to 11.

24. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 11.