CN116309634A - Image processing method, device, terminal equipment, storage medium and program product - Google Patents

Abstract

The application relates to an image processing method, an image processing device, a terminal device, a storage medium and a program product, wherein the method is applied to a first image processing model on the terminal device and comprises the following steps: splitting the image to be processed into a plurality of partitions based on the partition parameters; carrying out histogram statistics on each partition to obtain a histogram corresponding to each partition; determining mapping parameters corresponding to each partition according to the mapping parameter lookup table and the histogram corresponding to each partition; and adjusting pixel values of the corresponding subareas in the image to be processed by using the mapping parameters to obtain the processed image. According to the image processing method, the image processing flow can be simplified, the calculated amount is reduced, and the execution efficiency is improved while the accurate image details are maintained.

Description

Image processing method, device, terminal equipment, storage medium and program product

technical field

The present application relates to the field of image processing, and in particular to an image processing method, device, terminal equipment, storage medium and program product.

Background technique

An image signal processor (image signal processor, ISP) of an existing terminal device is generally used to output images with better visual effects for human eyes. However, this kind of image with good visual effect for human eyes may not be suitable for target detection, such as vehicle detection or pedestrian detection. Therefore, based on the existing image signal processor, it is necessary to design an image processing method to obtain an image suitable for target detection.

The image processing methods in the prior art are either difficult to retain accurate image details during image processing, or the implementation of the method is relatively complex, the amount of calculation is large, and the execution efficiency is low. Therefore, how to simplify the image while retaining accurate image details Processing flow, reducing the amount of calculation, and improving execution efficiency have become research hotspots in this field.

Contents of the invention

In view of this, an image processing method, device, terminal equipment, storage medium, and program product are proposed. According to the image processing method of the embodiment of the present application, while retaining accurate image details, the image processing flow can be simplified, and the computation time can be reduced. volume and improve execution efficiency.

In the first aspect, the embodiment of the present application provides an image processing method, the method is applied to the first image processing model on the terminal device, and the method includes: splitting the image to be processed into multiple partitions based on partition parameters , the partition parameter indicates the size of each partition; perform histogram statistics for each partition to obtain the histogram corresponding to each partition; determine each partition according to the mapping parameter lookup table and the histogram corresponding to each partition Corresponding mapping parameters; using the mapping parameters to adjust the pixel values of the corresponding partitions in the image to be processed to obtain a processed image.

According to the image processing method of the embodiment of the present application, the image to be processed is split into multiple partitions based on the partition parameter, and the partition parameter indicates the size of each partition, so that the overall receptive field of the first image processing model can be effectively improved through the partition parameter, At the same time control the amount of follow-up calculations and improve image processing efficiency; perform histogram statistics for each partition to obtain the histogram corresponding to each partition, and determine the corresponding histogram of each partition according to the mapping parameter lookup table and the histogram corresponding to each partition The mapping parameters greatly reduce the time-consuming of determining the mapping parameters, and further improve the image processing efficiency by reducing the delay; the processed image can be obtained by adjusting the pixel values of the corresponding partitions in the image to be processed by using the mapping parameters, where the mapping parameters are related to the partition One-to-one correspondence, so the image processing effect can be improved, making the processed image more adaptable to the scene. In this case, according to the image processing method of the embodiment of the present application, while retaining accurate image details, the image processing process can be simplified, the amount of calculation can be reduced, and the execution efficiency can be improved.

According to the first aspect, in the first possible implementation of the image processing method, the abscissa of the histogram is a plurality of preset pixel value intervals, and the ordinate of the histogram is the number of pixels; According to the mapping parameter lookup table and the histogram corresponding to each partition, determine the mapping parameters corresponding to each partition, including: for any partition, according to the number of pixels in each pixel value interval included in the histogram corresponding to the partition , to obtain the feature vector corresponding to the partition; find a set of mapping parameters matching the feature vector in the mapping parameter lookup table as the mapping parameter corresponding to the partition.

In this way, the determination of the mapping parameters corresponding to each partition can be completed. In the case of an existing mapping parameter lookup table, the mapping parameters can be determined without model calculation, thereby reducing data processing costs.

According to the first possible implementation of the first aspect, in the second possible implementation of the image processing method, the set of mapping parameters includes a tangent to a mapping curve from the image to be processed to the processed image The first mapping parameter corresponding to the slope of the mapping curve and the second mapping parameter corresponding to the intercept of the tangent of the mapping curve.

By replacing the complete mapping curve with a set of mapping parameters corresponding to the slope and intercept of the tangent, the amount of calculation can be greatly reduced and the execution efficiency can be improved.

According to the second possible implementation of the first aspect, in the third possible implementation of the image processing method, the use of the mapping parameters to adjust the pixel values of the corresponding partitions in the image to be processed is obtained by processing The final image includes: for any pixel in any partition, determine the product of the first mapping parameter corresponding to the partition and the pixel value of the pixel; according to the sum of the product and the second mapping parameter corresponding to the partition, Get the pixel value of the pixel in the processed image.

In this way, the pixel value adjustment of the image based on the mapping parameters can actually be a simple linear function mapping, and only a small amount of calculation is required to achieve fast and effective pixel value adjustment.

According to the first aspect, or any possible implementation of the above first aspect, in the fourth possible implementation of the image processing method, the method further includes: constructing multiple sample feature vectors based on partition parameters , each sample feature vector corresponds to all combinations of the number of pixels in each pixel value interval of the sample histogram under the partition parameter; the multiple sample feature vectors are respectively input into the mapping parameter prediction model, and the mapping parameter prediction The model outputs the mapping parameters corresponding to each sample feature vector; the mapping parameter lookup table is generated according to the plurality of sample feature vectors and the mapping parameters corresponding to each sample feature vector, and the mapping parameter lookup table records each sample feature vector and its corresponding set of mapping parameters.

After the mapping parameter lookup table is obtained in this way, when the terminal device obtains the image to be processed, it does not need to go through the mapping parameter prediction model for calculation, thereby effectively reducing the amount of calculation.

According to the fourth possible implementation manner of the first aspect, in the fifth possible implementation manner of the image processing method, the value of the receptive field of the mapping parameter prediction model is smaller than the first threshold.

In this way, the generation efficiency of the mapping parameter lookup table can be improved, and the accuracy of the mapping parameter lookup table can be guaranteed.

According to the fifth possible implementation manner of the first aspect, in the sixth possible implementation manner of the image processing method, the terminal device further includes a second image processing model, and the second image processing model includes At least one layer, each layer includes at least one first image processing model, when the second image processing model includes multiple layers, each layer is connected in series; when each layer includes multiple first image processing models, each first image processing model The models are connected in parallel, and the partition parameters of the first image processing models in the same layer are different.

Improving the receptive field through parallel paths and strengthening the interaction of each partition through series paths can improve the image processing effect.

According to the sixth possible implementation of the first aspect, in the seventh possible implementation of the image processing method, for any layer of the second image processing model, the layer includes a first image processing model, the processed image obtained by the first image processing model is used as the image to be processed by the first image processing model of the next layer, and when the layer includes multiple first image processing models, the multiple first image processing models The obtained mean value of the processed image is used as the image to be processed by the first image processing model of the next layer.

In this way, it can be ensured that the image processing effect of the second image processing model can be progressive layer by layer.

In the second aspect, the embodiment of the present application provides an image processing device, the device is applied to the first image processing model on the terminal device, and the device includes: a splitting module, which is used to split the image to be processed based on the partition parameters Split into multiple partitions, the partition parameters indicate the size of each partition; the statistics module is used to perform histogram statistics for each partition, and obtain the histogram corresponding to each partition; the prediction module is used to search according to the mapping parameters The table and the histogram corresponding to each partition determine the mapping parameters corresponding to each partition; the mapping module is used to use the mapping parameters to adjust the pixel values of the corresponding partitions in the image to be processed to obtain the processed image.

According to the second aspect, in the first possible implementation of the image processing device, the abscissa of the histogram is a plurality of preset pixel value intervals, and the ordinate of the histogram is the number of pixels; According to the mapping parameter lookup table and the histogram corresponding to each partition, determine the mapping parameters corresponding to each partition, including: for any partition, according to the number of pixels in each pixel value interval included in the histogram corresponding to the partition , to obtain the feature vector corresponding to the partition; find a set of mapping parameters matching the feature vector in the mapping parameter lookup table as the mapping parameter corresponding to the partition.

According to the first possible implementation of the second aspect, in the second possible implementation of the image processing device, the set of mapping parameters includes a tangent to a mapping curve from the image to be processed to the processed image The first mapping parameter corresponding to the slope of the mapping curve and the second mapping parameter corresponding to the intercept of the tangent of the mapping curve.

According to the second possible implementation of the second aspect, in the third possible implementation of the image processing device, the adjustment of the pixel value of the corresponding partition in the image to be processed by using the mapping parameters is processed The final image includes: for any pixel in any partition, determine the product of the first mapping parameter corresponding to the partition and the pixel value of the pixel; according to the sum of the product and the second mapping parameter corresponding to the partition, Get the pixel value of the pixel in the processed image.

According to the second aspect, or any possible implementation of the above second aspect, in the fourth possible implementation of the image processing device, the device further includes: a construction module, configured to construct A plurality of sample feature vectors, each sample feature vector corresponds to all combinations of the number of pixels in each pixel value interval of the sample histogram under the partition parameter; the input module is used to input the multiple sample feature vectors into the A mapping parameter prediction model, wherein the mapping parameter prediction model outputs mapping parameters corresponding to each sample feature vector; a generating module configured to generate the mapping parameters according to the multiple sample feature vectors and the mapping parameters corresponding to each sample feature vector A lookup table, the mapping parameter lookup table records each sample feature vector and its corresponding set of mapping parameters.

According to a fourth possible implementation manner of the second aspect, in a fifth possible implementation manner of the image processing apparatus, a value of a receptive field of the mapping parameter prediction model is smaller than a first threshold.

According to the fifth possible implementation manner of the second aspect, in the sixth possible implementation manner of the image processing apparatus, the terminal device further includes a second image processing model, and the second image processing model includes At least one layer, each layer includes at least one first image processing model, when the second image processing model includes multiple layers, each layer is connected in series; when each layer includes multiple first image processing models, each first image processing model The models are connected in parallel, and the partition parameters of the first image processing models in the same layer are different.

According to the sixth possible implementation of the second aspect, in the seventh possible implementation of the image processing device, for any layer of the second image processing model, the layer includes a first image processing model, the processed image obtained by the first image processing model is used as the image to be processed by the first image processing model of the next layer, and when the layer includes multiple first image processing models, the multiple first image processing models The obtained mean value of the processed image is used as the image to be processed by the first image processing model of the next layer.

In a third aspect, an embodiment of the present application provides a terminal device, including: a processor; a memory for storing processor-executable instructions; wherein, the processor is configured to implement the above-mentioned first An image processing method in one or several possible implementations of the first aspect or the first aspect.

In the fourth aspect, the embodiments of the present application provide a non-volatile computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the above-mentioned first aspect or the first aspect can be realized One or several image processing methods in various possible implementations.

In the fifth aspect, the embodiments of the present application provide a computer program product, including computer readable code, or a non-volatile computer readable storage medium bearing computer readable code, when the computer readable code is stored in an electronic When running in the device, the processor in the electronic device executes the image processing method of the first aspect or one or more of the multiple possible implementations of the first aspect.

These and other aspects of the present application will be made more apparent in the following description of the embodiment(s).

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the application and, together with the specification, serve to explain the principles of the application.

FIG. 1 shows the target detection effect of the original image and the enhanced image processed by the image signal processor in the prior art when used for target detection.

FIG. 2 shows an example of an original image and an image processed by an image signal processor in the prior art.

Fig. 3 shows an exemplary application scenario of the image processing method according to the embodiment of the present application.

Fig. 4 shows a flowchart of an image processing method according to an embodiment of the present application.

Fig. 5 shows an exemplary structure of a first image processing model according to an embodiment of the present application.

Fig. 6 shows an exemplary working manner of the mapping parameter prediction module 300 according to the embodiment of the present application.

Fig. 7 shows an exemplary structure of a second image processing model according to an embodiment of the present application.

FIG. 8 shows a comparison between the target detection results of the processed image obtained according to the embodiment of the present application and the target detection results of the image obtained by the prior art.

FIG. 9 shows a comparison between the object detection results of the processed image obtained according to the embodiment of the present application and the object detection results of the image obtained by the prior art.

FIG. 10 shows a comparison between the target detection results of the processed images obtained according to the embodiment of the present application and the target detection results of the images obtained in the prior art.

FIG. 11 shows a comparison between the target detection results of the processed image obtained according to the embodiment of the present application and the target detection results of the image obtained by the prior art.

FIG. 12 shows a comparison between the object detection results of the processed image obtained according to the embodiment of the present application and the object detection results of the image obtained by the prior art.

FIG. 13 shows an exemplary structural diagram of an image processing device according to an embodiment of the present application.

Fig. 14 shows an exemplary structural diagram of a terminal device according to an embodiment of the present application.

Detailed ways

Various exemplary embodiments, features, and aspects of the present application will be described in detail below with reference to the accompanying drawings. The same reference numbers in the figures indicate functionally identical or similar elements. While various aspects of the embodiments are shown in drawings, the drawings are not necessarily drawn to scale unless specifically indicated.

The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as superior or better than other embodiments.

In addition, in order to better illustrate the present application, numerous specific details are given in the following specific implementation manners. It will be understood by those skilled in the art that the present application may be practiced without certain of the specific details. In some instances, methods, means, components and circuits well known to those skilled in the art have not been described in detail in order to highlight the gist of the present application.

Terms that may appear in this article are described below.

Neural network (neural network, NN): a computational model that imitates the structure and function of biological neural networks, and is mostly used in the field of artificial intelligence. By designing different network structures and using different training data, neural networks can achieve various functions.

Image signal processor (image signal processor, ISP): a processing unit mounted on a camera or other terminal equipment equipped with an image sensor, used to convert the photoelectric signal captured by the image sensor into a three-channel RGB image. The image signal processor includes multiple sub-modules, and each sub-module is usually realized by using a traditional algorithm (non-neural network algorithm), or a combination of a traditional algorithm and a neural network algorithm, or a pure neural network algorithm.

Mean average precision (mAP) of all classes (mean average precision, mAP): an indicator indicating the accuracy of model detection, generally the average of the accuracy of each category. The higher the average correct rate of the whole class, the higher the detection accuracy of the model.

Peak signal-to-noise ratio (peak signal-to-noise ratio, PSNR): In this field, it is used as one of the indicators to measure the similarity between the tested image and the real image. This indicator only pays attention to the pixel value of the image, but ignores the structure and texture information of the image. Generally speaking, the higher the peak signal-to-noise ratio, the closer the pixel values representing the tested image and the real image are.

Structural similarity (structure similarity, SSIM): In this field, it is used as one of the indicators to measure the similarity between the tested image and the real image. This indicator pays attention to the brightness, contrast and structural information of the image at the same time. Generally speaking, the higher the structural similarity, the closer the brightness, contrast and structural information representing the tested image and the real image are.

Histogram: A representation of the distribution of pixel values in an image. Input an image, assuming that the pixel value range of the image is 0-1, set multiple pixel value intervals of equal length according to the pixel value range, and count the number of pixels in different pixel value intervals in the image. A histogram is generally presented in the form of a histogram.

Receptive field (receptive field): A term in the field of deep neural networks, used to indicate the size of the sensory range of neurons at different positions within the neural network to the original image. The larger the value of the neuron's receptive field, the larger the range of the original image it can touch, and it also means that the neuron may contain more global and higher semantic features; while the smaller the value of the neuron's receptive field, It means that the features it contains tend to be more local and detailed. The value of the receptive field can be roughly used to judge the abstraction level of each layer of the neural network.

Image (quality) enhancement: refers to the technology of processing the brightness, color, contrast, saturation, dynamic range, etc. of the image to make the human eye sensory effect or certain indicators of the image better. In this paper, image enhancement and image quality enhancement express the same concept.

With the iterative update of neural network technology, the image signal processing method based on deep learning neural network has made great progress in the field of image signal processing. On existing terminal devices, such as mobile phones, tablet computers, surveillance cameras, etc., most of the image signal processing methods based on deep learning neural networks are used to realize all or part of the image signal processor. After multiple rounds of tuning by developers, these neural network-based image signal processors can output very high-quality enhanced images, and the color and contrast of the enhanced images are very close to the visual effects of the human eye.

However, on some terminal devices, the enhanced image output by the image signal processor may not be used for display to the user, but for target detection. The images can be mainly used for vehicle or pedestrian detection. An enhanced image with better human vision does not mean higher object detection accuracy. FIG. 1 shows the target detection effect of the original image and the enhanced image processed by the image signal processor in the prior art when used for target detection.

As shown in Figure 1, the left side is the enhanced image processed by the image signal processor and the human visual effect is improved, and the right side is the original image. The image on the left has higher brightness and contrast. When the two images are sent to the same object detection model (not shown), it can be seen that the object detection model cannot detect pedestrians in images with higher brightness and contrast. This also shows that images with good visual effects for human eyes may not be suitable for target detection.

In addition, the performance of the deep learning neural network is usually positively correlated with the complexity, that is, to obtain higher quality images, the neural network model usually needs to be designed more complex, resulting in an increase in the complexity of the image signal processor. FIG. 2 shows an example of an original image and an image processed by an image signal processor in the prior art. As shown in Figure 2, the upper left corner is the original image, the lower right corner is the real image, and the upper right corner is the application of a simple neural network model (calculation amount = 27.8*10^9 times of floating-point operations/second (giga floating ing -point operations per second, GFLOPS)) of the enhanced image 1 output by the image signal processor, the peak signal-to-noise ratio PSNR and structural similarity SSIM are 26.22 and 0.9773 respectively; the lower left corner is the application of a complex neural network model (calculation amount = 4956*10^9 floating-point operations per second) of the enhanced image 2 output by the image signal processor, the peak signal-to-noise ratio PSNR and the structural similarity SSIM are 34.95 and 0.9976, respectively. It can be seen that the image output by the complex neural network model is closer to the real image because the peak signal-to-noise ratio (PSNR) and structural similarity SSIM of the image are higher, and the image brightness and contrast are closer to the real image.

However, there may be many subtasks running on the chip synchronously on some terminal devices, and the computing power that can be allocated to individual subtasks is limited. Therefore, the models that can be deployed on the chip are generally not too complex, and some models with a very large amount of calculation are difficult to deploy on the chip.

In order to reduce the amount of computation and improve execution efficiency, prior art 1 proposes an image processing solution. The program mainly includes the following steps:

(1) Down-sampling the input image to obtain a low-resolution image, and predicting a set of fusion parameters w ₁ , w ₂ , w ₃ based on the down-sampled image;

(2) According to the fusion parameters w ₁ , w ₂ , and w ₃ , the three basic color look-up tables are fused to obtain the fused color look-up table;

(3) According to the fused color lookup table, the input image is interpolated to obtain the enhanced image.

The disadvantage of this solution is that when the size of the original image is large, for example, the original image is a 4K resolution image with 12 million pixels, and the total number of pixels is 3072x4096. Accurate local detail information. Moreover, the operation of interpolating the input image by the fused color lookup table depends on highly customized operators, which cannot be realized by existing operators, making it very difficult to deploy the model on the chip of the terminal device.

To sum up, the image processing methods in the prior art are either difficult to retain accurate image details during image processing, or the method implementation is relatively complex, with a large amount of calculation and low execution efficiency. In view of this, an image processing method, device, terminal equipment, storage medium, and program product are proposed. According to the image processing method of the embodiment of the present application, while retaining accurate image details, the image processing flow can be simplified, and the computation time can be reduced. volume and improve execution efficiency.

Fig. 3 shows an exemplary application scenario of the image processing method according to the embodiment of the present application.

As shown in FIG. 3 , the application scenario may include a terminal device, and the terminal device may be provided with a processor for running the second image processing model. The second image processing model may include at least one layer, and each layer includes one or more first image processing models for executing the image processing method of the embodiment of the present application. An example of the structure of the second image processing model can be referred to FIG. 7 . An example of the structure of the first image processing model can be referred to FIG. 5 .

The terminal device may also include an image sensor for collecting image data and an image signal processor for processing the image data, and the image output by the image signal processor may be an enhanced image optimized for human visual effects as described above , and can be used as the image to be processed in the first image processing model of the first layer. Alternatively, the terminal device may be connected to other terminal devices equipped with image sensors and image signal processors, so as to obtain enhanced images output by the image signal processors of other terminal devices, as the to-be-processed images of the first image processing model of the first layer image. Alternatively, the terminal device may be provided with an image signal processor, and be connected to other terminal devices provided with an image sensor, so as to acquire image data collected by the image sensor of other terminal devices, and obtain an enhanced image through processing by the image signal processor, as The image to be processed of the first image processing model of the first layer. The embodiment of the present application does not limit the specific acquisition manner of the image to be processed.

In the second image processing model, the first image processing model of each layer can process the image to be processed to obtain a processed image by executing the image processing method of the embodiment of the present application, so that the processed image is more suitable for target detection. The processed image obtained by the first image processing model of the same layer can be used to obtain the image to be processed by the first image processing model of the next layer. The processed image obtained by the first image processing model of the last layer can be used to obtain an image for object detection.

The terminal device of this application can be a smart phone, a wearable electronic device (such as a smart bracelet, a smart watch, etc.), a virtual reality device, a vehicle, an aircraft, etc. The computing device to which the device is connected. Based on this, the image processing method of the embodiment of the present application can be applied to image/video quality enhancement tasks in various application scenarios such as mobile phone photography, automatic driving, and safe city.

Fig. 4 shows a flowchart of an image processing method according to an embodiment of the present application.

As shown in FIG. 4, in a possible implementation, the method is applied to the first image processing model on the terminal device, and the method includes steps S41-S44:

Step S41, splitting the image to be processed into multiple partitions based on partition parameters, where the partition parameters indicate the size of each partition.

Fig. 5 shows an exemplary structure of a first image processing model according to an embodiment of the present application. As shown in FIG. 5 , the first image processing model may include a partition module 100 for performing step S41. Assuming that the size of the image to be processed is 1000x1000, and the partition parameter is 4x4, the image to be processed can be split based on the partition parameter to obtain 250x250 partitions, and each partition can have the same size and are all 4x4. In this way, the image to be processed with a large resolution can be divided into several small blocks with a small resolution.

Step S42, performing histogram statistics for each partition to obtain a histogram corresponding to each partition.

As shown in FIG. 5 , the first image processing model may include a partition histogram calculation module 200 for performing step S42. When the histogram statistics are performed, the histogram statistics may be performed on each of the 250×250 partitions obtained in step S41 to obtain a corresponding histogram. The abscissa of the generated histogram may be a plurality of preset pixel value intervals, and the ordinate may be the number of pixels. Assuming that the pixel value range of the image is 0-1, 4 pixel value ranges can be preset, and the corresponding pixel value ranges are 0-0.25, 0.25-0.5, 0.5-0.75, 0.75-1, since the size of each partition is 4x4, that is, each partition includes 16 pixels, and the value range of the ordinate corresponding to any pixel value interval of each histogram can be between 0-16.

Step S43: Determine the mapping parameters corresponding to each partition according to the mapping parameter lookup table and the histogram corresponding to each partition.

As shown in FIG. 5 , the first image processing model may include a mapping parameter prediction module 300 for performing step S43. The mapping parameter prediction module 300 may include a mapping parameter lookup table 302 . The mapping parameter lookup table 302 can be obtained from the mapping parameter prediction model 301, and the mapping parameter prediction model 301 can be used as a part of the mapping parameter prediction module 300, or not set in the first image processing model, which is not limited in this application. For an exemplary manner of obtaining the mapping parameter lookup table 302, reference may be made to related descriptions below.

The mapping parameter lookup table 302 may indicate the correspondence between partitions and mapping parameters. In the embodiment of the present application, the mapping parameters may be grouped as a unit, and a group of mapping parameters may perform the same function as the mapping curve between the original image and the processed image in the prior art. For a certain partition, the histogram obtained in step S42 can be directly used as an index to search the mapping parameter lookup table 302 to obtain the corresponding mapping parameter. Since there is no need to input the histogram into the mapping parameter prediction model 301 for calculation on the premise of the existing mapping parameter lookup table 302, this greatly reduces the amount of calculation and significantly improves efficiency. For an exemplary implementation manner of step S43, reference may be made to the further description below.

Step S44, using the mapping parameters to adjust the pixel values of the corresponding partitions in the image to be processed to obtain a processed image.

As shown in FIG. 5 , the first image processing model may include a mapping module 400 for performing step S44. The mapping parameter can be associated with at least one attribute such as the contrast and brightness of the partition, so that the contrast and brightness of the partition can be adjusted by adjusting the pixel value. Since the mapping parameters are in one-to-one correspondence with the partitions, that is, different partitions can use different mapping parameters, so adjusting the pixel values of the corresponding partitions in the image to be processed can make the pixel values of the processed image closer to the requirements of the application scene. That is, it is more suitable for target detection. For an exemplary implementation manner of step S44, reference may be made to the further description below.

According to the image processing method of the embodiment of the present application, the image to be processed is split into multiple partitions based on the partition parameter, and the partition parameter indicates the size of each partition, so that the overall receptive field of the first image processing model can be effectively improved through the partition parameter, At the same time control the amount of follow-up calculations and improve image processing efficiency; perform histogram statistics for each partition to obtain the histogram corresponding to each partition, and determine the corresponding histogram of each partition according to the mapping parameter lookup table and the histogram corresponding to each partition The mapping parameters greatly reduce the time-consuming of determining the mapping parameters, and further improve the image processing efficiency by reducing the delay; the processed image can be obtained by adjusting the pixel values of the corresponding partitions in the image to be processed by using the mapping parameters, where the mapping parameters are related to the partition One-to-one correspondence, so the image processing effect can be improved, making the processed image more adaptable to the scene. In this case, according to the image processing method of the embodiment of the present application, while retaining accurate image details, the image processing process can be simplified, the amount of calculation can be reduced, and the execution efficiency can be improved.

An exemplary method for determining the mapping parameters corresponding to each partition (step S43 ) in the embodiment of the present application is introduced below.

In a possible implementation, the abscissa of the histogram is a preset plurality of pixel value intervals, and the ordinate of the histogram is the number of pixels; step S43 includes:

For any partition, according to the number of pixels in each pixel value interval included in the histogram corresponding to the partition, the feature vector corresponding to the partition is obtained;

A set of mapping parameters matching the feature vector is found in the mapping parameter lookup table as the mapping parameters corresponding to the partition.

Referring to the above, the mapping parameter lookup table 302 can indicate the correspondence between the partition and the mapping parameter, for example, it can be used to record the correspondence between the feature vector corresponding to the partition and the mapping parameter, so each histogram can be further simplified to form a feature vector form, and then find the matching mapping parameter from the mapping parameter lookup table 302. When 4 pixel value intervals are preset, the histogram can be simplified into a feature vector by expressing each histogram in the form of a feature vector with a length of 4, and each value represents the value corresponding to 1 pixel value interval in the histogram. Vertical coordinate; when there are 250x250 partitions, a total of 250x250x4 feature vectors can be obtained.

In this way, the determination of the mapping parameters corresponding to each partition can be completed. In the case of an existing mapping parameter lookup table, the mapping parameters can be determined without model calculation, thereby reducing data processing costs.

In a possible implementation manner, a set of mapping parameters includes a first mapping parameter corresponding to the slope of the tangent line of the mapping curve from the image to be processed to the processed image and a first mapping parameter corresponding to the intercept of the tangent line of the mapping curve Two mapping parameters.

For example, the solutions in the prior art often adopt an adjustment strategy based on a mapping curve when adjusting the pixel value of an image, that is, the same mapping curve is used to adjust the pixel value at any position. As a result, while the pixel values in some areas are stretched, the pixel values in other areas must be compressed, that is, the same mapping curve is not suitable for different areas. If a mapping curve is predicted for each region, although the accuracy of pixel value adjustment of the image can be guaranteed, it will lead to serious calculation waste.

In this regard, the embodiment of the present application proposes to use the slope and intercept of the tangent of the curve instead of the complete mapping curve as parameters for pixel value adjustment. Since the pixel values of the pixels in a partition are relatively close, a partition can be approximately regarded as a point on the curve. Therefore, the determination of the complete mapping curve corresponding to the partition can be simplified as determining the slope and intercept of the tangent line at this point on the complete mapping curve. The slope and the intercept can be represented by the first mapping parameter and the second mapping parameter respectively. The first mapping parameter and the second mapping parameter of the tangent at the same point can be used as a set of mapping parameters. In this case, for any pixel in the same partition, although the pixel values may be different, they can correspond to the same mapping parameters.

By replacing the complete mapping curve with a set of mapping parameters corresponding to the slope and intercept of the tangent, the amount of calculation can be greatly reduced and the execution efficiency can be improved.

The following describes an exemplary method of adjusting the pixel values of the corresponding partitions in the image to be processed by using the mapping parameters to obtain the processed image (step S44 ) in the embodiment of the present application.

In a possible implementation, step S44 includes:

For any pixel in any partition, determine the product of the first mapping parameter corresponding to the partition and the pixel value of the pixel;

The pixel value of the pixel in the processed image is obtained according to the sum of the product and the second mapping parameter corresponding to the partition.

For example, assuming that the partition parameter is 4x4 and there are 250x250 partitions in total, the pixel values of pixels in partition i (1≤i≤250x250) can be regarded as a 4x4 matrix of pixel values. The first mapping parameter and the second mapping parameter obtained in step S43 are x1 and x2 respectively, then the product of the first mapping parameter x1 corresponding to the partition i and the pixel value matrix img_i of the partition i can be x1*img_i; the processed image The pixel value matrix out_i of the partition i may be the sum of x1*img_i and the second mapping parameter x2 corresponding to the partition i, that is, out_i=x1*img_i+x2.

In this way, the pixel value adjustment of the image based on the mapping parameters can actually be a simple linear function mapping, and only a small amount of calculation is required to achieve fast and effective pixel value adjustment.

Taking the mapping parameter prediction model 301 set in the mapping parameter prediction module 300 as an example, the training method of the mapping parameter prediction model 301 and an exemplary method of using the trained mapping parameter prediction model 301 to generate a mapping parameter lookup table are introduced below.

The mapping parameter prediction model 301 can be a neural network model, which can be trained in the way of the prior art. Through multiple rounds of iterations, the mapping parameter prediction model has the capability of "inputting the feature vector corresponding to the histogram of a partition and predicting the corresponding mapping parameters". Function. For example, the sample image can be split into multiple sample partitions based on the partition parameters, and the histogram statistics are performed on each sample partition to obtain the sample histogram corresponding to each sample partition. The exemplary implementation method is the same as that described above Steps S41 and S42 may be the same, and may also be completed by the partition module 100 and the partition histogram calculation module 200 described above, and will not be repeated here. Wherein, the partition parameters used when splitting the sample image may be the same as or different from the partition parameters used in step S41, which is not limited in this application.

After the sample histogram corresponding to the partition is obtained, the sample histogram can be input into the above-mentioned mapping parameter prediction module 300 . Fig. 6 shows an exemplary working manner of the mapping parameter prediction module 300 according to the embodiment of the present application. As shown in FIG. 6, when the terminal device is in the training phase, the mapping parameter prediction module 300 can first obtain the feature vector according to the histogram, as the input of the mapping parameter prediction model 301, and train the mapping parameter prediction model 301 through multiple iterations. The application does not limit the specific training method of the mapping parameter prediction model 301 . When the terminal device is not in the training phase, the mapping parameter prediction module 300 can execute the mapping parameter lookup table 302 to perform the above-mentioned step S43.

An exemplary acquisition method of the mapping parameter lookup table 302 is introduced below.

In a possible implementation, the method further includes:

Construct multiple sample feature vectors based on partition parameters, each sample feature vector corresponds to all combinations of the number of pixels in each pixel value interval of the sample histogram under the partition parameters;

Inputting a plurality of sample feature vectors into the mapping parameter prediction model respectively, and the mapping parameter prediction model outputs the mapping parameters corresponding to each sample feature vector;

A mapping parameter lookup table is generated according to the multiple sample feature vectors and the mapping parameters corresponding to each sample feature vector, and the mapping parameter lookup table records each sample feature vector and a corresponding set of mapping parameters.

For example, after the mapping parameter prediction model 301 is trained, multiple sample feature vectors can be constructed based on the partition parameters, so that each sample feature vector corresponds to the number of pixels in each pixel value interval of the sample histogram under the partition parameters. Combining, that is, traversing all possible situations of the histogram under each partition parameter, obtaining the corresponding sample feature vector and inputting multiple sample feature vectors into the mapping parameter prediction model 301, and the mapping parameter prediction model 301 outputs each sample feature Vector corresponding to the mapping parameters. This process can be done offline. A mapping parameter lookup table is generated according to multiple sample feature vectors and mapping parameters corresponding to each sample feature vector, so that the mapping parameter lookup table records each sample feature vector and its corresponding set of mapping parameters, for example, all <input feature Vector, output the mapping parameter> save it, and then the mapping parameter lookup table 302 can be formed.

After the mapping parameter lookup table is obtained in this way, when the terminal device obtains the image to be processed, it does not need to go through the mapping parameter prediction model for calculation, thereby effectively reducing the amount of calculation.

In a possible implementation manner, the value of the receptive field of the mapping parameter prediction model is smaller than a first threshold.

See above, the receptive field represents the size of the receptive range of neurons at different positions in the neural network to the original image. For the mapping parameter prediction model, the receptive field size can be the convolution of the convolution operation in the model nuclear size. It can be seen that the larger the receptive field is, the more complex the convolution operation is, and the slower the speed of generating the mapping parameter lookup table 302 is, meanwhile, the complex convolution operation has a greater impact on the accuracy of the mapping parameters.

In this regard, the embodiment of the present application proposes to make the value of the receptive field of the mapping parameter prediction model smaller than the first threshold, so as to ensure the generation speed of the mapping parameter lookup table, and reduce the size of the receptive field of the mapping parameter prediction model on the accuracy of the mapping parameters. Influence. The specific numerical value of the first threshold can be set as required, which is not limited in the present application. In one example, the value of the receptive field of the mapping parameter prediction model can be set to 1×1, in this case, the mapping parameter can only be related to the partition histogram, and not be affected by other factors. And because the receptive field of the first image processing model in the embodiment of the present application can be increased by adjusting the partition parameters, even if the receptive field of the mapping parameter prediction model is reduced, the loss can be reduced by increasing the partition parameters in subsequent operations. The accuracy of the network is compensated, so that it does not reduce the overall effect of the network.

In this way, the generation efficiency of the mapping parameter lookup table can be improved, and the accuracy of the mapping parameter lookup table can be guaranteed.

In a possible implementation manner, the terminal device further includes a second image processing model, the second image processing model includes at least one layer, and each layer includes at least one first image processing model,

When the second image processing model includes multiple layers, each layer is connected in series;

When each layer includes multiple first image processing models, the first image processing models are connected in parallel, and the partition parameters of the first image processing models in the same layer are different.

For example, the partition parameters of the first image processing model are usually unchanged, so in order to further improve the image processing effect, the embodiment of the present application also proposes to connect multiple first image processing models in series and/or in parallel to obtain the second image processing model , using the second image processing model to perform image processing. Fig. 7 shows an exemplary structure of a second image processing model according to an embodiment of the present application.

On the one hand, the image processing effect of the neural network-based model is strongly related to the receptive field of the network. The receptive field of a single first image processing model is relatively limited. In order to further improve the receptive field, one layer of the second image processing model can be designed to include parallel multiple first image processing models, and make the partition parameters of the first image processing models of the same layer different, for example, the partition parameters of the first image processing models of the same layer can be increased sequentially to realize the increase of the receptive field . As shown in FIG. 7 , each layer can be set to include three parallel first image processing models, and the partition parameters of each first image processing model are 4x4, 8x8, and 16x16 respectively.

Those skilled in the art should understand that the increase of the receptive field can also be achieved by means of other existing technologies. For example, one existing technology is to increase the receptive field by reserving "holes". When using this method, the partition can be maintained The parameters remain unchanged. When the image to be processed is split into multiple partitions, some pixels are skipped, so that the pixels of the same partition have a larger area in the image to be processed. The present application does not limit the way of increasing the receptive field.

On the other hand, the obtained partitions of the model with only one layer are independent of each other. In order to realize the interaction of different partitions, the second image processing model can be designed to include multiple layers, and the layers are connected in series. Through multiple serial channels, the final image processing effect is progressively improved layer by layer. In this embodiment, as shown in FIG. 7 , three layers can be set, and the partition parameters of the first image processing model in each layer are 4x4, 8x8, and 16x16 respectively. In actual application, the partition parameters of the first image processing model of each layer may also be different, which is not limited in the present application.

Improving the receptive field through parallel paths and strengthening the interaction of each partition through series paths can improve the image processing effect.

In a possible implementation, for any layer of the second image processing model, when the layer includes a first image processing model, the processed image obtained by the first image processing model is used as the first layer of the next layer. The image to be processed of the image processing model, when the layer includes a plurality of first image processing models, the mean value of the processed image obtained by the plurality of first image processing models is used as the image to be processed of the first image processing model of the next layer .

For example, a layer of the second image processing model may include one first image processing model, or may include multiple first image processing models connected in parallel. For these two different cases, the overall output of this layer may be different.

Wherein, when one layer of the second image processing model includes a first image processing model, the processed image obtained by the first image processing model can be directly used as the image to be processed by the first image processing model of the next layer, if the layer is already the last layer, the processed image obtained by the first image processing model can be directly output as the final processed image.

When one layer of the second image processing model includes a plurality of first image processing models, the mean value of the processed image obtained by the plurality of first image processing models may be used as the image to be processed of the first image processing model of the next layer, If the layer is already the last layer, the mean value of the processed images obtained by multiple first image processing models can be output as the final processed image.

In this way, it can be ensured that the image processing effect of the second image processing model can be progressive layer by layer.

Table 1 and Table 2 show the processed image obtained using the first image processing model and the second image processing model of the embodiment of the present application, and the processed image obtained by the image processing method of the prior art when applied to target detection The detection effect of the scene.

Table 1

AP1 AP2 AP3 AP4 mAP FPS Benchmark 1 68.59 75.47 86.55 70.18 75.20 - Benchmark 2 76.04 79.14 87.94 67.16 77.57 164.32 Ours 11 78.84 78.08 89.94 69.33 79.05 554.71 Ours 33 79.53 78.84 89.87 70.51 79.69 63.24

Table 2

AP1 AP2 AP3 mAP FPS Benchmark 1 50.33 40.30 53.99 48.21 - Benchmark 2 48.81 42.99 58.12 49.97 164.32 Ours 11 49.10 44.19 59.75 51.01 554.71 Ours 33 49.58 45.89 60.75 52.07 63.24

In Table 1 and Table 2, benchmark 1 indicates that what is input to the detector is an image obtained by an image signal processor; benchmark 2 indicates that what is input to the detector is an image obtained by adopting the scheme of prior art one; Ours 11 indicates that what is input to the detector It is the image obtained by adopting a first image processing model of the embodiment of the present application; Ours 33 represents the image obtained by the second image processing model of the embodiment of the present application, and Ours 33 represents the input detector, wherein the second image processing model includes three layers , each layer includes three first image processing models.

Table 1 shows the detection results of the images obtained by each scheme on the two-stage detector. Table 2 shows the detection results of the images obtained by each scheme on the one-stage detector. Both detectors are existing detectors in the prior art, and their detection principles are different.

The average accuracy rate (average precision, AP) is an index indicating the detection accuracy of a detector on a certain class. The higher the index, the higher the detection accuracy of the detector on this category. In Table 1 and Table 2, the detection accuracy of 3 categories (AP1, AP2, AP3) is detected.

Mean average precision (mAP) is the arithmetic mean of the average correct rate of each category. Generally, the larger the value, the higher the detection accuracy of the detector. In Table 1 and Table 2, it can be the arithmetic mean of AP1, AP2, and AP3.

Frame rate (frames per second, FPS) is an indicator of calculation speed, representing the number of executions in one second. In Table 1 and Table 2, it may be the calculation speed of the first image processing model/the second image processing model for an image to be processed with a resolution of 3072x4096 on an existing graphics processor platform. Among them, Benchmark 1 does not post-process the image output by the image signal processor, so the corresponding calculation speed cannot be counted.

According to the above two tables, it can be seen that the embodiments of the present application can have good effects on detectors with different design principles.

8 to 12 show the comparison of the target detection results of the processed images obtained according to the embodiment of the present application and the target detection results of the images obtained in the prior art.

The object detection results of the image obtained by the image signal processor, the image obtained by the prior art 1, and the image obtained by using the first image processing model of the embodiment of the present application are compared. For the convenience of comparison, the differences in the target detection results of the three schemes are marked with arrows in Figure 8, Figure 9, and Figure 12. It can be seen that in Fig. 8, 2 objects are detected in the target detection result of the original image, and 3 objects are detected in the target detection result of the image obtained in prior art 1, and the first image processing of the embodiment of the present application is adopted There are 4 objects detected in the object detection result of the image obtained by the model.

In Figure 9, 2 objects are detected in the target detection result of the original image, but the detection frame of one of the objects deviates greatly from the actual object. One object is detected in the target detection result of the image obtained in the prior art, and 2 objects are detected in the target detection result of the image obtained by using the first image processing model of the embodiment of the present application, and each object is different from the actual object The matching degree of the detection boxes is relatively high.

In Fig. 10, no object is detected in the target detection result of the original image, and no object is detected in the target detection result of the image obtained in prior art 1, and the target of the image obtained by using the first image processing model of the embodiment of the present application 1 object was detected in the detection result.

In Fig. 11, 1 object is detected in the object detection result of the original image. One object is detected in the object detection result of the image obtained in prior art 1, and two objects are detected in the object detection result of the image obtained by using the first image processing model of the embodiment of the present application.

In Figure 12, 4 objects are detected in the object detection results of the original image, including 3 pedestrians and 1 vehicle. In the target detection result of the image obtained in prior art 1, 5 objects are detected, including 2 pedestrians and 3 vehicles. In the object detection result of the image obtained by using the first image processing model of the embodiment of the present application, 6 objects are detected, including 3 pedestrians and 3 vehicles.

Therefore, the image obtained by the first image processing model in the embodiment of the present application is superior to the comparison scheme in terms of false detection, missed detection, matching degree between the detection frame and the actual object, and the like.

An embodiment of the present application provides an image processing device, and FIG. 13 shows an exemplary structural diagram of the image processing device according to the embodiment of the present application.

As shown in Figure 13, the device is applied to the first image processing model on the terminal device, and the device includes:

A splitting module 141, configured to split the image to be processed into multiple partitions based on partition parameters, the partition parameters indicating the size of each partition;

Statistical module 142, is used for carrying out histogram statistics for each partition, obtains the histogram corresponding to each partition;

A prediction module 143, configured to determine the mapping parameters corresponding to each partition according to the mapping parameter lookup table and the histogram corresponding to each partition;

The mapping module 144 is configured to use the mapping parameters to adjust pixel values of corresponding subregions in the image to be processed to obtain a processed image.

Wherein, the splitting module 141 can be realized by the above-mentioned partition module 100, the statistical module 142 can be realized by the above-mentioned partition histogram calculation module 200, and the prediction module 143 can be realized by the above-mentioned mapping parameter prediction model 301, and the mapping module 144 can be realized by the above-mentioned mapping module 400.

In a possible implementation manner, the abscissa of the histogram is a plurality of preset pixel value intervals, and the ordinate of the histogram is the number of pixels; the lookup table according to the mapping parameters and each partition Corresponding histogram, determining the mapping parameters corresponding to each partition, including: for any partition, according to the number of pixels in each pixel value interval included in the histogram corresponding to the partition, obtain the feature vector corresponding to the partition; in the A set of mapping parameters matching the feature vector is found in the mapping parameter lookup table as the mapping parameters corresponding to the partition.

In a possible implementation manner, the set of mapping parameters includes a first mapping parameter corresponding to the slope of the tangent line of the mapping curve from the image to be processed to the processed image and the intercept of the tangent line to the mapping curve The corresponding second mapping parameter.

In a possible implementation manner, the adjusting the pixel value of the corresponding partition in the image to be processed by using the mapping parameters to obtain the processed image includes: for any pixel in any partition, determining that the partition corresponds to The product of the first mapping parameter and the pixel value of the pixel; according to the sum of the product and the second mapping parameter corresponding to the partition, obtain the pixel value of the pixel in the processed image.

In a possible implementation manner, the device further includes:

A building block for constructing a plurality of sample feature vectors based on partition parameters, each sample feature vector corresponding to all combinations of the number of pixels in each pixel value interval of the sample histogram under the partition parameters;

An input module, configured to respectively input the plurality of sample feature vectors into the mapping parameter prediction model, and the mapping parameter prediction model outputs a mapping parameter corresponding to each sample feature vector;

A generation module, configured to generate the mapping parameter lookup table according to the plurality of sample feature vectors and the mapping parameters corresponding to each sample feature vector, and the mapping parameter lookup table records each sample feature vector and its corresponding set of mappings parameter.

Wherein, the construction module, the input module, and the generation module can be realized by the above-mentioned mapping parameter prediction module 300 .

In a possible implementation manner, the value of the receptive field of the mapping parameter prediction model is smaller than a first threshold.

In a possible implementation manner, the terminal device further includes a second image processing model, the second image processing model includes at least one layer, each layer includes at least one first image processing model, and the second image processing model When the processing model includes multiple layers, the layers are connected in series; when each layer includes multiple first image processing models, the first image processing models are connected in parallel, and the partition parameters of the first image processing models in the same layer are different.

In a possible implementation manner, for any layer of the second image processing model, when the layer includes a first image processing model, the processed image obtained by the first image processing model is used as the The image to be processed by the first image processing model, when the layer includes multiple first image processing models, the mean value of the processed image obtained by the multiple first image processing models is used as the image to be processed by the first image processing model of the next layer Process images.

An embodiment of the present application provides a terminal device, including: a processor and a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above method when executing the instructions.

An embodiment of the present application provides a non-volatile computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the foregoing method is implemented.

An embodiment of the present application provides a computer program product, including computer-readable codes, or a non-volatile computer-readable storage medium bearing computer-readable codes, when the computer-readable codes are stored in a processor of an electronic device When running in the electronic device, the processor in the electronic device executes the above method.

Fig. 14 shows an exemplary structural diagram of a terminal device according to an embodiment of the present application.

As shown in Figure 14, terminal devices may include mobile phones, foldable electronic devices, handheld computers, ultra-mobile personal computers (ultra-mobile persona l computer, UMPC), personal digital assistants (persona l digital l ass i stant, PDA) ), augmented reality (augmented real ity, AR) equipment, virtual reality (virtua l real ity, VR) equipment, artificial intelligence (artificia l intelligence, AI) equipment, wearable equipment, vehicle-mounted equipment, smart At least one of household equipment, or smart city equipment. The embodiment of the present application does not specifically limit the specific type of the terminal device.

The terminal device may include a processor 110, a memory 121, and a communication module 160. It can be understood that, the structure shown in the embodiment of the present application does not constitute a specific limitation on the terminal device. In other embodiments of the present application, the terminal device may include more or fewer components than shown in the figure, or combine certain components, or separate certain components, or arrange different components. The illustrated components can be realized in hardware, software or a combination of software and hardware.

The processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (app lication processor, AP), a modem processor, a graphics processing unit (graphics processing ingun it, GPU) , image signal processor (image signal processor, ISP), controller, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural network processor ( neura l-network processing unit it, NPU), etc. Wherein, different processing units may be independent devices, or may be integrated in one or more processors.

The processor can generate an operation control signal according to the instruction opcode and the timing signal, and complete the control of fetching and executing the instruction.

A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in processor 110 may be a cache memory. The memory may store instructions or data used or frequently used by the processor 110, such as partition parameters in the embodiment of the present application. If the processor 110 needs to use the instruction or data, it can be called directly from the memory. Repeated access is avoided, and the waiting time of the processor 110 is reduced, thereby improving the efficiency of the system.

The memory 121 may be used to store computer-executable program code including instructions. The memory 121 may include an area for storing programs and an area for storing data. Wherein, the stored program area can store an operating system, an application program required by at least one function (such as a mapping parameter lookup table) and the like. The storage data area may store data acquired or created during the use of the terminal device (such as a histogram, etc.). In addition, the memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, flash memory device, universal flash memory (universal lf flash storage, UFS) and the like. The processor 110 implements various methods performed by the terminal device above by executing instructions stored in the memory 121 and/or instructions stored in a memory provided in the processor.

The communication module 160 can be used to receive data (such as image data and enhanced images in the embodiment of the present application) from other devices or devices through wireless communication/wired communication, and output data to other devices or devices. For example, WLAN (such as Wi-Fi network), Bluetooth (Bluetooth, BT), global navigation satellite system (globa lnavigation satellite system, GNSS), frequency modulation (frequency modu l at ion, FM), short-distance Wireless communication technology (near field communication, NFC), infrared technology (infrared, IR) and other wireless communication solutions.

A computer readable storage medium may be a tangible device that can retain and store instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (non-exhaustive list) of computer-readable storage media include: portable computer disk, hard disk, random access memory (Random Access Memory, RAM), read only memory (Read Only Memory, ROM), erasable Type Programmable Read-Only Memory (Electrica l ly Programmable Read-Only-Memory, EPROM or flash memory), Static Random-Access Memory (Static Random-Access Memory, SRAM), Portable Compact Disk Read-Only Memory (Compact Di scRead-Only Memory (CD-ROM), Digital Versatile Disk (DVD), memory sticks, floppy disks, mechanically encoded devices such as punched cards or raised in grooves on which instructions are stored structure, and any suitable combination of the above.

Computer readable program instructions or codes described herein may be downloaded from a computer readable storage medium to a respective computing/processing device, or downloaded to an external computer or external storage device over a network, such as the Internet, local area network, wide area network, and/or wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and/or edge servers. A network adapter card or a network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing/processing device .

Computer program instructions for performing the operations of the present application may be assembly instructions, instruction set architecture (Instruction Set Architecture, ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or in the form of a or any combination of programming languages, including object-oriented programming languages—such as Small ltalk, C++, etc., and conventional procedural programming languages—such as “C” or similar programming languages. Computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server implement. In cases involving a remote computer, the remote computer can be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or it can be connected to an external computer such as use an Internet service provider to connect via the Internet). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (Field-Programmable Gate Array, FPGA) or programmable logic arrays ( Programmable Logic Array, PLA), the electronic circuit can execute computer-readable program instructions, thereby realizing various aspects of the present application.

Aspects of the present application are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It should be understood that each block of the flowcharts and/or block diagrams, and combinations of blocks in the flowcharts and/or block diagrams, can be implemented by computer-readable program instructions.

These computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine such that when executed by the processor of the computer or other programmable data processing apparatus , producing an apparatus for realizing the functions/actions specified in one or more blocks in the flowchart and/or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause computers, programmable data processing devices and/or other devices to work in a specific way, so that the computer-readable medium storing instructions includes An article of manufacture comprising instructions for implementing various aspects of the functions/acts specified in one or more blocks in flowcharts and/or block diagrams.

It is also possible to load computer-readable program instructions into a computer, other programmable data processing device, or other equipment, so that a series of operational steps are performed on the computer, other programmable data processing device, or other equipment to produce a computer-implemented process , so that instructions executed on computers, other programmable data processing devices, or other devices implement the functions/actions specified in one or more blocks in the flowcharts and/or block diagrams.

The flowchart and block diagrams in the figures show the architecture, functions and operations of possible implementations of apparatuses, systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in a flowchart or block diagram may represent a module, a portion of a program segment, or an instruction that includes one or more Executable instructions. In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks in succession may, in fact, be executed substantially concurrently, or they may sometimes be executed in the reverse order, depending upon the functionality involved.

It should also be noted that each block in the block diagrams and/or flowcharts, and combinations of blocks in the block diagrams and/or flowcharts, can be implemented with hardware (such as a circuit or an ASIC (App l icationSpecific Integrated Circuit (ASIC)) or a combination of hardware and software, such as firmware.

Although the present invention has been described in conjunction with various embodiments herein, in the process of implementing the claimed invention, those skilled in the art can understand and Other variations of the disclosed embodiments are implemented. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that these measures cannot be combined to advantage.

Having described various embodiments of the present application above, the foregoing description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and alterations will be apparent to those of ordinary skill in the art without departing from the scope of the described embodiments. The terminology used herein is chosen to best explain the principle of each embodiment, practical application or improvement of technology in the market, or to enable other ordinary skilled in the art to understand each embodiment disclosed herein.

Claims (12)

1. An image processing method, characterized in that the method is applied to a first image processing model on a terminal device, the method comprising:

splitting an image to be processed into a plurality of partitions based on partition parameters, the partition parameters indicating a size of each partition;

carrying out histogram statistics on each partition to obtain a histogram corresponding to each partition;

determining mapping parameters corresponding to each partition according to the mapping parameter lookup table and the histogram corresponding to each partition;

and adjusting pixel values of corresponding partitions in the image to be processed by using the mapping parameters to obtain the processed image.

2. The method of claim 1, wherein the abscissa of the histogram is a preset plurality of pixel value intervals, and the ordinate of the histogram is the number of pixels;

the determining the mapping parameter corresponding to each partition according to the mapping parameter lookup table and the histogram corresponding to each partition includes:

aiming at any one partition, obtaining a feature vector corresponding to the partition according to the pixel number of each pixel value interval included in the histogram corresponding to the partition;

and finding a group of mapping parameters matched with the feature vector in the mapping parameter lookup table as mapping parameters corresponding to the partition.

3. The method of claim 2, wherein the set of mapping parameters includes a first mapping parameter corresponding to a slope of a tangent of a mapping curve of the image to be processed to the processed image and a second mapping parameter corresponding to an intercept of the tangent of the mapping curve.

4. A method according to claim 3, wherein said adjusting pixel values of corresponding partitions in the image to be processed using said mapping parameters to obtain a processed image comprises:

for any pixel in any one partition, determining the product of a first mapping parameter corresponding to the partition and the pixel value of the pixel;

and obtaining the pixel value of the pixel in the processed image according to the sum of the product and the second mapping parameter corresponding to the partition.

5. The method according to any one of claims 1-4, further comprising:

constructing a plurality of sample feature vectors based on the partition parameters, wherein each sample feature vector corresponds to all combinations of pixel numbers of each pixel value interval of the sample histogram under the partition parameters;

respectively inputting the plurality of sample feature vectors into a mapping parameter prediction model, and outputting mapping parameters corresponding to each sample feature vector by the mapping parameter prediction model;

And generating the mapping parameter lookup table according to the sample feature vectors and the mapping parameters corresponding to the sample feature vectors, wherein the mapping parameter lookup table records each sample feature vector and a group of mapping parameters corresponding to each sample feature vector.

6. The method of claim 5, wherein the magnitude of the receptive field of the mapping parametric prediction model is less than a first threshold.

7. The method of claim 6, further comprising a second image processing model on the terminal device, the second image processing model comprising at least one layer, each layer comprising at least one first image processing model,

when the second image processing model comprises a plurality of layers, the layers are connected in series;

when each layer comprises a plurality of first image processing models, the first image processing models are connected in parallel, and the partition parameters of the first image processing models of the same layer are different.

8. The method according to claim 7, wherein for any one layer of the second image processing model, the layer includes a first image processing model, a processed image obtained by the first image processing model is taken as an image to be processed of a first image processing model of a next layer, and the layer includes a plurality of first image processing models, and a mean value of processed images obtained by the plurality of first image processing models is taken as an image to be processed of a first image processing model of a next layer.

9. An image processing apparatus, characterized in that the apparatus is applied to a first image processing model on a terminal device, the apparatus comprising:

the splitting module is used for splitting the image to be processed into a plurality of partitions based on partition parameters, and the partition parameters indicate the size of each partition;

the statistics module is used for carrying out histogram statistics on each partition to obtain a histogram corresponding to each partition;

the prediction module is used for determining the mapping parameters corresponding to each partition according to the mapping parameter lookup table and the histogram corresponding to each partition;

and the mapping module is used for adjusting the pixel values of the corresponding partitions in the image to be processed by using the mapping parameters to obtain the processed image.

10. A terminal device, comprising:

a processor;

a memory for storing processor-executable instructions;

wherein the processor is configured to implement the method of any of claims 1-8 when executing the instructions.

11. A non-transitory computer readable storage medium having stored thereon computer program instructions, which when executed by a processor, implement the method of any of claims 1-8.

12. A computer program product comprising computer readable code, or a non-transitory computer readable storage medium carrying computer readable code, characterized in that a processor in an electronic device performs the method of any one of claims 1-8 when the computer readable code is run in the electronic device.

Images