CN116993604B

CN116993604B - Image enhancement method and device, electronic equipment and readable storage medium

Info

Publication number: CN116993604B
Application number: CN202310863685.1A
Authority: CN
Inventors: 陈学伟; 冯杰
Original assignee: Longxin Zhongke Shanxi Technology Co ltd
Current assignee: Longxin Zhongke Shanxi Technology Co ltd
Priority date: 2023-07-13
Filing date: 2023-07-13
Publication date: 2026-01-27
Anticipated expiration: 2043-07-13
Also published as: CN116993604A

Abstract

This application discloses an image enhancement method, apparatus, electronic device, and readable storage medium, comprising: amplifying the pixel intensity of an acquired original low-light image according to a pre-amplification factor to obtain a pre-amplified image; performing multiple downsampling operations on the pre-amplified image to obtain a downsampled image corresponding to each downsampling operation; inputting each downsampled image into a corresponding scale calculation model to output a scale calculation image corresponding to the downsampled image; the scale calculation model is used to perform at least one convolution processing calculation on the downsampled image; fusing all the scale calculation images to obtain a fused image, performing convolution processing, and outputting a target image after upsampling. The target image has higher image quality than the original low-light image, realizing the enhancement of low-light images without the need for manual processing by staff, improving work efficiency, and solving the problem of low work efficiency in prior art.

Description

Image enhancement method and device, electronic equipment and readable storage medium

Technical Field

The application relates to an image enhancement method, an image enhancement device, electronic equipment and a readable storage medium.

Background

In order to enhance a low-light image, an image enhancement method is required.

In the prior art, workers manually enhanced low-light images using picture editing software (e.g., adobe Photoshop).

In the process of realizing the application, the inventor finds that at least the following problems exist in the prior art, namely, the working efficiency is low because workers use picture editing software to manually enhance the low-light image.

Disclosure of Invention

The application aims to provide an image enhancement method, an image enhancement device, electronic equipment and a readable storage medium, which at least solve the problem of low working efficiency caused by manually enhancing a low-light image by using picture editing software by staff in the prior art.

In order to solve the technical problems, the application is realized as follows:

in a first aspect, an embodiment of the present application provides a method for enhancing an image, including:

amplifying the pixel intensity of the acquired original low-light image according to the pre-amplification coefficient to obtain a pre-amplified image;

Respectively carrying out multiple downsampling operations on the pre-amplified image to respectively obtain downsampled images corresponding to each downsampling operation, wherein the sampling multiples corresponding to the multiple downsampling operations are sequentially decreased;

Inputting each downsampled image into a corresponding scale calculation model respectively, and outputting a scale calculation image corresponding to the downsampled image, wherein the scale calculation model is used for carrying out convolution processing calculation on the downsampled image at least once;

and fusing all the scale calculation images to obtain a fused image, performing convolution processing, and outputting a target image through up-sampling operation.

In a second aspect, an embodiment of the present application further provides an image enhancement apparatus, where the apparatus includes:

the pre-amplification module is used for amplifying the pixel intensity of the acquired original low-light image according to the pre-amplification coefficient to obtain a pre-amplified image;

The downsampling module is used for respectively carrying out downsampling operation on the pre-amplified image for a plurality of times to respectively obtain downsampled images corresponding to each downsampling operation;

The scale calculation module is used for respectively inputting each downsampled image into a corresponding scale calculation model and outputting a scale calculation image corresponding to the downsampled image, wherein the scale calculation model is used for carrying out convolution processing calculation on the downsampled image at least once, and the relation between the sampling multiple and the times of the convolution processing calculation of the scale calculation model is positive correlation;

and the fusion module is used for fusing all the scale calculation images to obtain a fused image, then carrying out convolution processing, and outputting a target image through up-sampling operation.

In a third aspect, an embodiment of the present application further provides an electronic device, including a processor, a memory, and a program or instructions stored on the memory and executable on the processor, the program or instructions implementing the steps of the method according to the first aspect when executed by the processor.

In a fourth aspect, embodiments of the present application also provide a readable storage medium having stored thereon a program or instructions which when executed by a processor implement the steps of the method according to the first aspect.

In the embodiment of the application, the pixel intensity of an acquired original low-light image is amplified according to a pre-amplification coefficient to obtain the pre-amplified image, the pre-amplified image is respectively subjected to a plurality of downsampling operations to respectively obtain downsampling images corresponding to each downsampling operation, the sampling times corresponding to each downsampling operation are sequentially reduced, each downsampling image is respectively input into a corresponding scale calculation model to output a scale calculation image corresponding to the downsampling image, the scale calculation model is used for carrying out convolution processing calculation on the downsampling image at least once, the relation between the sampling times and the convolution processing times of the scale calculation model is positive correlation, all the scale calculation images are fused to obtain a fused image and then are subjected to convolution processing, a target image is output through upsampling operation, the target image has higher image quality compared with the original low-light image, the enhancement of the low-light image is realized, the work efficiency is improved without the manual enhancement of a worker using picture editing software, and the problem of low-light image is low due to the manual enhancement of the worker is solved.

Drawings

FIG. 1 is a flow chart of steps of an image enhancement method according to an embodiment of the present application;

FIG. 2 is a flowchart illustrating specific steps of an image enhancement method according to an embodiment of the present application;

Fig. 3 is a schematic diagram of an image enhancement process corresponding to an image enhancement method according to an embodiment of the present application;

fig. 4 is a block diagram of an image enhancement apparatus according to an embodiment of the present application;

fig. 5 is a schematic hardware structure of an electronic device according to an embodiment of the present application.

Detailed Description

The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are some, but not all embodiments of the application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.

In the embodiment of the present application, the original low-light image is a low-light image, for example, an image under short exposure and an image under very low ambient light under long exposure, and the target image is an image obtained by enhancing the original low-light image, that is, an image obtained by processing the problems of low brightness, low contrast, noise, artifact and the like existing in the original low-light image with insufficient illumination, and improving the visual quality of the original low-light image.

The image enhancement method provided by the embodiment of the application is described in detail below through specific embodiments and application scenes thereof with reference to the accompanying drawings.

Fig. 1 is a flowchart of steps of a method for enhancing an image according to an embodiment of the present application, as shown in fig. 1, the method may include:

and step 101, amplifying the pixel intensity of the acquired original low-light image according to the pre-amplification coefficient to obtain a pre-amplified image.

It should be noted that, the pixel intensity refers to the brightness of the pixels of the image, the brightness value of the pixels is between 0 and 255, the brightness of the pixels close to 255 is higher, the brightness close to 0 is lower, and the rest part belongs to the middle tone. This distinction in luminance is an absolute distinction, i.e., pixels near 255 are high light, pixels near 0 are dark, and intermediate is near 128.

Specifically, in some embodiments, the original low-light image is obtained by a manner of inputting by a worker, for example, the original low-light image is obtained by a manner of uploading a picture by the worker, and the original low-light image is obtained by a manner of taking a picture by the worker.

Optionally, in some embodiments, the pre-amplification factor is used as a magnification factor of the pixel intensity of the original low-light image, so that the pixel intensity of each pixel of the original low-light image is obtained by multiplying the pixel intensity of each pixel of the pre-amplified image by the pre-amplification factor, and the pixel intensity of each pixel of the pre-amplified image is greater than the pixel intensity of the corresponding pixel of the original low-light image. For example, if the pixel intensity of the (a, b) th pixel of the original low-light image is 10 and the pre-magnification coefficient is 20, the pixel intensity of the (a, b) th pixel corresponding to the pre-magnified image is 200 (obtained by 10×20). And calculating a pre-amplification coefficient according to the weighted average value of the pixel intensities of all the pixel points of the original low-light image and a preset amplification parameter.

After the pre-amplification factor is obtained, the pixel intensity of each pixel point of the obtained original low-light image is multiplied by the pre-amplification factor to obtain the pixel intensity of each pixel point of the pre-amplification image, namely, the pixel intensity of the obtained original low-light image is subjected to amplification processing to obtain the pre-amplification image, so that the original low-light image is subjected to preliminary image enhancement processing, the pixel intensity of the pre-amplification image obtained through the step is improved compared with the pixel intensity of the original low-light image, and the overall brightness of the pre-amplification image is improved compared with the original low-light image in visual sense of human eyes, so that the visual quality of the image is improved.

And 102, respectively performing a plurality of downsampling operations on the pre-amplified image to respectively obtain downsampled images corresponding to each downsampling operation.

And the sampling multiples corresponding to the downsampling operations are sequentially decreased.

Specifically, in some embodiments, the downsampling operation is an operation of reducing the resolution of the pre-amplified image, that is, the resolution of the pre-amplified image is divided by the sampling multiple of downsampling to obtain the resolution of the downsampled image, and the larger the sampling multiple of downsampling is, the smaller the resolution of the downsampled image obtained by the downsampling operation is, for example, the downsampling multiple of downsampling is 2, that is, the resolution of the pre-amplified image is divided by 2, so that the image with the 1/2 of the resolution of the pre-amplified image is the downsampled image. For example, the resolution of the pre-magnified image is 640 x 320, and the downsampled image with the resolution of 320 x 160 is obtained by performing a downsampling operation by a factor of 2, i.e., dividing 640 by 2 (resulting in 320) and dividing 320 by 2 (resulting in 160).

Alternatively, in some embodiments, the number of downsampling operations may be 2, 3,4, 5, 6, etc., e.g., the resolution of the pre-amp image is 640×320, the number of downsampling operations is 4, respectively, the downsampling operation of 4 samples the pre-amp image to obtain a downsampled image a with a resolution of 160×80, the downsampling operation of 8 samples the pre-amp image to obtain a downsampled image B with a resolution of 80×40, the downsampling operation of 16 samples the pre-amp image to obtain a downsampled image C with a resolution of 40×20, and the downsampling operation of 32 samples the pre-amp image to obtain a downsampled image D with a resolution of 20×10.

And respectively obtaining downsampled images corresponding to each downsampling operation by respectively carrying out multiple downsampling operations on the pre-amplified images, wherein the downsampling operation is the first step of further enhancing the original low-light images, and the obtained multiple downsampled images are respectively input into corresponding scale calculation models for further processing.

And 103, respectively inputting each downsampled image into a corresponding scale calculation model, and outputting a scale calculation image corresponding to the downsampled image.

The scale calculation model is used for carrying out convolution processing calculation on the downsampled image at least once, and the relation between the sampling multiple and the times of the convolution processing calculation of the scale calculation model is positive correlation.

Specifically, in some embodiments, the larger the sampling multiple of the downsampling corresponding to the downsampled image, the more times the convolution process calculation in the scale calculation model corresponding to the downsampled image, and the convolution process calculation may be a single-layer convolution process calculation, a multi-layer convolution process calculation, a hybrid calculation including a one-pass shuffling convolution process calculation and a one-pass grouping convolution process calculation, a residual density block calculation including a plurality of convolution layer calculations for performing nonlinear correction, or the like.

It should be noted that, the calculation of the convolution processing of the single-layer convolution layer and the calculation of the convolution processing of the multi-layer convolution layer are two different convolution manners, and the main difference between them is the number of convolution layers and the parameter amount. The convolution processing calculation of a single convolution layer refers to a convolution neural network model with only one convolution layer. Such a model is suitable for processing simple images or feature extraction. The convolution processing calculation of the single-layer convolution layer only comprises one convolution layer and one activation function, and the input data is subjected to convolution operation to output a characteristic diagram. Because the number of layers is small, the number of parameters is small, and the training speed is relatively high. However, convolution processing calculations of a single convolution layer lack sufficient feature extraction capability and may not capture a complex representation of the image. The convolution processing calculation of the multi-layer convolution layer refers to a model comprising a plurality of convolution layers in the convolution neural network, wherein each convolution layer adds some nonlinear operations, such as activation functions, batch normalization and the like. The convolution processing calculation of the multi-layer convolution layer can improve the capability of feature extraction, so that better performance is obtained. The number of convolution layers is large, and the number of parameters is correspondingly increased, so that the training speed is slower. But the convolution processing calculation of the multi-layer convolution layer can capture more image features, and has better effect on processing complex image tasks.

For example, the resolution of the pre-amplified image is 640×320, the number of downsampling operations is 4, respectively, by performing downsampling operations of 4 times on the pre-amplified image to obtain downsampled image A of 160×80 in resolution, performing downsampling operations of 8 times on the pre-amplified image to obtain downsampled image B of 80×40 in resolution, performing downsampling operations of 16 times on the pre-amplified image to obtain downsampled image C of 40×20 in resolution, performing downsampling operations of 32 times on the pre-amplified image to obtain downsampled image D of 20×10 in resolution,

Each downsampled image is respectively input into a corresponding scale calculation model, a scale calculation image corresponding to the downsampled image is output, namely, a downsampled image A with the resolution of 160 multiplied by 80 is input into the corresponding scale calculation model A to be subjected to one convolution processing calculation, a corresponding scale calculation image A is output, a downsampled image B with the resolution of 80 multiplied by 40 is input into the corresponding scale calculation model B to be subjected to two convolution processing calculations, a corresponding scale calculation image B is output, a downsampled image C with the resolution of 40 multiplied by 20 is input into the corresponding scale calculation model C to be subjected to three convolution processing calculations, a corresponding scale calculation image C is output, a downsampled image D with the resolution of 20 multiplied by 10 is input into the corresponding scale calculation model D to be subjected to four convolution processing calculations, and a corresponding scale calculation image D is output.

By inputting each downsampled image into the corresponding scale calculation model respectively, outputting a scale calculation image corresponding to the downsampled image, which is a second step of further enhancing the original low-light image, the obtained plurality of scale calculation images are subjected to further fusion processing.

And 104, fusing all the scale calculation images to obtain a fused image, performing convolution processing, and outputting a target image through up-sampling operation.

Specifically, in some embodiments, the step-by-step fusion may be performed on all the scale calculation images, for example, after the first to-be-fused image is fused with the second to-be-fused image, a fusion image is obtained, the fusion image is fused with the third to-be-fused image, in the process of fusing all the scale calculation images, the further processing may be performed on the fused images, for example, the scale calculation image A, the scale calculation image B, the scale calculation image C and the scale calculation image D are required to be fused, the target image is output, the scale calculation image C is fused with the downsampled image C corresponding to the first to-be-fused image C1, the downsampled image D corresponding to the scale calculation image D is subjected to one convolution processing calculation to obtain the image D1, the fusion image D2 is obtained by fusing the downsampled image D1 and the scale calculation image D, the further processing may be performed on the fused image D2, for two convolution processing operations may be performed on the fused image, for example, the scale calculation image A, the scale calculation image B and the downsampled image D is required to be fused, the target image C is output, the primary convolution processing may be performed on the downsampled image C1 is obtained by convoluting the downsampled image C1 and the downsampled image E1, and the upsampled image E is obtained by performing the convolution processing operation on the upsampled image E1 and the upsampled image E1.

The target image obtained by fusing all the scale calculation images has higher visual quality compared with the pre-amplified image, and the visual quality of the target image is obviously improved compared with the original low-light image.

In summary, the embodiment of the application can realize that the pixel intensity of an obtained original low-light image is amplified according to the pre-amplification coefficient to obtain the pre-amplified image, the pre-amplified image is respectively subjected to a plurality of downsampling operations to respectively obtain downsampling images corresponding to each downsampling operation, the sampling multiples corresponding to each downsampling operation are sequentially reduced, each downsampling image is respectively input into a corresponding scale calculation model to output a scale calculation image corresponding to the downsampling image, the scale calculation model is used for carrying out convolution processing calculation on the downsampling image at least once, the relation between the sampling multiples and the convolution processing times of the scale calculation model is positive correlation, all the scale calculation images are fused to obtain a fused image, and then a target image is output through the upsampling operation.

Fig. 2 is a flowchart of specific steps of a method for enhancing an image according to an embodiment of the present application, as shown in fig. 2, the method may include:

And step 201, amplifying the pixel intensity of the acquired original low-light image according to the pre-amplification coefficient to obtain a pre-amplified image.

The implementation of this step is similar to the implementation of the "step 101" described above, and will not be repeated here.

Optionally, in some embodiments, before performing "step 201", the method further comprises:

Step 200, obtaining a pre-amplification coefficient according to the pixel intensity of the original low-light image.

Specifically, in some embodiments, according to the pixel intensities of all the pixels of the original low-light image, the pre-amplification coefficient is calculated and obtained by designing a calculation method of the pre-amplification coefficient, which is used as a basis for performing the preliminary processing of image enhancement on the original low-light image.

Optionally, in some embodiments, "step 200" includes the following steps (step 2001, step 2002, step 2003, step 2004):

step 2001, obtaining the pixel intensity of each pixel point of the original low-light image.

Specifically, in some embodiments, the pixel intensity of each pixel of the original low-light image is read by parsing the original low-light image, for example, the pixel intensity of the (c, d) th pixel of the original low-light image is read to be 50, i.e., the brightness of the pixel of the (c, d) th pixel of the original low-light image is read to be 50.

By parsing the original low-light image, obtaining feature information for each pixel of the original low-light image, including the pixel intensity of each pixel, data may be provided for further processing of the original low-light image such that the original low-light image is enhanced.

Step 2002, obtaining the weight of each pixel point of the original low-light image according to the pixel intensity of each pixel point of the original low-light image.

Specifically, in some embodiments, according to the pixel intensity of each pixel of the original low-light image, the weight of each pixel of the original low-light image is obtained through calculation by designing a weight calculation method, which is used as the basis for further processing the original low-light image, for example, the weight of the (i, j) th pixel of the original low-light image is obtained through calculation according to the following mathematical expression:

Wherein ω _i,j is the weight of the (i, j) th pixel of the original low-light image, k e [1, n ], b _k is the kth container edge, b _k-1 is the kth-1 container edge, x _i,j is the pixel intensity of the (i, j) th pixel of the original low-light image.

Optionally, in some embodiments, the kth container edge is calculated according to the following mathematical expression:

Wherein b _k is the kth container edge, b ₀ is the 0 th container edge, k ε [1, n ].

For example, the pixel intensity range of a pixel point [0,1] is quantified as n bins, k ranges are [1, n ], assuming n is 100, n bins may be represented as [0,0.01], [0.01,0.02], [0.02,0.03], [0.97,0.98], [0.98,0.99], [0.99,1 ].

Assuming that k=1, the pixel intensity range of the pixel point is [0,0.01], substituted into the mathematical expression

The method comprises the following steps:

I.e. the weight omega _i,j of the (i, j) th pixel point of the original low-light image is 1;

Assuming that k=100, the pixel intensity range of the pixel point is [0.99,1], substituted into the mathematical expression

The method comprises the following steps:

I.e., the (i, j) th pixel point of the original low-light image has a weight ω _i,j of about 0;

The design can lead the pixel intensity of the original low-light image to occupy less weight, so that the pixel intensity of the pixel with smaller pixel intensity affects the pixel intensity in the preliminary processing of the original low-light image more than the pixel intensity, namely, the pixel intensity of the original low-light image is amplified according to a pre-amplification coefficient obtained based on the weight in the subsequent steps, the pixel intensity of the original low-light image is more focused on the effect of taking the amplification of the pixel intensity of a dark region of the original low-light image into consideration, the obtained pre-amplification image has higher visual quality, and the possibility of excessively taking the effect of excessively taking the pixel intensity of the original low-light image into consideration to cause the bright region of the pre-light region of the original low-light image to appear is reduced.

Step 2003, obtaining a weighted average value according to the weights of all the pixels of the original low-light image and the pixel intensities of all the pixels.

Specifically, in some embodiments, according to the weights of all the pixels of the original low-light image and the pixel intensities of all the pixels, a weighted average of the pixel intensities of all the pixels of the original low-light image may be obtained, and the weighted average may be obtained according to the following mathematical expression by calculating as a basis for further processing the original low-light image:

Where P is the weighted average, ω _i,j is the weight of the (i, j) th pixel of the original low-light image, and x _i,j is the pixel intensity of the (i, j) th pixel of the original low-light image.

And 2004, obtaining the pre-amplification coefficient according to the weighted average value and a preset amplification parameter.

Specifically, in some embodiments, according to the weighted average value and the preset amplification parameter, the pre-amplification coefficient is calculated and obtained, as a basis for further processing of the original low-light image, according to the following mathematical expression, the pre-amplification coefficient is calculated and obtained:

wherein T is a pre-amplification factor, m is a preset amplification parameter, ω _i,j is a weight of the (i, j) th pixel of the original low-light image, and x _i,j is a pixel intensity of the (i, j) th pixel of the original low-light image.

It should be noted that the preset amplification parameter is a parameter preset by a worker, for example, the preset amplification parameter may be 0.5.

The method and the device can be used for obtaining the pixel intensity of each pixel point of the original low-light image by executing the steps 2001, 2002, 2003 and 2004, obtaining the weight of each pixel point of the original low-light image according to the pixel intensity of each pixel point of the original low-light image, obtaining a weighted average value according to the weights of all pixel points of the original low-light image and the pixel intensities of all pixel points, and obtaining a pre-amplification coefficient according to the weighted average value and a preset amplification parameter, wherein the pre-amplification coefficient is used as the basis for the primary enhancement processing of the original low-light image.

Optionally, in some embodiments, the relationship between the weight of the pixel point and the pixel intensity of the pixel point is a negative correlation, and the relationship between the pre-amplification factor and the weighted average value is a negative correlation.

Specifically, in some embodiments, the relationship between the weight of the pixel point and the pixel intensity of the pixel point is a negative correlation as shown in the mathematical expression of the weight, and the relationship between the pre-amplification factor and the weighted average is a negative correlation as shown in the mathematical expression of the pre-amplification factor.

The relation between the weight of the pixel point and the pixel intensity of the pixel point and the relation between the pre-amplification coefficient and the weighted average value are taken as the basis for the primary enhancement processing of the original low-light image.

And 202, respectively performing a plurality of downsampling operations on the pre-amplified image to respectively obtain downsampled images corresponding to each downsampling operation.

The implementation of this step is similar to the implementation of "step 102" described above, and will not be repeated here.

Optionally, in some embodiments, the relationship of the sampling times and the resolution of the corresponding downsampled image is a negative correlation.

Specific examples are as described above, and are not repeated here.

Optionally, in some embodiments, "step 202" specifically includes the following steps (step 2021, step 2022, step 2023):

step 2021, performing a first downsampling operation on the pre-amplified image to obtain a first downsampled image.

Specifically, in some embodiments, the sampling multiple of the first downsampling may be 2, that is, the resolution of the pre-amplified image divided by 2, to obtain an image with 1/2 of the resolution of the pre-amplified image as the first downsampled image. For example, the resolution of the pre-amplified image is 640 x 320, and the first downsampled image with the resolution 320 x 160 is obtained by performing a first downsampling operation by a factor of 2, i.e., dividing 640 by 2 (resulting in 320) and dividing 320 by 2 (resulting in 160).

Step 2022, performing a second downsampling operation on the pre-amplified image to obtain a second downsampled image.

Specifically, in some embodiments, the second downsampled may have a sampling multiple of 8, that is, the resolution of the pre-amplified image divided by 8, resulting in an image with 1/8 of the resolution of the pre-amplified image as the second downsampled image. For example, the resolution of the pre-amplified image is 640 x 320, and a second downsampling operation by a factor of 8, i.e., 640 divided by 8 (resulting in 80) and 320 divided by 8 (resulting in 40), results in a second downsampled image having a resolution of 80 x 40.

Step 2023, performing a third downsampling operation on the pre-amplified image to obtain a third downsampled image.

Specifically, in some embodiments, the third downsampled may have a sampling multiple of 32, i.e., the resolution of the pre-magnified image is divided by 32, resulting in an image with a 1/32 pre-magnified image resolution being the third downsampled image. For example, the resolution of the pre-amplified image is 640 x 320, and a third downsampling operation by a factor of 32, that is, 640 divided by 32 (resulting in 20) and 320 divided by 32 (resulting in 10), results in a third downsampled image having a resolution of 20 x 10.

Optionally, in some embodiments, the first downsampling operation is less than the second downsampling operation, and the second downsampling operation is less than the third downsampling operation. For example, the first downsampling operation has a sampling multiple of 2, the second downsampling operation has a sampling multiple of 8, and the third downsampling operation has a sampling multiple of 32.

By executing step 2021, step 2022 and step 2023 in the embodiment of the present application, downsampling is performed on the pre-amplified image by different sampling multiples, so that downsampled images with different resolutions can be obtained, which is favorable for further refined image enhancement processing, and the target image obtained after final fusion has higher visual quality.

Optionally, in some embodiments, the scale calculation model corresponding to the first downsampled image includes a convolution processing calculation, and outputs a first scale calculation image, the scale calculation model corresponding to the second downsampled image includes two convolution processing calculations, and outputs a high-quality second scale calculation image, and the scale calculation model corresponding to the third downsampled image includes a mixing calculation and a residual error density block calculation, and outputs a third scale calculation image with the highest quality of the three.

According to the embodiment of the application, further refined enhancement processing can be carried out on each downsampled image, namely corresponding refined convolution processing calculation is carried out on different downsampled images, so that the final target image has higher visual quality.

It should be noted that, the residual dense block Residual Dense Block (RDB) can make full use of all layered features of the image to be processed.

Optionally, in some embodiments, the blending calculation is performed prior to the residual density block calculation.

And the third downsampled image is preprocessed by performing mixed calculation before the residual error dense block calculation, so that the refinement processing of the subsequent residual error dense block calculation is facilitated.

Step 203, inputting each downsampled image into a corresponding scale calculation model respectively, and outputting a scale calculation image corresponding to the downsampled image.

The implementation of this step is similar to the implementation of the above-mentioned "step 103", and will not be repeated here.

Optionally, in some embodiments, "step 203" specifically includes the following steps (step 2031, step 2032, step 2033):

Step 2031, inputting the first downsampled image into a corresponding scale calculation model to perform convolution processing calculation of a single-layer convolution layer once, and outputting a first scale calculation image.

The convolution processing calculation of the single-layer convolution layer can improve the visual quality of the final output target image.

And outputting the first scale calculation image to facilitate the subsequent further refined image enhancement processing.

Step 2032, inputting the second downsampled image into a corresponding scale calculation model to perform convolution processing calculation of the multilayer convolution layer twice, and outputting a second scale calculation image.

The convolution processing calculation realizes operations such as sharpening filtering and Gaussian filtering on the image by carrying out convolution operation on the image and the convolution kernel, thereby improving the visual quality of the image. The sharpening filtering is to enhance the edge and detail of the image through the sharpening coefficient in the convolution kernel, so that the sharpening processing of the image is realized, and the Gaussian filtering is to smooth the image through the Gaussian function in the convolution kernel, so that the denoising processing of the image is realized, and therefore, the vision quality of the finally output target image can be improved through the convolution processing calculation of the multi-layer convolution layer. And outputting the second scale calculation image, so that the subsequent further refined image enhancement processing is facilitated.

Step 2033, inputting the third downsampled image into a corresponding scale calculation model to perform primary mixing calculation and primary residual error density block calculation, and outputting a third scale calculation image.

The mixed calculation comprises a primary channel shuffling convolution processing calculation and a primary grouping convolution processing calculation, and the residual error density block calculation comprises a plurality of convolution layer calculations for carrying out nonlinear correction.

It should be noted that, since the hybrid computation includes one channel shuffling convolution processing computation and one grouping convolution processing computation, which can play a role in enhancing edges and details of an image and denoising the image, the residual dense block computation includes a plurality of convolution layer computations for performing nonlinear correction, that is, convolution processing computation of a plurality of convolution layers, which can capture more image features, and thus, the hybrid computation and the residual dense block computation can improve visual quality of a final output target image.

And outputting the third scale calculation image, so that the subsequent further refined image enhancement processing is facilitated.

The execution of step 2031, step 2032 and step 2033 in the embodiment of the application can realize that the first scale calculation image, the second scale calculation image and the third scale calculation image with higher visual quality of the output image are favorable for further refined image enhancement processing of the first scale calculation image, the second scale calculation image and the third scale calculation image.

And 204, fusing all the scale calculation images to obtain a fused image, performing convolution processing, and outputting a target image through up-sampling operation.

The implementation of this step is similar to the implementation of "step 104" described above, and will not be repeated here.

Optionally, in some embodiments, "step 204" specifically includes the following steps (step 2041, step 2042, step 2043):

Step 2041, fusing the third scale calculation image and the second scale calculation image to obtain a first fused image.

It should be noted that, fusing the third scale calculation image and the second scale calculation image may improve the visual quality of the finally output target image.

The fusion refers to combining two image features with the same resolution, the resolution of the image is not changed in the fusion process, but the number of channels is increased, the more channels are, the more image features can be extracted, the higher the visual quality of the finally output target image is, for example, the image feature x1= (length pixel number, width pixel number, channel 1), the image feature x2= (length pixel number, width pixel number, channel 2), and the fused image feature x3= (length pixel number, width pixel number, channel 1+channel 2).

Optionally, in some embodiments, "step 2041" specifically includes the following steps (step 2041a, step 2041b, step 2041c, step 2041 d):

and 2041a, fusing the third scale calculation image and the image obtained by the mixed calculation of the third downsampling image to obtain a third fused image.

Specifically, in some embodiments, first, a mixing calculation is performed on the third downsampled image, where the mixing calculation includes a channel shuffling convolution processing calculation and a grouping convolution processing calculation, the image obtained from the mixing calculation is fused with a third scale calculation image, and the image obtained by mixing the third scale calculation image and the third downsampled image is fused, so that the number of channels of image features is increased, and further, the information amount of image feature extraction is improved, so that the visual quality of a target image which is finally output is improved.

Since the same convolution computation requires a longer time for the convolution computation of the high-resolution image than for the convolution computation of the low-resolution image and the third downsampled image has a lower resolution than the first downsampled image and the second downsampled image, the hybrid computation, i.e., the one-pass shuffling convolution computation and the one-pass grouping convolution computation of the third downsampled image, is employed to allocate as many convolution computations as possible to the third downsampled image having a lower resolution in view of the time cost.

And 2041b, fusing the second scale calculation image with the second downsampled image to obtain a fourth fused image.

The second scale calculation image and the second downsampling image are fused, so that the number of channels of image features is increased, the information quantity of image feature extraction is further improved, and the visual quality of a finally output target image can be improved.

And 2041c, performing up-sampling operation on the third fused image to obtain a first up-sampled image.

Specifically, in some embodiments, the upsampling of the third fused image by the upsampling operation may be performed by a sampling multiple of 4, that is, by multiplying the resolution of the third fused image by 4, to obtain an image of 4 times the resolution of the third fused image as the first upsampled image. For example, the resolution of the third fused image is 20×10, and the up-sampling operation with a multiple of 4, i.e., 20 times 4 (resulting in 80) and 10 times 4 (resulting in 40), results in a first up-sampled image with a resolution of 80×40.

The step 2041c may be implemented to perform an upsampling operation on the third fused image, thereby obtaining a first upsampled image, so that the first upsampled image and the fourth fused image have the same resolution, and further fusion processing is performed on the first upsampled image and the fourth fused image.

And 2041d, fusing an image obtained by performing convolution processing calculation on the first up-sampling image through a single-layer convolution layer with an image obtained by performing convolution processing calculation on the fourth fused image through a single-layer convolution layer to obtain the first fused image.

Specifically, in some embodiments, the first upsampled image is first subjected to a convolution processing calculation of a single convolution layer to obtain an image 1, the fourth fused image is subjected to a convolution processing calculation of a single convolution layer to obtain an image 2, and then the image 1 and the image 2 are fused to obtain a first fused image, so that the visual quality of the finally output target image can be improved by fusing the image 1 and the image 2.

According to the embodiment of the application, the steps 2041a, 2041b, 2041c and 2041d are executed, and further refined image enhancement processing is carried out on the third scale calculation image and the second scale calculation image, so that the obtained first fusion image has higher visual quality of the image, and further image enhancement processing is facilitated.

And 2042, fusing the first fused image and the first scale calculation image to obtain a second fused image.

It should be noted that, fusing the first fused image and the first scale calculation image may improve the visual quality of the finally output target image.

Optionally, in some embodiments, "step 2042" specifically includes the following steps (step 2042a, step 2042 b):

step 2042a, performing up-sampling operation on the image obtained by performing one-time mixed calculation and one-time convolution processing calculation on the multi-layer convolution layer on the first fused image, so as to obtain a second up-sampling image.

Optionally, in some embodiments, the sampling multiple of the upsampling performed on the image obtained by performing the first blending calculation and the convolution processing calculation of the first multilayer convolution layer may be 4, that is, the resolution of the image obtained by performing the first blending calculation and the convolution processing calculation of the first multilayer convolution layer on the first blending image is multiplied by 4, so that the image with the image resolution obtained by performing the first blending calculation and the convolution processing calculation of the first multilayer convolution layer on the first blending image is the second upsampled image. For example, the resolution of the image obtained by performing a mixing calculation and a convolution processing calculation of a multi-layer convolution layer on the first fused image is 80×40, and the second up-sampled image with the resolution of 320×160 is obtained by performing up-sampling operation with a multiple of 4, that is, multiplying 80 by 4 (to obtain 320) and multiplying 40 by 4 (to obtain 160).

Specifically, in some embodiments, the first fused image is subjected to a mixing calculation and a convolution processing calculation of a plurality of convolution layers, thereby obtaining an image 3, and the image 3 is subjected to an up-sampling operation, thereby obtaining a second up-sampled image.

Compared with the convolution processing calculation of the single-layer convolution layer, the convolution processing calculation of the multi-layer convolution layer can capture more image features, and the visual quality of a target image finally output is improved.

The step 2042a may be implemented to obtain a second upsampled image, where the second upsampled image has the same resolution as the first scale calculated image, which is advantageous for further fusion processing of the second upsampled image and the first scale calculated image.

Step 2042b, fusing the second upsampled image with the first scale calculation image to obtain the second fused image.

It should be noted that, fusing the second upsampled image and the first scale calculation image may improve the visual quality of the finally output target image.

By executing the steps 2042a and 2042b in the embodiment of the present application, further refined image enhancement processing is performed on the first fused image and the first scale calculation image, so that a second fused image is obtained, which is beneficial to further image enhancement processing.

In the same convolution calculation, since the time required for the convolution calculation of the high-resolution image is longer than the time required for the convolution calculation of the low-resolution image, and the time required for the convolution calculation of the single-layer convolution layer is longer than the time required for the convolution calculation of the single-layer convolution layer of the same image, in the embodiment of the present application, as much as possible of the convolution calculation is allocated to the low-resolution image, so that the time required for the whole process, for example, the first fusion image is lower than the resolution of the second fusion image, the first fusion image is allocated to the first fusion image, the first fusion image is processed, and the second fusion image is processed by allocating the convolution calculation of the single-layer convolution layer to the second fusion image.

And 2043, performing up-sampling operation on the image obtained by performing convolution processing calculation on the second fusion image through a single-layer convolution layer to obtain a target image.

Optionally, in some embodiments, the sampling multiple of up-sampling performed on the image obtained by performing the convolution processing on the single-layer convolution layer on the second fused image may be 2, that is, the resolution of the image obtained by performing the convolution processing on the single-layer convolution layer on the second fused image is multiplied by 2, so that the image with the image resolution obtained by performing the convolution processing on the single-layer convolution layer on the second fused image is the target image. For example, the resolution of the image obtained by performing convolution processing calculation on the single-layer convolution layer on the second fused image is 320×160, and the target image with the resolution of 640×320 is obtained by performing up-sampling operation with a multiple of 2, namely, multiplying 320 by 2 (obtaining 640) and multiplying 160 by 2 (obtaining 320).

Specifically, in some embodiments, the second fused image is first subjected to convolution processing calculation of a single convolution layer to obtain an image 4, and then the image 4 is subjected to up-sampling operation, so as to obtain a target image.

It is achieved by step 2043 that a target image is obtained, the target image having the same resolution as the original low-light image and the target image having a higher visual quality of the image than the original low-light image.

According to the embodiment of the application, step 2041, step 2042 and step 2043 are executed, further image enhancement refinement processing is carried out on all the scale calculation images, the target image obtained by fusing all the scale calculation images has the same resolution as the original low-light image, and the target image has higher visual quality than the original low-light image, so that the enhancement of the low-light image is realized.

Optionally, in some embodiments, the up-sampling multiple of the up-sampling operation performed on the image obtained by performing the convolution processing of the single-layer convolution layer on the second fused image is equal to the up-sampling multiple of the first downsampling operation performed on the image obtained by performing the convolution processing of the single-layer convolution layer on the second fused image, the product of the up-sampling multiple of the up-sampling operation performed on the image obtained by performing the convolution processing of the single-layer convolution layer on the first fused image and the up-sampling multiple of the up-sampling operation performed on the image obtained by performing the convolution processing of the single-layer convolution layer on the first fused image is equal to the up-sampling multiple of the second downsampling operation performed on the image obtained by performing the convolution processing of the single-layer convolution layer on the first fused image, and the up-sampling multiple of the up-sampling operation performed on the third fused image is equal to the up-sampling multiple of the third downsampling operation performed on the image obtained by performing the convolution processing of the first fused image.

Optionally, in some embodiments, before performing a residual block calculation on an image obtained after performing a mixing calculation on the third downsampled image, a default number of times of calculation of a convolution layer in the residual block calculation is adjusted to a target number of times, where the target number of times is smaller than the default number of times.

In the residual dense block calculation, too much use of the convolution layer calculation with nonlinear correction may reduce the accuracy of the image enhancement processing, but too little use of the convolution layer calculation with nonlinear correction may reduce the visual quality of the image. In the embodiment of the application, compared with the convolution layer calculation for carrying out nonlinear correction of the default times, the convolution layer calculation for carrying out nonlinear correction of the target times reduces the side effect of negative clipping, simultaneously maintains enough nonlinear correction, and is beneficial to improving the visual quality of images.

Optionally, in some embodiments, computing the first downsampling operation on the pre-amplified image and inputting the first downsampled image into the corresponding scale calculation model for convolution processing computes a total number of operands according to the mathematical expression:

Wherein M ₁ is the total operand of the first downsampling operation on the pre-amplified image and the convolution processing calculation of the first downsampled image input to the corresponding scale calculation model, the original low-light image has a resolution of H×W (the number of length pixels is H, the number of width pixels is W), r ₁ is the sampling multiple of downsampling, and the convolution kernel has a size of (I.e., k ₁×k₁),I₁ is the number of channels in, O ₁ is the number of channels out, and N ₁ is the number of convolutional layers.

Optionally, in some embodiments, computing the second downsampling operation on the pre-amplified image and inputting the second downsampled image into the corresponding scale-calculation model for convolution processing computes a total number of operands according to the mathematical expression:

Wherein M ₂ is an operand for performing a second downsampling operation on the pre-amplified image and inputting the second downsampled image into a corresponding scale calculation model for convolution processing calculation, the resolution of the original low-light image is H×W (the number of length pixels is H and the number of width pixels is W), r ₂ is a sampling multiple of downsampling, and the size of the convolution kernel is (I.e., k ₂×k₂),I₂ is the number of channels in, O ₂ is the number of channels out, and N ₂ is the number of convolutional layers.

Optionally, in some embodiments, the third downsampling operation on the pre-amplified image and the inputting of the third downsampled image into the corresponding scale calculation model are calculated according to the following mathematical expression to convolve the total operand:

Wherein M ₃ is an operand for performing a third downsampling operation on the pre-amplified image and inputting the third downsampled image into a corresponding scale calculation model for convolution processing calculation, the resolution of the original low-light image is H×W (the number of length pixels is H and the number of width pixels is W), r ₃ is a sampling multiple of downsampling, and the size of the convolution kernel is (I.e., k ₃×k₃),I₃ is the channel width of the packet convolution, N ₃ is the number of convolution layers, S ₃ is the growth factor calculated for each convolution process in the residual block, [1, η ], η is the number of convolution layers calculated for each convolution process in the residual block.

Specifically, in some embodiments, for the mathematical expression:

Setting r ₁ to 2, the size of the convolution kernel (I.e., k ₁×k₁) 7 ²,I₁ 4,O ₁ and n ₁ 1, there are:

For mathematical expressions:

Setting r ₂ to 8, the size of the convolution kernel (I.e., k ₂×k₂) is 3 ²,I₂ is 64, O ₂ is 64, and N ₂ is 1, then there are:

For mathematical expressions:

setting r ₃ to 32, the size of the convolution kernel (I.e., k ₃×k₃) is 3, I ₃ is 64, N ₃ is 3, S ₃ is 32, l ε [1, η ] η is5, then there are:

In the embodiment of the application, the total operand of convolution processing calculation is performed by performing a first downsampling operation on the pre-amplified image and inputting the first downsampled image into a corresponding scale calculation model, the total operand of convolution processing calculation is performed by performing a second downsampling operation on the pre-amplified image and inputting the second downsampled image into a corresponding scale calculation model, the total operand of convolution processing calculation is within the same operand range (570×h×w to 600×h×w) by performing a third downsampling operation on the pre-amplified image and inputting the third downsampled image into a corresponding scale calculation model, and parallel calculation can be performed.

Optionally, in some embodiments, referring to FIG. 3, an image enhancement process corresponding to an image enhancement method may include inputting an original low-light image by X1; the method comprises the steps of X2, 2 times of downsampling, X3, 8 times of downsampling, X4, 32 times of downsampling, X5, single-layer convolution layer calculation, X6, multi-layer convolution layer calculation, X7, multi-layer convolution layer calculation, X8, fusion, X9, single-layer convolution layer calculation, X10, hybrid calculation, X11, residual error density block calculation, X12, fusion, X13, 4 times of upsampling, X14, single-layer convolution layer calculation, X15, fusion, X16, hybrid calculation, X17, multi-layer convolution layer calculation, X18, 4 times of upsampling, X19, fusion, X20, single-layer convolution layer calculation, X21, 2 times of upsampling, X22 and output target images, wherein the hybrid calculation comprises one-time channel shuffling convolution processing calculation and one-time grouping convolution processing calculation, the residual error density block calculation comprises a plurality of convolution layer calculations used for carrying out nonlinear correction, compared with an original low-light image, the target image can have higher image quality, enhancement of the low-light image is realized, staff use of picture enhancement software to manually edit the low-light image, low-efficiency is improved, the staff use of image enhancement software is used for manually editing the low-light image, the staff efficiency is low, and the staff efficiency is low due to work efficiency is improved.

Fig. 4 is a block diagram of an image enhancement apparatus according to an embodiment of the present application, and as shown in fig. 4, the apparatus 300 includes:

the pre-amplification module 301 is configured to amplify the pixel intensity of the obtained original low-light image according to a pre-amplification coefficient to obtain a pre-amplified image;

The downsampling module 302 is configured to perform downsampling operations on the pre-amplified image for multiple times, and obtain downsampled images corresponding to each downsampling operation respectively;

the scale calculation module 303 is configured to input each downsampled image into a corresponding scale calculation model, and output a scale calculation image corresponding to the downsampled image, where the scale calculation model is configured to perform convolution processing calculation on the downsampled image at least once, and a relationship between the sampling multiple and the number of times of the convolution processing calculation of the scale calculation model is positive correlation;

and the fusion module 304 is configured to fuse all the scale calculation images, obtain a fused image, perform convolution processing, and output a target image through up-sampling operation.

Optionally, the downsampling module 302 specifically includes:

the first downsampling submodule is used for carrying out first downsampling operation on the pre-amplified image to obtain a first downsampled image;

the second downsampling submodule is used for carrying out second downsampling operation on the pre-amplified image to obtain a second downsampled image;

a third downsampling sub-module, configured to perform a third downsampling operation on the pre-amplified image to obtain a third downsampled image;

the sampling multiple of the first downsampling operation is smaller than that of the second downsampling operation, and the sampling multiple of the second downsampling operation is smaller than that of the third downsampling operation.

Optionally, the scale calculation model corresponding to the first downsampled image includes one convolution processing calculation to output a first scale calculation image, the scale calculation model corresponding to the second downsampled image includes two convolution processing calculations to output a high-quality second scale calculation image, and the scale calculation model corresponding to the third downsampled image includes one mixing calculation and one residual error density block calculation to output a third scale calculation image with the highest quality.

Optionally, the scale calculation module 303 specifically includes:

the first scale calculation sub-module is used for inputting the first downsampled image into a corresponding scale calculation model to perform convolution processing calculation of a single-layer convolution layer once and outputting a first scale calculation image;

the second scale calculation sub-module is used for inputting the second downsampled image into the corresponding scale calculation model to perform convolution processing calculation of the multi-layer convolution layer twice and outputting a second scale calculation image;

The third scale calculation sub-module is used for inputting the third downsampled image into a corresponding scale calculation model to perform primary mixed calculation and primary residual error density block calculation, and outputting a third scale calculation image, wherein the mixed calculation comprises primary channel shuffling convolution processing calculation and primary grouping convolution processing calculation, and the residual error density block calculation comprises a plurality of convolution layer calculations for performing nonlinear correction.

Optionally, the fusion module 304 specifically includes:

The first fusion sub-module is used for fusing the third scale calculation image and the second scale calculation image to obtain a first fusion image;

the second fusion sub-module is used for fusing the first fusion image and the first scale calculation image to obtain a second fusion image;

And the third fusion sub-module is used for carrying out up-sampling operation on the image obtained by carrying out convolution processing calculation on the single-layer convolution layer on the second fusion image to obtain a target image.

Optionally, the first fusion sub-module specifically includes:

The first fusion sub-module is used for fusing the image obtained by mixing the third scale calculation image and the third downsampling image to obtain a third fusion image;

The second fusion sub-module is used for fusing the second scale calculation image with the second downsampled image to obtain a fourth fusion image;

The third fusion sub-module is used for carrying out up-sampling operation on the third fusion image to obtain a first up-sampling image;

And the fourth fusion sub-module is used for fusing an image obtained by carrying out convolution processing calculation on the first up-sampling image through a single-layer convolution layer with an image obtained by carrying out convolution processing calculation on the fourth fusion image through a single-layer convolution layer to obtain the first fusion image.

Optionally, the second fusion sub-module specifically includes:

A fifth fusion sub-module, configured to perform an upsampling operation on an image obtained by performing a hybrid computation and a convolution processing computation on a multi-layer convolution layer on the first fused image, to obtain a second upsampled image;

and a sixth fusion sub-module, configured to fuse the second upsampled image with the first scale calculation image to obtain the second fused image.

Optionally, before performing primary residual error density block calculation on an image obtained after primary mixing calculation on the third downsampled image, adjusting a default number of times of convolution layer calculation in the residual error density block calculation to a target number of times, where the target number of times is smaller than the default number of times.

Optionally, the apparatus 300 further comprises:

the acquisition module is used for acquiring the pixel intensity of each pixel point of the original low-light image before amplifying the acquired pixel intensity of the original low-light image according to the pre-amplification coefficient to acquire the pre-amplified image;

the weight calculation module is used for obtaining the weight of each pixel point of the original low-light image according to the pixel intensity of each pixel point of the original low-light image;

The weighted average calculation module is used for obtaining a weighted average according to the weights of all the pixel points of the original low-light image and the pixel intensities of all the pixel points;

and the pre-amplification factor calculation module is used for obtaining the pre-amplification factor according to the weighted average value and a preset amplification parameter.

Optionally, the relation between the weight of the pixel point and the pixel intensity of the pixel point is negative correlation, and the relation between the pre-amplification coefficient and the weighted average value is negative correlation.

The image enhancement device in the embodiment of the application can be a device, a component in a terminal, an integrated circuit or a chip. The device may be a mobile electronic device or a non-mobile electronic device. By way of example, the mobile electronic device may be a mobile phone, a tablet computer, a notebook computer, a palm computer, a vehicle-mounted electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a Personal Digital Assistant (PDA), etc., and the non-mobile electronic device may be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a Television (TV), a teller machine, a self-service machine, etc., and the embodiments of the present application are not limited in particular.

The image enhancement device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an ios operating system, or other possible operating systems, and the embodiment of the present application is not limited specifically.

The image enhancement device provided in the embodiment of the present application can implement each process implemented by the image enhancement device in the method embodiment of fig. 1 and fig. 2, and in order to avoid repetition, a description is omitted here.

In summary, in the embodiment of the application, the pixel intensity of the original low-light image is amplified according to the pre-amplification coefficient to obtain the pre-amplified image, the pre-amplified image is respectively subjected to a plurality of downsampling operations to respectively obtain downsampled images corresponding to each downsampling operation, the sampling multiples corresponding to each downsampling operation are sequentially reduced, each downsampled image is respectively input into a corresponding scale calculation model to output a scale calculation image corresponding to the downsampled image, the scale calculation model is used for carrying out convolution processing calculation on the downsampled image at least once, the relation between the sampling multiples and the convolution processing times of the scale calculation model is positive correlation, all the scale calculation images are fused to obtain a fusion image, and then a target image is output through the upsampling operation.

Optionally, the embodiment of the present application further provides an electronic device, including a processor, a memory, and a program or an instruction stored in the memory and capable of running on the processor, where the program or the instruction is executed by the processor to implement each process of the embodiment of the image enhancement method, and the process can achieve the same technical effect, so that repetition is avoided, and details are not repeated here.

The electronic device in the embodiment of the application includes the mobile electronic device and the non-mobile electronic device.

Fig. 5 is a schematic hardware structure of an electronic device implementing an embodiment of the present application.

The electronic device 400 includes, but is not limited to, a radio frequency unit 401, a network module 402, an audio output unit 403, an input unit 404, a sensor 405, a display unit 406, a user input unit 407, an interface unit 408, a memory 409, and a processor 410.

Those skilled in the art will appreciate that the electronic device 400 may also include a power source (e.g., a battery) for powering the various components, which may be logically connected to the processor 410 by a power management system to perform functions such as managing charge, discharge, and power consumption by the power management system. The electronic device structure shown in fig. 5 does not constitute a limitation of the electronic device, and the electronic device may include more or less components than shown, or may combine certain components, or may be arranged in different components, which are not described in detail herein.

The processor 410 is configured to amplify the pixel intensity of the obtained original low-light image according to the pre-amplification coefficient to obtain a pre-amplified image;

Optionally, the processor 410 is further configured to perform downsampling operations on the pre-amplified image for multiple times, respectively to obtain downsampled images corresponding to each downsampling operation, where the downsampling operations include performing a first downsampling operation on the pre-amplified image to obtain a first downsampled image, performing a second downsampling operation on the pre-amplified image to obtain a second downsampled image, performing a third downsampling operation on the pre-amplified image to obtain a third downsampled image, and the sampling multiple of the first downsampling operation is smaller than the sampling multiple of the second downsampling operation, and the sampling multiple of the second downsampling operation is smaller than the sampling multiple of the third downsampling operation.

Optionally, the processor 410 is further configured to output a first scale calculation image by performing one convolution calculation on the scale calculation model corresponding to the first downsampled image, output a high-quality second scale calculation image by performing two convolution calculations on the scale calculation model corresponding to the second downsampled image, and output a third scale calculation image with the highest quality of the three by performing one mixing calculation and one residual error density block calculation on the scale calculation model corresponding to the third downsampled image.

Optionally, the processor 410 is further configured to input each of the downsampled images into a corresponding scale calculation model, output a scale calculation image corresponding to the downsampled image, and the method includes inputting the first downsampled image into the corresponding scale calculation model to perform convolution processing calculation of a single-layer convolution layer once, outputting a first scale calculation image, inputting the second downsampled image into the corresponding scale calculation model to perform convolution processing calculation of multiple-layer convolution layers twice, outputting a second scale calculation image, inputting the third downsampled image into the corresponding scale calculation model to perform one-time mixed calculation and one-time residual error dense block calculation, and outputting a third scale calculation image, where the mixed calculation includes one-time channel shuffling convolution processing calculation and one-time grouping convolution processing calculation, and the residual error dense block calculation includes multiple convolution layer calculations for performing nonlinear correction.

Optionally, the processor 410 is further configured to fuse all the scale calculation images to obtain a fused image, and then perform convolution processing, and output a target image through an up-sampling operation, where the up-sampling operation includes fusing the third scale calculation image and the second scale calculation image to obtain a first fused image, fusing the first fused image and the first scale calculation image to obtain a second fused image, and performing an up-sampling operation on an image obtained by performing convolution processing on the second fused image once with a single-layer convolution layer to obtain the target image.

Optionally, the processor 410 is further configured to fuse the third scale calculation image with the second scale calculation image to obtain a first fused image, where the fusing includes fusing the third scale calculation image with the image obtained by performing the mixed calculation on the third scale calculation image and the third downsampling image to obtain a third fused image, fusing the second scale calculation image with the second downsampling image to obtain a fourth fused image, performing an upsampling operation on the third fused image to obtain a first upsampled image, and fusing the image obtained by performing the convolution processing on the first upsampled image with the image obtained by performing the convolution processing on the single-layer convolution layer once with the image obtained by performing the convolution processing on the fourth fused image with the image obtained by performing the convolution processing on the single-layer once to obtain the first fused image.

Optionally, the processor 410 is further configured to fuse the first fused image with the first scale calculation image to obtain a second fused image, where the fusing includes performing an upsampling operation on an image obtained by performing a hybrid calculation and a convolution processing calculation on a multi-layer convolution layer on the first fused image to obtain a second upsampled image, and fusing the second upsampled image with the first scale calculation image to obtain the second fused image.

Optionally, the processor 410 is further configured to adjust a default number of times of calculation of the convolution layer in the residual error density block calculation to a target number of times before performing the one-time residual error density block calculation on the image obtained after performing the one-time mixing calculation on the third downsampled image, where the target number of times is smaller than the default number of times.

Optionally, the processor 410 is further configured to, before performing an amplification process on the pixel intensities of the obtained original low-light image according to a pre-amplification factor to obtain a pre-amplified image, obtain the pixel intensity of each pixel of the original low-light image, obtain the weight of each pixel of the original low-light image according to the pixel intensity of each pixel of the original low-light image, obtain a weighted average value according to the weights of all pixels of the original low-light image and the pixel intensities of all pixels, and obtain the pre-amplification factor according to the weighted average value and a preset amplification parameter.

Optionally, the processor 410 is further configured to make a relationship between the weight of the pixel point and the pixel intensity of the pixel point be a negative correlation, and make a relationship between the pre-amplification factor and the weighted average be a negative correlation.

It should be appreciated that in embodiments of the present application, the input unit 404 may include a graphics processor (Graphics Processing Unit, GPU) 4041 and a microphone 4042, with the graphics processor 4041 processing image data of still pictures or video obtained by an image capture device (e.g., a camera) in a video capture mode or an image capture mode. The display unit 406 may include a display panel 4061, and the display panel 4061 may be configured in the form of a liquid crystal display, an organic light emitting diode, or the like. The user input unit 407 includes at least one of a touch panel 4071 and other input devices 4072. The touch panel 4071 is also referred to as a touch screen. The touch panel 4071 may include two parts, a touch detection device and a touch controller. Other input devices 4072 may include, but are not limited to, a physical keyboard, function keys (e.g., volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and so forth, which are not described in detail herein.

Memory 409 may be used to store software programs as well as various data. The memory 409 may mainly include a first memory area storing programs or instructions and a second memory area storing data, wherein the first memory area may store an operating system, application programs or instructions (such as a sound playing function, an image playing function, etc.) required for at least one function, and the like. Further, the memory 409 may include volatile memory or nonvolatile memory, or the memory 409 may include both volatile and nonvolatile memory. The nonvolatile Memory may be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically Erasable EPROM (EEPROM), or a flash Memory. The volatile memory may be random access memory (Random Access Memory, RAM), static random access memory (STATIC RAM, SRAM), dynamic random access memory (DYNAMIC RAM, DRAM), synchronous Dynamic Random Access Memory (SDRAM), double data rate Synchronous dynamic random access memory (Double DATA RATE SDRAM, DDRSDRAM), enhanced Synchronous dynamic random access memory (ENHANCED SDRAM, ESDRAM), synchronous link dynamic random access memory (SYNCH LINK DRAM, SLDRAM), and Direct random access memory (DRRAM). Memory 409 in embodiments of the application includes, but is not limited to, these and any other suitable types of memory.

Processor 410 may include one or more processing units and, optionally, processor 410 integrates an application processor that primarily processes operations involving an operating system, user interface, application program, etc., and a modem processor that primarily processes wireless communication signals, such as a baseband processor. It will be appreciated that the modem processor described above may not be integrated into the processor 410.

The embodiment of the application also provides a readable storage medium, on which a program or an instruction is stored, which when executed by a processor, implements each process of the image enhancement method embodiment, and can achieve the same technical effects, and in order to avoid repetition, a detailed description is omitted here.

Wherein the processor is a processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium such as a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a magnetic disk or an optical disk, and the like.

The embodiment of the application further provides a chip, the chip comprises a processor and a communication interface, the communication interface is coupled with the processor, the processor is used for running programs or instructions, the processes of the image enhancement method embodiment can be realized, the same technical effects can be achieved, and the repetition is avoided, and the description is omitted here.

It should be understood that the chips referred to in the embodiments of the present application may also be referred to as system-on-chip chips, chip systems, or system-on-chip chips, etc.

The embodiments of the present application have been described above with reference to the accompanying drawings, but the present application is not limited to the above-described embodiments, which are merely illustrative and not restrictive, and many forms may be made by those having ordinary skill in the art without departing from the spirit of the present application and the scope of the claims, which are to be protected by the present application.

Claims

1. A method of enhancing an image, the method comprising:

Amplifying the pixel intensity of the obtained original low-light image according to a pre-amplification coefficient to obtain a pre-amplification image, wherein the pre-amplification coefficient is obtained according to a weighted average value of the pixel intensities of all pixel points of the original low-light image and preset amplification parameters;

Fusing a third scale calculation image obtained by carrying out primary mixing calculation and primary residual error density block calculation on a third downsampled image with a second scale calculation image obtained by carrying out convolution processing calculation on a second downsampled image twice to obtain a first fused image;

Fusing a first scale calculation image obtained by carrying out convolution processing calculation on the first downsampled image with the first fusion image to obtain a second fusion image;

And carrying out convolution processing on the second fusion image, and outputting a target image through up-sampling operation.

2. The method according to claim 1, wherein the performing downsampling operations on the pre-amplified image for a plurality of times, respectively, to obtain downsampled images corresponding to each downsampling operation, respectively, includes:

performing a first downsampling operation on the pre-amplified image to obtain the first downsampled image;

Performing a second downsampling operation on the pre-amplified image to obtain a second downsampled image;

performing a third downsampling operation on the pre-amplified image to obtain the third downsampled image;

3. The method of claim 1, wherein the hybrid computation comprises a one-pass channel shuffle convolution process computation and a one-pass block convolution process computation, and wherein the residual density block computation comprises a plurality of convolution layer computations for performing non-linear corrections.

4. The method of claim 1, wherein convolving the second fused image to output a target image via an upsampling operation, comprising:

and performing up-sampling operation on an image obtained by performing convolution processing calculation on the second fusion image through a single-layer convolution layer to obtain the target image.

5. The method according to claim 1, wherein the fusing the third scale calculation image obtained by performing a mixing calculation and a residual error density block calculation on the third downsampled image with the second scale calculation image obtained by performing a convolution processing calculation on the second downsampled image twice to obtain the first fused image includes:

fusing the image obtained by mixing the third scale calculation image and the third downsampling image to obtain a third fused image;

Fusing the second scale calculation image and the second downsampled image to obtain a fourth fused image;

Performing up-sampling operation on the third fusion image to obtain a first up-sampling image;

And fusing an image obtained by performing convolution processing calculation on the first up-sampling image through a single-layer convolution layer with an image obtained by performing convolution processing calculation on the fourth fusion image through a single-layer convolution layer to obtain the first fusion image.

6. The method according to claim 1, wherein the fusing the first scale calculation image obtained by performing a convolution processing calculation on the first downsampled image with the first fused image to obtain a second fused image includes:

Performing up-sampling operation on an image obtained by performing one-time mixed calculation and one-time convolution processing calculation on the first fusion image by using a plurality of convolution layers to obtain a second up-sampling image;

and fusing the second up-sampling image with the first scale calculation image to obtain a second fused image.

7. The method of claim 1, wherein a default number of convolution layer calculations in the residual block calculation is adjusted to a target number of times before performing a residual block calculation on an image obtained after performing a mixing calculation on the third downsampled image, wherein the target number of times is less than the default number of times to reduce negative clipping side effects of the non-linearly corrected convolution layer calculation.

8. The method of claim 1, wherein prior to amplifying the pixel intensities of the acquired original low-light image by a pre-amplification factor to obtain a pre-amplified image, the method further comprises:

Acquiring the pixel intensity of each pixel point of the original low-light image;

obtaining the weight of each pixel point of the original low-light image according to the pixel intensity of each pixel point of the original low-light image, wherein the relation between the weight of each pixel point and the pixel intensity of each pixel point is in negative correlation;

obtaining the weighted average value according to the weights of all the pixel points of the original low-light image and the pixel intensities of all the pixel points;

and obtaining the pre-amplification coefficient according to the weighted average value and the preset amplification parameter, wherein the relation between the pre-amplification coefficient and the weighted average value is in negative correlation.

9. An apparatus for enhancing an image, the apparatus comprising:

The pre-amplification module is used for amplifying the pixel intensity of the acquired original low-light image according to a pre-amplification coefficient to obtain a pre-amplified image, wherein the pre-amplification coefficient is obtained according to a weighted average value of the pixel intensities of all pixel points of the original low-light image and preset amplification parameters;

The first fusion sub-module is used for fusing a third scale calculation image obtained by carrying out primary mixing calculation and primary residual error density block calculation on a third downsampled image and a second scale calculation image obtained by carrying out convolution processing calculation on a second downsampled image for two times to obtain a first fusion image;

the second fusion sub-module is used for fusing the first scale calculation image obtained by carrying out convolution processing calculation on the first downsampled image with the first fusion image to obtain a second fusion image;

and the third fusion sub-module is used for carrying out convolution processing on the second fusion image and outputting a target image through up-sampling operation.

10. An electronic device comprising a processor, a memory and a program or instruction stored on the memory and executable on the processor, which when executed by the processor, implements the steps of the image enhancement method according to any one of claims 1 to 8.

11. A readable storage medium, characterized in that the readable storage medium has stored thereon a program or instructions which, when executed by a processor, implement the steps of the image enhancement method according to any of claims 1 to 8.