CN111179282B - Image processing method, image processing device, storage medium and electronic device - Google Patents

Abstract

The present disclosure provides an image processing method, an image processing device, a storage medium and an electronic device, and relates to the field of image processing technology. The image processing method comprises: identifying the sky area in the first image based on a segmentation model, and obtaining a mask image corresponding to the sky area; segmenting a foreground area image from the first image by the mask image, and segmenting a background area image from the second image; splicing the foreground area image and the background area image to obtain a target image. The present disclosure implements the "sky replacement" processing of the image, and can automatically segment the sky area in the image without the user manually cutting out the image, which is easy to use.

Description

Image processing method, image processing device, storage medium and electronic device

Technical Field

The present disclosure relates to the field of image processing technology, and in particular to an image processing method, an image processing device, a computer-readable storage medium, and an electronic device.

Background technique

With the development of image processing technology, the function of changing the image background has appeared in some image processing and image beautification software. For example, the background can be replaced with a solid color curtain, scenery or other special effects to meet the diverse needs of users.

In the related art, when changing the background of an image, the user is generally required to manually cut out the foreground part of the image, and then replace the remaining background part. Some software can automatically identify the portrait in the image and replace the background of the part other than the portrait, but it cannot be applied to non-portraits and will mistake the foreground part other than the portrait as the background. Therefore, if you want to replace the background of a non-portrait image, such as replacing the sky background in a landscape image, the related art requires the user to manually cut out the foreground, which is inconvenient to use.

It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field.

Summary of the invention

The present disclosure provides an image processing method, an image processing device, a computer-readable storage medium and an electronic device, thereby at least to a certain extent solving the problem that the related art requires the user to manually extract the foreground.

Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

According to a first aspect of the present disclosure, there is provided an image processing method, comprising: identifying a sky area in the first image based on a segmentation model to obtain a mask image (Mask) corresponding to the sky area; segmenting a foreground area image from the first image and a background area image from the second image through the mask image; and splicing the foreground area image and the background area image to obtain a target image.

According to a second aspect of the present disclosure, there is provided an image processing device, comprising: a sky recognition module, used to recognize a sky area in the first image, and obtain a mask image corresponding to the sky area; an image segmentation module, used to segment a foreground area image from the first image and a background area image from the second image through the mask image; and an image stitching module, used to stitch the foreground area image and the background area image to obtain a target image.

According to a third aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned image processing method is implemented.

According to a fourth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the above-mentioned image processing method by executing the executable instructions.

The technical solution disclosed in this disclosure has the following beneficial effects:

According to the above-mentioned image processing method, image processing device, storage medium and electronic device, the sky area in the first image is first identified by the segmentation model to obtain a mask image corresponding to the sky area; then the mask image is used to segment the foreground area image from the first image, and the background area image from the second image; finally, the foreground area image and the background area image are spliced to obtain the target image. On the one hand, this solution can accurately segment the sky area from the image based on the recognition of the sky area by the segmentation model and the processing of the mask image, without the need for the user to manually cut out the image, with a high degree of intelligence, easy to use and good user experience. On the other hand, this solution realizes the "sky replacement" processing of the image, replacing the sky area in the first image with the corresponding part in the second image, which is highly interesting and can meet the diverse needs of users.

It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and are used together with the specification to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

FIG1 is a schematic diagram showing a system architecture of this exemplary embodiment;

FIG2 is a schematic diagram showing an electronic device according to the present exemplary embodiment;

FIG3 is a flowchart showing an image processing method according to the present exemplary embodiment;

FIG4 shows a sub-flow chart of an image processing method according to this exemplary embodiment;

FIG5 shows a sub-flow chart of another image processing method according to this exemplary embodiment;

FIG6 shows a schematic flow of image processing according to this exemplary embodiment;

FIG7 is a block diagram showing a structure of an image processing device according to the exemplary embodiment;

FIG. 8 is a schematic diagram showing a computer-readable storage medium according to the present exemplary embodiment.

Detailed ways

Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and/or processor devices and/or microcontroller devices.

FIG1 shows a schematic diagram of a system architecture of an exemplary embodiment of the present disclosure. As shown in FIG1 , the system architecture 100 may include: a terminal 110, a network 120, and a server 130. The terminal 110 may be various electronic devices with an image capture function, including but not limited to mobile phones, tablet computers, digital cameras, personal computers, etc. The network 120 is used to provide a medium for a communication link between the terminal 110 and the server 130, and may include various connection types, such as wired, wireless communication links or optical fiber cables, etc. It should be understood that the number of terminals, networks, and servers in FIG1 is merely schematic. Depending on the implementation requirements, there may be any number of terminals, networks, and servers. For example, the server 130 may be a server cluster consisting of multiple servers, etc.

The image processing method provided in the embodiments of the present disclosure may be executed by the terminal 110, for example, after the terminal 110 captures the image, the image is processed; or may be executed by the server 130, for example, after the terminal 110 captures the image, the image is uploaded to the server 130, so that the server 130 processes the image. The present disclosure does not limit this.

An exemplary embodiment of the present disclosure provides an electronic device for implementing an image processing method, which may be the terminal 110 or the server 130 in Figure 1. The electronic device includes at least a processor and a memory, the memory is used to store executable instructions of the processor, and the processor is configured to execute the image processing method by executing the executable instructions.

Electronic devices can be implemented in various forms, for example, they can include mobile devices such as mobile phones, tablet computers, laptops, personal digital assistants (PDAs), navigation devices, wearable devices, drones, and fixed devices such as desktop computers and smart TVs. Below, the mobile terminal 200 in Figure 2 is taken as an example to illustrate the structure of the electronic device. It should be understood by those skilled in the art that, in addition to components specifically used for mobile purposes, the structure in Figure 2 can also be applied to fixed-type devices. In other embodiments, the mobile terminal 200 may include more or fewer components than shown, or combine certain components, or split certain components, or arrange different components. The illustrated components can be implemented in hardware, software, or a combination of software and hardware. The interface connection relationship between the components is only schematically shown and does not constitute a structural limitation on the mobile terminal 200. In other embodiments, the mobile terminal 200 may also adopt an interface connection method different from Figure 2, or a combination of multiple interface connection methods.

As shown in FIG2 , the mobile terminal 200 may specifically include: a processor 210, an internal memory 221, an external memory interface 222, a Universal Serial Bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 271, a receiver 272, a microphone 273, an earphone interface 274, a sensor module 280, a display screen 290, a camera module 291, an indicator 292, a motor 293, a button 294, and a Subscriber Identification Module (SIM) card interface 295, etc. The sensor module 280 may include a depth sensor 2801, a pressure sensor 2802, a gyroscope sensor 2803, an air pressure sensor 2804, etc.

The processor 210 may include one or more processing units, for example, the processor 210 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor and/or a neural network processor (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

A memory may also be provided in the processor 210 for storing instructions and data. The memory may store instructions for implementing six modular functions: detection instructions, connection instructions, information management instructions, analysis instructions, data transmission instructions, and notification instructions, and the execution is controlled by the processor 210. In some embodiments, the memory in the processor 210 is a cache memory. The memory may store instructions or data that the processor 210 has just used or circulated. If the processor 210 needs to use the instruction or data again, it may be directly called from the memory. Repeated access is avoided, the waiting time of the processor 210 is reduced, and the efficiency of the system is improved.

In some embodiments, the processor 210 may include one or more interfaces. The interface may include an Inter-Integrated Circuit (I2C) interface, an Inter-Integrated Circuit Sound (I2S) interface, a Pulse Code Modulation (PCM) interface, a Universal Asynchronous Receiver/Transmitter (UART) interface, a Mobile Industry Processor Interface (MIPI), a General-Purpose Input/Output (GPIO) interface, a Subscriber Identity Module (SIM) interface, and/or a Universal Serial Bus (USB) interface, etc. Connections are formed with other components of the mobile terminal 200 through different interfaces.

The USB interface 230 is an interface that complies with the USB standard specification, and specifically can be a MiniUSB interface, a MicroUSB interface, a USB Type C interface, etc. The USB interface 230 can be used to connect a charger to charge the mobile terminal 200, can also be connected to headphones to play audio through the headphones, and can also be used to connect the mobile terminal 200 to other electronic devices, such as computers, peripheral devices, etc.

The charging management module 240 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging implementations, the charging management module 240 may receive charging input from a wired charger through the USB interface 230. In some wireless charging implementations, the charging management module 240 may receive wireless charging input through a wireless charging coil of the mobile terminal 200. While the charging management module 240 is charging the battery 242, it may also power the electronic device through the power management module 241.

The power management module 241 is used to connect the battery 242, the charging management module 240 and the processor 210. The power management module 241 receives input from the battery 242 and/or the charging management module 240, and supplies power to the processor 210, the internal memory 221, the display screen 290, the camera module 291 and the wireless communication module 260. The power management module 241 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 241 can also be set in the processor 210. In other embodiments, the power management module 241 and the charging management module 240 can also be set in the same device.

The wireless communication function of the mobile terminal 200 can be implemented through the antenna 1, the antenna 2, the mobile communication module 250, the wireless communication module 260, the modem processor and the baseband processor.

Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the mobile terminal 200 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

The mobile communication module 250 can provide solutions for wireless communications including 2G/3G/4G/5G applied to the mobile terminal 200. The mobile communication module 250 may include at least one filter, a switch, a power amplifier, a low noise amplifier (Low Noise Amplifier, LNA), etc. The mobile communication module 250 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 250 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 250 can be set in the processor 210. In some embodiments, at least some of the functional modules of the mobile communication module 250 can be set in the same device as at least some of the modules of the processor 210.

The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 271, a receiver 272, etc.), or displays an image or video through a display screen 290. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 210 and be set in the same device as the mobile communication module 250 or other functional modules.

The wireless communication module 260 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) network), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the mobile terminal 200. The wireless communication module 260 can be one or more devices integrating at least one communication processing module. The wireless communication module 260 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and filters it, and sends the processed signal to the processor 210. The wireless communication module 260 can also receive the signal to be sent from the processor 210, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

In some embodiments, the antenna 1 of the mobile terminal 200 is coupled to the mobile communication module 250, and the antenna 2 is coupled to the wireless communication module 260, so that the mobile terminal 200 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include Global System for Mobilecommunications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division-Synchronous Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), New Radio (NR), BT, GNSS, WLAN, NFC, FM, and/or IR technology. The GNSS may include a Global Positioning System (GPS), a Global Navigation Satellite System (GLONASS), a Beidou Navigation Satellite System (BDS), a Quasi-Zenith Satellite System (QZSS) and/or a Satellite Based Augmentation System (SBAS).

The mobile terminal 200 implements the display function through a GPU, a display screen 290, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 290 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 210 may include one or more GPUs that execute program instructions to generate or change display information.

The display screen 290 is used to display images, videos, etc. The display screen 290 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the mobile terminal 200 may include 1 or N display screens 290, where N is a positive integer greater than 1.

The mobile terminal 200 can realize the shooting function through the ISP, the camera module 291, the video codec, the GPU, the display screen 290 and the application processor.

ISP is used to process the data fed back by the camera module 291. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in the camera module 291.

The camera module 291 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the mobile terminal 200 may include 1 or N camera modules 291, where N is a positive integer greater than 1. If the mobile terminal 200 includes N cameras, one of the N cameras is a main camera.

The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the mobile terminal 200 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

Video codecs are used to compress or decompress digital videos. The mobile terminal 200 may support one or more video codecs. Thus, the mobile terminal 200 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

The external memory interface 222 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the mobile terminal 200. The external memory card communicates with the processor 210 through the external memory interface 222 to implement a data storage function, such as storing music, video and other files in the external memory card.

The internal memory 221 can be used to store computer executable program codes, which include instructions. The internal memory 221 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the mobile terminal 200 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 221 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash memory (Universal Flash Storage, UFS), etc. The processor 210 executes various functional applications and data processing of the mobile terminal 200 by running instructions stored in the internal memory 221 and/or instructions stored in a memory provided in the processor.

The mobile terminal 200 can implement audio functions such as music playing and recording through the audio module 270 , the speaker 271 , the receiver 272 , the microphone 273 , the earphone interface 274 and the application processor.

The audio module 270 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signal. The audio module 270 can also be used to encode and decode audio signals. In some embodiments, the audio module 270 can be arranged in the processor 210, or some functional modules of the audio module 270 can be arranged in the processor 210.

The speaker 271 , also called a “speaker”, is used to convert an audio electrical signal into a sound signal. The mobile terminal 200 can listen to music or listen to a hands-free call through the speaker 271 .

The receiver 272, also called a "handset", is used to convert audio electrical signals into sound signals. When the mobile terminal 200 receives a call or voice message, the voice can be received by placing the receiver 272 close to the ear.

Microphone 273, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can make a sound by putting his mouth close to the microphone 273 to input the sound signal into the microphone 273. The mobile terminal 200 can be provided with at least one microphone 273. In other embodiments, the mobile terminal 200 can be provided with two microphones 273, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the mobile terminal 200 can also be provided with three, four or more microphones 273 to realize the collection of sound signals, noise reduction, identification of sound sources, and directional recording function, etc.

The earphone interface 274 is used to connect a wired earphone and can be a USB interface 230 or a 3.5 mm Open Mobile Terminal Platform (OMTP) standard interface or a Cellular Telecommunications Industry Association of the USA (CTIA) standard interface.

The depth sensor 2801 is used to obtain the depth information of the scene. In some embodiments, the depth sensor can be arranged in the camera module 291.

The pressure sensor 2802 is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 2802 can be disposed on the display screen 290. There are many types of pressure sensors 2802, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc.

The gyro sensor 2803 can be used to determine the motion posture of the mobile terminal 200. In some embodiments, the angular velocity of the mobile terminal 200 around three axes (i.e., x, y, and z axes) can be determined by the gyro sensor 2803. The gyro sensor 2803 can be used for anti-shake shooting. For example, when the shutter is pressed, the gyro sensor 2803 detects the angle of the shaking of the mobile terminal 200, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the mobile terminal 200 through reverse movement to achieve anti-shake. The gyro sensor 2803 can also be used for navigation and somatosensory game scenes.

The air pressure sensor 2804 is used to measure air pressure. In some implementations, the mobile terminal 200 calculates the altitude through the air pressure value measured by the air pressure sensor 2804 to assist positioning and navigation.

In addition, according to actual needs, sensors with other functions can also be set in the sensor module 280, such as magnetic sensors, acceleration sensors, distance sensors, proximity light sensors, fingerprint sensors, temperature sensors, touch sensors, ambient light sensors, bone conduction sensors, etc.

The key 294 includes a power key, a volume key, etc. The key 294 can be a mechanical key or a touch key. The mobile terminal 200 can receive key input and generate key signal input related to user settings and function control of the mobile terminal 200.

The motor 293 can generate vibration prompts, such as vibration prompts for incoming calls, alarms, and received messages, and can also be used for touch vibration feedback, such as touch operations on different applications (such as taking pictures, games, audio playback, etc.), or touch operations on different areas of the display screen 290, which can correspond to different vibration feedback effects. The touch vibration feedback effect can support customization.

Indicator 292 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

The SIM card interface 295 is used to connect a SIM card. The SIM card can be connected to and separated from the mobile terminal 200 by inserting it into the SIM card interface 295 or pulling it out from the SIM card interface 295. The mobile terminal 200 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 295 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 295 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 295 can also be compatible with different types of SIM cards. The SIM card interface 295 can also be compatible with external memory cards. The mobile terminal 200 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the mobile terminal 200 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the mobile terminal 200 and cannot be separated from the mobile terminal 200.

The image processing method and the image processing apparatus according to the exemplary embodiments of the present disclosure will be described in detail below.

FIG3 shows a flow chart of an image processing method in this exemplary embodiment, including the following steps S310 to S330:

Step S310: identifying a sky area in the first image based on the segmentation model, and obtaining a mask image corresponding to the sky area.

Among them, the first image is an image to be processed containing the sky, which can be the original image captured by the camera when the user takes a photo. The segmentation model is a pre-trained machine learning model, which can be used to perform feature processing on the image and identify the sky area in the image. For example, YOLO (You Look Only Once, a real-time target detection algorithm framework, including multiple versions such as v1, v2, v3, etc., any one of which can be used in the present disclosure), SSD (Single Shot Multibox Detector, single-step multi-box target detection), R-CNN (Region-Convolutional Neural Network, regional convolutional neural network, or Fast R-CNN, Faster R-CNN and other improved versions) and other target detection neural networks can be used to detect the sky area in the first image. After obtaining the position of the sky area in the first image, the pixels in the sky area can be set to 1 (white), and the pixels outside the sky area can be set to 0 (black), thereby obtaining a mask image corresponding to the sky area.

In an optional implementation, referring to FIG. 4 , step S310 may include the following steps S401 to S403:

Step S401, normalizing the pixel values of the first image to obtain a normalized image;

Step S402, processing the normalized image based on a pre-trained fully convolutional neural network to obtain a response spectrum of the normalized image to the sky;

Step S403: binarize the response spectrum to obtain a mask image corresponding to the sky area in the first image.

Among them, the Fully Convolutional Network (FCN) is an image processing network based on semantic segmentation. It can extract local features in the image by convolution and downsampling the image, and then restore it to the original image size through deconvolution and upsampling to achieve pixel-level classification. The Fully Convolutional Neural Network includes a variety of upgraded versions, such as Unet (a segmentation model). The following takes Unet as an example to explain the network training process in detail:

According to actual needs, set up Unet with single-channel input or three-channel input. The single channel is used to input grayscale images, and the three channels are used to input RGB color images. Taking the three-channel as an example, obtain a sample image, and manually cut out the image through image processing software to mark the sky area in the sample image as the label corresponding to the sample image. The RGB pixel values of the sample image are respectively set to 255 for normalization, and then input into the three channels of Unet respectively. By calculating the error between the output data and the label, the parameters of Unet are adjusted for training. When the accuracy of Unet in the test set reaches a certain standard, the training is completed and a usable Unet is obtained.

In practical applications, the pixel values of the first image can be normalized according to the requirements of the input channel of the full convolutional neural network to obtain a normalized image; then the full convolutional neural network is input to output the response spectrum of the normalized image to the sky. In the response spectrum, the value of each pixel position represents the probability that the pixel belongs to the sky area. Then the response spectrum is binarized, and the threshold is manually set or adaptively calculated, and each pixel point is classified as 0/1 using the threshold, so that the binary image corresponding to the sky area in the first image, that is, the mask image, can be obtained.

Step S320: segmenting a foreground area image from the first image and segmenting a background area image from the second image using the mask image.

The second image is an image required to replace the sky background in the first image, and may be a pre-configured template image with sky material as the main image content. For example, some template images containing the sky may be downloaded from the Internet, and when the user needs to change the image background, a template image selection interface is displayed, allowing the user to select one of them as the second image.

After obtaining the above-mentioned mask image, the mask image can be used to extract the part outside the sky area from the first image to obtain the foreground area image; conversely, the part corresponding to the sky area is extracted from the second image to obtain the background area image.

In an optional implementation, step S320 may include:

multiplying the inverse mask image of the mask image and the first image to segment a foreground area image from the first image;

The mask image is multiplied by the second image to segment the background region image from the second image.

The first image is IMG1, the second image is IMG2, the mask image is Mask, the inverse mask image is Max-Mask, and Max is the maximum pixel value, which is generally 1 or 255. As shown in the following formulas (1) and (2):

IMG _F =IMG1×(Max-Mask)/Max; (1)

IMG _B =IMG2×Mask/Max; (2)

Among them, IMG _F is the foreground area image, and IMG _B is the background area image. In the mask image, the sky area is white, and the part outside the sky is black; in the inverse mask image, the sky area is black, and the part outside the sky is white. The inverse mask image can be obtained by inverting the mask image. Multiplying the inverse mask image with the first image is equivalent to retaining the part of the image outside the sky, that is, the foreground area image; multiplying the mask image with the second image is equivalent to retaining the part of the image corresponding to the sky area, that is, the background area image

Step S330, stitching the foreground area image and the background area image to obtain a target image.

Among them, the sky part of the target image relative to the first image is the corresponding partial image in the second image, thereby achieving a "sky replacement" processing effect.

In an optional implementation, the foreground area image and the background area image may be added together to obtain the target image, as shown in the following formula (3):

IMG _T =IMG _F +IMG _B ; (3)

Where IMG _T is the target image. Since the pixel value of the black part itself is 0, after addition, the foreground area image and the background area image just complement each other to obtain a complete target image.

In an optional embodiment, after obtaining the target image, the target image can also be subjected to edge smoothing. The purpose is to form a color transition effect at the edge portion where the foreground area and the background area are spliced, alleviate the impact of segmentation jaggedness, and reduce the visual sense of mutation. Generally, when splicing the foreground area image and the background area image, the position coordinates of the edge portion can be determined; then in the target image, the edge portion is expanded to a certain extent, for example, each pixel point of the edge portion can be used as the center of the circle, and each pixel point can be circularly expanded by a preset radius (related to the size of the target image, such as 5 pixels) to form an edge area to be smoothed; then any smoothing method can be used for processing, for example, Gaussian smoothing can be used: based on the two-dimensional normal distribution of the target image, the center point of the edge area to be smoothed is selected to construct a smoothing algorithm formula based on the normal distribution; a 3x3, 5x5 or higher-order weight matrix is calculated and convolved with the target image; the number of iterations, that is, the number of convolutions, is set as needed, and finally a target image with edge transition is obtained.

Since the tones of the first image and the second image may be quite different, after the foreground area image and the background area image are spliced, the target image obtained may have two tones with large contrast. Therefore, color migration can be performed to merge the tones of the first image and the second image, so that the target image is more realistic and natural visually. Referring to FIG5 , color migration can be achieved by following the steps S501 and S502.

Step S501, obtaining the mean and variance of the second image in the Lab color space;

Step S502: adjusting the target image in the Lab color space according to the mean and variance to perform color migration.

The Lab color space is a color mode that conforms to the visual perception characteristics of the human eye. L represents brightness, and a and b are two color channels. a is from dark green (low brightness value) to gray (medium brightness value) and then to bright pink (high brightness value), and b is from bright blue (low brightness value) to gray (medium brightness value) and then to yellow (high brightness value). In color migration, it is usually hoped that when changing one color attribute, other color attributes will not be affected. Since the three channels of the RGB color space have a high correlation, and the channels of the Lab color space have a low correlation, the Lab color space is chosen for color migration.

First, both the second image and the target image are converted from the RGB color space to the Lab color space; then, the Lab channel value of each pixel in the second image is counted, and the mean and variance (or standard deviation, the two indicators have basically the same meaning, and the present disclosure does not make a special distinction) of each channel are calculated respectively; then, according to the mean and variance, the color of the pixels in the target image is adjusted, and the Lab channel value of the target image can be moved as a whole according to the difference between the Lab channel means of the target image and the second image, and the distribution of the Lab channel value of the target image is adjusted according to the Lab channel variance of the second image, so as to integrate the color features of the second image into the target image.

It should be noted that the above-mentioned edge smoothing and color migration are two ways of post-processing the target image, and their purpose is to eliminate the defects that may be caused by image processing and improve the quality and visual realism of the image. Therefore, other post-processing methods can also be adopted, such as filtering the entire target image, optimizing the image brightness and contrast, eliminating image distortion, etc.

FIG6 shows a schematic flow of image processing in this exemplary embodiment. As shown in FIG6, after acquiring the first image, the sky area in the first image is segmented by a segmentation model to extract a mask image from the first image; the mask image is inverted, that is, the pixel value of each pixel is subtracted from 1 to obtain an inverse mask image; the inverse mask image is multiplied with the first image to extract a foreground area image from the first image; the mask image is multiplied with the second image to extract a background area image from the second image; then the foreground area image and the background area image are spliced to obtain an initial target image (i.e., target image 1); and the target image is post-processed by edge smoothing, color migration, etc. to obtain a final target image (i.e., target image 2).

In summary, in this exemplary embodiment, based on the above-mentioned image processing method, on the one hand, this solution can accurately segment the sky area from the image based on the recognition of the sky area by the segmentation model and the processing of the mask image, without the need for the user to manually cut out the image, with a high degree of intelligence, easy to use, and good user experience. On the other hand, this solution realizes the "sky replacement" processing of the image, replacing the sky area in the first image with the corresponding part in the second image, which is highly interesting and can meet the diverse needs of users.

FIG. 7 shows an image processing device 700 according to this exemplary embodiment, which may include the following modules:

A sky recognition module 710 is used to recognize a sky area in the first image and obtain a mask image corresponding to the sky area;

An image segmentation module 720, configured to segment a foreground region image from the first image and a background region image from the second image using a mask image;

The image stitching module 730 is used to stitch the foreground area image and the background area image to obtain a target image.

In an optional implementation, the sky recognition module 710 is configured to obtain the mask image by executing the following method:

Normalizing the pixel values of the first image to obtain a normalized image;

The normalized image is processed based on a pre-trained fully convolutional neural network to obtain the response spectrum of the normalized image to the sky;

The response spectrum is binarized to obtain a mask image corresponding to the sky area in the first image.

In an optional embodiment, the image segmentation module 720 is used to multiply the inverse mask image of the mask image with the first image to segment the foreground area image from the first image, and to multiply the mask image with the second image to segment the background area image from the second image.

In an optional implementation, the image stitching module 730 is used to add the foreground area image and the background area image to obtain a target image.

In an optional implementation, the image processing device 700 further includes: a post-processing module, configured to perform edge smoothing processing on the target image after obtaining the target image.

In an optional embodiment, the image processing device 700 further includes: a post-processing module for adjusting the target image in the Lab color space according to the mean and variance of the second image in the Lab color space after obtaining the target image to perform color migration.

In an optional implementation, the first image may be an original image captured by a camera, and the second image may be a preconfigured template image.

The specific details of each module in the above device have been described in detail in the implementation method of the method part. The undisclosed details can be found in the implementation method of the method part, so they will not be repeated here.

It will be appreciated by those skilled in the art that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits", "modules" or "systems".

The exemplary embodiments of the present disclosure also provide a computer-readable storage medium on which a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary embodiments of the present disclosure described in the above "Exemplary Method" section of the present specification, for example, any one or more steps in FIG. 3, FIG. 4, or FIG. 5 may be executed.

Referring to FIG8 , a program product 800 for implementing the above method according to an exemplary embodiment of the present disclosure is described, which may adopt a portable compact disk read-only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto, and in this document, a readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, apparatus, or device.

The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, wherein readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and embodiments are to be regarded as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims (8)

1. An image processing method, comprising:

Identifying a sky area in the first image based on the segmentation model to obtain a mask image corresponding to the sky area;

dividing a foreground region image except the sky region from the first image through the mask image, and dividing a background region image corresponding to the sky region from a second image;

Splicing the foreground region image and the background region image to obtain a target image;

Acquiring the mean value and the variance of the second image in the Lab color space;

according to the difference between the average value of the target image in the Lab color space and the average value of the second image in the Lab color space, the Lab channel value of the target image is integrally moved;

according to the variance of the second image in the Lab color space, the distribution of Lab channel values of the target image is adjusted;

Determining an edge portion between the foreground region image and the background region image in the target image; expanding the edge part to form an edge region to be smoothed; selecting a center point of the region to be smoothed of the edge based on the two-dimensional normal distribution of the target image, constructing a smoothing algorithm formula based on the normal distribution, and calculating a weight matrix; and carrying out iteration convolution on the target image by utilizing a weight matrix according to the set iteration times.

2. The method of claim 1, wherein the identifying, based on the segmentation model, a sky region in the first image to obtain a mask image corresponding to the sky region comprises:

Normalizing the pixel value of the first image to obtain a normalized image;

processing the normalized image based on a pre-trained full convolution neural network to obtain a response spectrum of the normalized image to sky;

And performing binarization processing on the response spectrum to obtain a mask image corresponding to the sky area in the first image.

3. The method of claim 1, wherein the segmenting the foreground region image from the first image and the background region image from the second image by the mask image comprises:

Multiplying an inverse mask image of the mask image with the first image to segment a foreground region image from the first image;

the mask image is multiplied with the second image to divide a background area image from the second image.

4. A method according to claim 3, wherein said stitching said foreground region image and said background region image to obtain a target image comprises:

And adding the foreground region image and the background region image to obtain the target image.

5. The method of any one of claims 1 to 4, wherein the first image is a raw image captured by a camera and the second image is a pre-configured template image.

6. An image processing apparatus, comprising:

The sky identification module is used for identifying a sky area in the first image based on the segmentation model to obtain a mask image corresponding to the sky area;

The image segmentation module is used for segmenting foreground region images except the sky region from the first image through the mask image and segmenting background region images corresponding to the sky region from the second image;

the image stitching module is used for stitching the foreground region image and the background region image to obtain a target image;

The post-processing module is used for acquiring the mean value and the variance of the second image in the Lab color space; according to the difference between the average value of the target image in the Lab color space and the average value of the second image in the Lab color space, the Lab channel value of the target image is integrally moved; according to the variance of the second image in the Lab color space, the distribution of Lab channel values of the target image is adjusted;

The post-processing module is further configured to determine an edge portion between the foreground region image and the background region image in the target image; expanding the edge part to form an edge region to be smoothed; selecting a center point of the region to be smoothed of the edge based on the two-dimensional normal distribution of the target image, constructing a smoothing algorithm formula based on the normal distribution, and calculating a weight matrix; and carrying out iteration convolution on the target image by utilizing a weight matrix according to the set iteration times.

7. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the method of any one of claims 1 to 5.

8. An electronic device, comprising:

A processor; and

A memory for storing executable instructions of the processor;

wherein the processor is configured to perform the method of any one of claims 1 to 5 via execution of the executable instructions.