JP2025071579A - Image processing device and learning method for image processing device - Google Patents

Abstract

To provide a learning method for an image processing apparatus that naturally reproduces the movement of original moving images.SOLUTION: An image processing parameter learning method comprises: image processing means; means that acquires, from time-series images, a plurality of original images, target images corresponding to the plurality of original images, and information on the movement between two images of the target images, as learning data; means that calculates a first error from an estimated image obtained by inputting at least one image of the plurality of acquired original images to the image processing means and the target image corresponding to an input image; means that estimates information on the movement between first and second estimated images obtained by acquiring first and second original images from learning data acquisition means and inputting them to the image processing means; means that calculates a second error from the estimated movement information and information on the movement between target images corresponding to the first and second original images acquired by the learning data acquisition means; and learning means that learns the parameter of the image processing means, by using the error determined by first error calculation means and the error determined by second error calculation means, as a loss function.SELECTED DRAWING: Figure 3

Description

The present invention relates to an image processing device that uses machine learning to increase the resolution of images, and a learning method for the same.

Conventionally, methods of enlarging an image using bilinear interpolation or the like have been commonly used to improve image resolution. However, when such simple numerical data interpolation methods are applied to an image, there are problems such as blurring the enlarged image and reducing sharpness. To solve these problems, methods of super-resolution processing using machine learning have been proposed in recent years. In particular, super-resolution processing using neural networks using deep learning has significantly improved image quality (see Non-Patent Document 1). In addition, a method has been proposed for further improving image quality in moving images by using images from adjacent frames (Non-Patent Document 2).

Dong et al. Image Super-Resolution Using Deep Convolutional Networks. Proceedings of the European Conference on Computer Vision, 2014 Caballero et al. Realtime video super-resolution with spatio-temporal networks and motion compensation. IEEE Conference on Computer Vision and Pattern Recognition, 2017 Shi et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. Proceedings of the European Conference on Computer Vision, 2016 Xie et al. Image Denoising and Inpainting with Deep Neural Network. Conference on Neural Information Processing Systems, 2012

In the method proposed in Non-Patent Document 2, an image is output that has been subjected to super-resolution processing of a single target frame for each frame of a video. Therefore, when a time-series image is reconstructed from the processing results, the movement of the original video image cannot be guaranteed, and the movement may be reproduced unnaturally. In addition, there is also the issue that when using images from adjacent frames, it is necessary to estimate and compensate for the movement between frames.

The object of the present invention is to provide an image processing device that naturally reproduces the movement of the original video in an image processing device that increases the resolution of video images in order to solve the above problems, and a learning method thereof.

To achieve the above object, the learning method of the image processing device of the present invention has the following configuration.

That is, the learning method of the image processing device of the present invention includes an image processing means, a means for acquiring a plurality of original images and target images corresponding to the plurality of original images from a time series image as learning data, and motion information between two images of the target images, a means for calculating a first error from an estimated image obtained by inputting at least one image of the plurality of original images acquired by the learning data acquisition means into the image processing means and a target image corresponding to the input image, a means for estimating motion information between a first and a second estimated image obtained by acquiring a first and a second original image from the learning data acquisition means and inputting each of them into the image processing means, a means for calculating a second error from the estimated motion information and motion information between target images corresponding to the first and second original images acquired by the learning data acquisition means, and a learning means for learning parameters of the image processing means using the error calculated by the first error calculation means and the error calculated by the second error calculation means as a loss function.

According to the present invention, in learning image processing parameters in an image processing device, not only the error in the estimated image of the image processing means but also the estimated error in the motion between multiple frames in a video is learned as a loss function, so that when a time-series image is reconstructed from the results of image processing, the motion of the original video can be guaranteed.

1 is a diagram showing the functional configuration of an entire system for performing processing and learning of an image processing device according to an embodiment of the present invention. 1 is a diagram illustrating a functional configuration of an image processing apparatus according to an embodiment of the present invention. FIG. 2 is a diagram illustrating a functional configuration of a learning device according to an embodiment of the present invention. FIG. 2 is a diagram illustrating an example of the configuration of an image processing unit that performs high-resolution processing. FIG. 11 is a diagram showing a flow of learning processing of the image processing device. FIG. 2 is a diagram showing a processing flow of the image processing device. 1 is a diagram showing a hardware configuration of an entire system for performing processing and learning of an image processing device according to an embodiment of the present invention.

The following describes an embodiment of the present invention with reference to the drawings.

Figure 1 is a diagram showing the functional configuration of the entire system for performing image processing and learning in an image processing device according to an embodiment of the present invention. The image processing device 100 acquires image data to be processed and outputs the image processing results. The learning device 200 learns parameters for the image processing device 100 to perform image processing.

Figure 7 is a diagram showing the hardware configuration of the entire system for performing processing and learning of the image processing device according to this embodiment. The system for performing processing and learning of the image processing device includes an arithmetic processing device 1, a storage device 2, an input device 3, and an output device 4. Each device is configured to be able to communicate with each other, and is connected by a bus or the like.

The arithmetic processing device 1 controls the operation of the image processing device 100 and the learning device 200, executes programs stored in the memory device 2, and is composed of a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The memory device 2 is a storage device such as a magnetic memory device or semiconductor memory, and stores programs loaded based on the operation of the arithmetic processing device 1, data that must be stored for a long time, and the like. In this embodiment, the arithmetic processing device 1 performs processing according to the procedures of the programs stored in the memory device 2, thereby realizing the functions of the image processing device 100 and the learning device 200 and the processing related to the flowcharts described below. The memory device 2 also stores images and processing results to be processed by the image processing device 100 according to the embodiment of the present invention, and learning data to be processed by the learning device 200.

The input device 3 is a mouse, keyboard, touch panel device, button, etc., and is used to input various instructions. The input device 3 also includes an imaging device such as a camera. The output device 4 is a liquid crystal panel, external monitor, etc., and is used to output various information.

The hardware configuration of the entire system is not limited to the above-mentioned configuration. For example, the image processing device 100 may be equipped with an I/O device for communicating between various devices. For example, the I/O device is an input/output unit such as a memory card or a USB cable, or a transmission/reception unit via wired or wireless connection.

Figure 2 is a diagram showing the functional configuration of the image processing device 100. As shown in the figure, the processing and functions of the image processing device of the present invention are realized by an image acquisition means 110, an image processing means 120, and a moving image reconstruction means 130.

The image acquisition means 110 acquires image data from time-series video images captured by the camera of the input device 3.

The image processing means 120 processes the image data acquired by the image acquisition means 110 and outputs a high-resolution image.

The video reconstruction means 130 acquires multiple frames of high-resolution images obtained by the image processing means 120 and reconstructs a time-series video. The reconstructed video is output to the monitor of the output device 4, etc.

Figure 3 is a diagram showing the functional configuration of the learning device 200. As shown in the figure, the processing and functions of the learning device of the present invention are realized by a learning data storage means 210, a learning data acquisition means 220, a first error calculation means 230, a motion estimation means 240, a second error calculation means 250, and a parameter learning means 260.

The learning data storage means 210 stores and holds learning data for the learning device 200 to use for learning. The learning data consists of a set of a low-resolution original image and a high-resolution target image corresponding to each original image for each frame of a time-series video. For example, this set of original image and target image is reduced (e.g., halved) by bilinear interpolation with a single image constituting the video as the target image to obtain the original image. The learning data also holds motion information between the target images of two adjacent frames. The motion information is associated in advance with position coordinates as a motion vector for each pixel of the target image at the previous and next times.

Here, the video images that serve as learning data can be obtained by capturing various scenes with a camera. The motion information between frames can be obtained by, for example, using the Lucas-Kanade method to obtain the motion vector between two images for each pixel. Alternatively, it may be obtained by a person associating characteristic points such as corners in the images between the two images. In addition, the video images that serve as learning data can be pseudo-created by performing geometric transformations such as translation, enlargement, reduction, and rotation on a single still image, and the transformation information used when creating the video images can be used to determine the motion vectors and use them as the motion information. Even when a sufficient amount of actually captured video images cannot be obtained, the amount of learning data can be sufficiently expanded by utilizing geometric transformations.

The learning data acquisition means 220 acquires the learning data stored in the learning data storage means 210.

The first error calculation means 230 calculates a first error from a high-resolution estimated image obtained by inputting the low-resolution original image acquired by the learning data acquisition means 220 into the image processing means 120 of the image processing device 100 and a high-resolution target image corresponding to the original image.

The motion estimation means 240 estimates motion information between high-resolution estimated images obtained by inputting the low-resolution original images of two adjacent frames acquired by the learning data acquisition means 220 into the image processing means 120 of the image processing device 100.

The second error calculation means 250 calculates a second error from the motion information estimated by the motion estimation means 240 and the motion information between the target images corresponding to the original images of two adjacent frames acquired by the learning data acquisition means 220.

The parameter learning means 260 learns the parameters of the image processing means 120 of the image processing device 100 using the first error calculated by the first error calculation means 230 and the second error calculated by the second error calculation means 250 as a loss function.

The learning and image processing operations of the image processing device according to the embodiment of the present invention will be described below. Here, the image processing means 120 increases the resolution of the input image, for example using the method proposed in Non-Patent Document 1. Figure 4 shows the configuration. The image enlargement process 1210 takes an RGB image with a resolution of H x W as input and enlarges the image to a resolution of 2H x 2W using bicubic interpolation. The convolutional neural network 1220 is a neural network consisting of three convolutional layers, and processes the image enlarged by the image enlargement process 1210 so that it becomes the target high-resolution image.

Figure 5 shows the process flow for learning an image processing device.

The learning data acquisition means 220 acquires the learning data stored in the learning data storage means 210 (S201). As described above, the learning data consists of a set of low-resolution source images and high-resolution target images corresponding to each of the source images. In the present invention, in order to guarantee the movement of the original video image when a time-series image is reconstructed from the results of image processing, N pairs of source and target images of two adjacent frames are acquired. In addition, motion information between the target images of two adjacent frames is also acquired.

The image processing means 120 of the image processing device 100 inputs the original image of the learning data acquired in S201 and obtains a high-resolution estimated image (S202). In this embodiment, as described above, the image processing means 120 estimates a high-resolution image by the image enlargement process and convolutional neural network shown in FIG. 4. A high-resolution image is obtained for each of the N pairs of original images of the learning data.

The first error calculation means 230 calculates a first error from the high-resolution estimated image acquired in S202 and the target image of the learning data acquired in S201 (S203). If one estimated image of the n-th (1≦n≦N) pair of learning data and the target image that is the correct answer are I _n,k and I′ _n,k , respectively, the first error E1 _n,k can be calculated by (Equation 1). Here, k represents one of the two learning data that make up the pair, taking the value 1 or 2. |I _n,k -I′ _n,k | represents the sum of absolute values of the differences in pixel values of corresponding pixels that make up the images I _n,k and I _{′ n,k} .
E1 _n,k = |I _n,k -I' _n,k | (Equation 1)

The motion estimation means 240 estimates motion information between the high-resolution estimated images of the two adjacent frames acquired in S202 (S204). For example, the motion information can be obtained by estimating the motion vector between the two images for each pixel using the Lucas-Kanade method. Alternatively, the motion vector can be estimated using a neural network with the two images as input.

The second error calculation means 250 calculates a second error from the motion information estimated in S204 and the motion information in the learning data acquired in S201 (S205). If the motion vector estimated from the n-th pair of learning data and the correct motion vector are f _n and f' _n , respectively, the second error E2 _n can be calculated by (Equation 2). Here, |f _n -f' _n | represents the sum of the absolute values of the differences between the corresponding horizontal and vertical components constituting the motion vectors f _n and f' _n .
E2 _n = |f _n −f' _n | (Formula 2)

In this embodiment, the first error and the second error are calculated as the absolute value of the difference between the estimated value and the target value as shown in (Equation 1) and (Equation 2), but other methods, such as squaring the difference, may also be used.

The parameter learning means 260 calculates a total loss value of the first error calculated in S203 and the second error calculated in S205 (S206). The loss function L is as shown in (Equation 3). Here, L1 and L2 are the sums of the first error and the second error for the learning data, respectively. Also, λ is a weighting coefficient representing the weighting of the first error and the second error in the loss function. Σ represents the sum of the learning data.
L = L1+λ・L2 (Formula 3)
L1 = ΣE1 _n,k , L2 = ΣE2 _n

In addition, in (Equation 3), the first error sum L1 may be calculated for both k=1 and k=2, i.e., for two adjacent frames, or may be calculated as the sum for only one of the images.

In addition, to prevent overlearning of the neural network, a regularization term such as the sum of the absolute values of the desired parameters may be added to (Equation 3).

The parameter learning means 260 updates the parameters of the neural network, which is the image processing means, based on the loss value calculated in S206 (S207). The parameters to be updated are the weight coefficients of the convolution layer of the neural network. The parameter updating is performed using the backpropagation method or the like.

Through steps S201 to S207, learning for the learning data acquired in S201 is completed.

The parameter learning means 260 judges whether or not learning has been completed according to a preset termination condition (S208). As the termination condition, a method is used in which learning data for accuracy verification is prepared separately from the learning data for parameter update, and the above-mentioned processes from S201 to S206 are performed to obtain a loss value, and whether or not the loss value has become equal to or less than a predetermined value is judged. Alternatively, the judgment may be made based on the number of times steps from S201 to S207 are repeated. If it is judged that learning has not been completed, the process returns to S201.

The above describes the learning of the image processing device. Below, we will explain the processing of the image processing device using the parameters learned by the above-mentioned learning method. Figure 6 shows the processing flow of the image processing device.

The image acquisition means 110 acquires a single image data for each frame from the time-series video image to be subjected to high-resolution processing (S301).

The image processing means 120 inputs the image data acquired in S301 and obtains a high-resolution estimated image (S302). The estimated image is obtained by the image enlargement process shown in FIG. 4 and the convolutional neural network, as described above. Here, the parameters of the convolutional neural network are obtained by the learning method described above.

As shown in FIG. 6, the process of S301 to S302 is repeated for each frame of the moving image acquired by the image acquisition means 110.

The video reconstruction means 303 reconstructs a time-series video from the multiple high-resolution images obtained in S302 (S303). The reconstructed video is output to the monitor of the output device 4.

As described above, in the embodiment of the present invention, not only the estimation error of the image processing means but also the estimation error of the motion between multiple frames in the video is learned as a loss function. Then, using the trained neural network parameters, the image processing device estimates a high-resolution image for each frame of the video and reconstructs the video. By performing such learning processing, it is possible to obtain a high-resolution video that guarantees the motion of the original video and reproduces natural motion. Furthermore, in the image processing of the above-mentioned embodiment, there is no need to perform processing to estimate and compensate for the motion between frames as in the method of Non-Patent Document 2, and high resolution of the video can be achieved with a simple configuration.

In this embodiment, the second error is calculated by estimating a motion vector as the motion information, but other methods may be used. For example, geometric transformation parameters such as translation, enlargement, and rotation that represent the motion of the entire image may be used as the motion information. In this case, the geometric transformation parameters of two adjacent frames are stored in advance as learning data, the geometric transformation parameters between the estimated images are estimated in S204, and the difference between the geometric transformation parameters is calculated as the second error in S205. In addition, acceleration may be used as well as speed as the motion information. In this case, the motion vector and acceleration vector are calculated from three consecutive frames. The motion vectors and acceleration vectors of three consecutive frames are stored in advance as learning data. Then, the motion vector and acceleration vector between the estimated images are estimated in S204, and the difference between the motion vector and acceleration vector is calculated as the second error in S205.

In addition, the learning method of the Generative Adversarial Networks can be applied to the learning method of the present invention. In the learning of the Generative Adversarial Network, in the learning of the Generator that generates images, first, a Discriminator that identifies images generated by the Generator is learned. Then, the Generator is trained to deceive the trained Discriminator, so that it generates images that are indistinguishable from the real thing. When applied to the above-mentioned embodiment, the loss function L during learning of the convolutional neural network 1220, which is the Generator, can be calculated by the following (Equation 4).
L = L1+λ・L2+λ′・L3 (Formula 4)
L3 = ΣE3 _n,k

Here, L3 is the adversarial loss for deceiving the Discriminator, and E3 _n,k is the loss for each estimated image. In (Equation 4), when a high-resolution estimated image estimated by the image processing means 120 is input to the Discriminator, if it is identified as a true high-resolution image, E3 _n,k = 0, and if it is identified as an estimate by the Generator, E3 _n,k = 1. λ' is a weighting coefficient representing the weighting of the adversarial loss in the loss function.

In the above explanation, the method proposed in Non-Patent Document 1 was used as the image processing means for increasing the resolution of the input image, but other methods may also be used. For example, the method proposed in Non-Patent Document 3 uses Subpixel Convolution without using image enlargement processing to obtain a high-resolution image from the input image using only a neural network.

Another feature of the present invention is that when learning the parameters of the image processing means, a second error based on motion information is used as the loss function, and it is possible to combine this with the method proposed in Non-Patent Document 2.

Although the present invention has been described above as an example of increasing the resolution of an image, the present invention can also be applied to other image processing. For example, in Non-Patent Document 4, noise reduction processing and image restoration of an input image are performed using a neural network.

Even with such a neural network, if image processing is applied to each frame of a video and a time series of images is reconstructed from the processing results, there is a possibility that the movement will be reproduced unnaturally. In learning the parameters of the neural network, by learning not only the image estimation error but also the motion estimation error between multiple frames as a loss function, it is possible to guarantee the movement of the original video, as in the embodiments of the present invention.

Needless to say, the object of the present invention can also be achieved by supplying a recording medium (or storage medium) on which is recorded software program code that realizes the functions of the above-mentioned embodiments to a system or device, and having the computer (or CPU or MPU) of that system or device read and execute the program code stored on the recording medium. In this case, the program code read from the recording medium itself realizes the functions of the above-mentioned embodiments, and the recording medium on which the program code is recorded constitutes the present invention.

Furthermore, it goes without saying that not only are the functions of the above-mentioned embodiments realized by the computer executing the program code it has read, but also that the above-mentioned functions of the embodiments are realized by the operating system (OS) running on the computer carrying out all or part of the actual processing based on the instructions of the program code.

Furthermore, it goes without saying that this also includes cases where the program code read from the recording medium is written into a memory provided on a function expansion card inserted into a computer or a function expansion unit connected to a computer, and then a CPU provided on the function expansion card or function expansion unit performs some or all of the actual processing based on the instructions of the program code, thereby realizing the functions of the above-mentioned embodiments.

When the present invention is applied to the above-mentioned recording medium, the recording medium stores program code corresponding to the processing flow described above.

Reference Signs List 1 Processing device 2 Storage device 3 Input device 4 Output device 100 Image processing device 110 Image acquisition means 120 Image processing means 130 Moving image reconstruction means 200 Learning device

Claims (6)

A method for learning image processing parameters, comprising: an image processing means; a means for acquiring, as learning data, a plurality of original images and target images corresponding to the plurality of original images from a time series of images, and motion information between two of the target images; a means for calculating a first error from an estimated image obtained by inputting at least one image of the plurality of original images acquired by the learning data acquisition means into the image processing means and a target image corresponding to the input image; a means for estimating motion information between a first and a second estimated image obtained by acquiring a first and a second original image from the learning data acquisition means and inputting each of them into the image processing means; a means for calculating a second error from the estimated motion information and motion information between target images corresponding to the first and the second original images acquired by the learning data acquisition means; and a learning means for learning parameters of the image processing means using the error calculated by the first error calculation means and the error calculated by the second error calculation means as a loss function. The learning method according to claim 1, characterized in that the image processing means performs one of high-resolution processing, noise reduction processing, and image restoration processing of the input image. The learning method according to claim 1, characterized in that the motion information is a motion vector. The learning method according to claim 1, characterized in that the image processing means includes a convolutional neural network, and the learning means learns parameters of the convolutional neural network. An image processing device equipped with image processing means trained using the learning method according to any one of claims 1 to 4. 6. The image processing device according to claim 5, further comprising: a means for acquiring an image from a time series of images; and a moving image reconstructing means for reconstructing a time series of images from an estimated image obtained by inputting each of a plurality of images acquired by the image acquiring means into the image processing means.

Images