JP6727642B2 - Focus correction processing method by learning algorithm - Google Patents

Description

The present invention relates to a focus correction processing method using a learning algorithm.

Low-resolution viewfinder for image confirmation In order to improve the visibility of images, a method to enable focus adjustment in super high-definition images with a low-resolution viewfinder by multiplexing signals for assisting focus adjustment is available. , Non-Patent Document 1 below.

Further, focus correction by image processing has a long history as image restoration processing in various fields, and there are many studies. The celestial body image is disclosed in detail in Non-Patent Document 2 below.

Ryohei Funazu, Takayuki Yamashita, Koji Mitani, Yuji Nojiri, Focus Auxiliary Signal for Super Hi-Vision Cameras, Journal of Image Information Media, 65-4 (April 2011), 531-539. J.-L. Starck, and F. Murtagh, Astronomical Image and Data Analysis, Springer, 2006. W. H. Richardson, Bayesian-based iterative method of image restoration, Journal of the Optical Society of America, 62-1, pp. 55-59 (1972) L. B. Lucy, An iterative technique for the rectification of observed distributions, Astronomical Journal, 79-6, pp. 745-754(1974)

JP, 2010-061541, A JP, 2014-099048, A

Even if a method of multiplexing a signal for assisting focus adjustment on an image is used, it is not always possible to perform sufficient focus adjustment as long as it is visible. Many of the methods in image processing are reconstruction-type processes, which are iterative. A typical method is the Richardson-Lucy algorithm, but the iterative restoration process entails enormous processing costs. Although the number of repetitions has been reduced or the number of repetitions has been fixed, it is still impossible to process a high-definition image in real time.

A Wiener filter is known as a filter that performs optimal deflutter restoration in the meaning of least squares, but a true image and a power spectrum of a noise component are required for optimal restoration. The fact that the true image information is needed for the process of restoring the true image is a problem for eggs and chickens. Although information about such an image or noise component may be obtained approximately, it is usually an empirical adjustment parameter. Since the pixels of the entire image are used for processing, a frame memory is required. Moreover, since the processing is performed in the frequency domain, the processing cost is high. The present invention has been made in view of the above problems, and an object of the present invention is to propose a focus correction processing method using a learning algorithm for processing a high-definition image in real time.

The present invention uses a convolutional neural network that is a learning algorithm for deblurring processing. Generates a smoothed input image assuming a focus shift from a true image (output expected image). The parameters of the network are estimated (learned) in advance so that the sum of squared differences between the restored image obtained by processing the smoothed input image by the network and the true image is minimized using such input/output images as learning data. .. The learning parameter is determined so that the average ISNR of the deblurring restoration results in another evaluation image different from the learning image becomes maximum. In this way, focus correction processing is performed using the convolutional neural network based on the learned parameters determined.

According to the present invention, it is possible to propose a focus correction processing method using a learning algorithm that processes a high-definition image in real time.

It is a figure explaining the block diagram of a convolutional neural network, and is a block diagram of a convolutional neural network which performs deblurring restoration processing for focus correction. It is a block diagram explaining the nonlinear enhancer process in 1-pass video super-resolution. It is a block diagram explaining the deterioration process of the image (video) for explaining a Wiener filter. It is a figure which shows operation|movement of a nonlinear enhancer process. FIG. 6 is a block diagram of a convolutional neural network that performs deblurring processing for focus correction. The Kodak color evaluation image (Kodak Lossless True Color Image Suite, http://r0k.us/graphics/kodak/) used for learning is shown. (a) shows the learning curve (enlarged view of the entire upper right), and (b) shows the average ISNR of the restoration results of the training image (training) and the evaluation image (test) with respect to the number of iterations. FIG. It is a figure explaining the example of the deblurring process result of the image for evaluation by a learning parameter, and is a Gaussian smoothing input image (sigma=1.0), the restoration image by a deblurring process, and a true image (output expectation image) from the left. (a) is a visualization of one input convolution weight parameter of the feature map in the learning parameter as an image. The parameter images in all feature maps are almost the same, but the actual sizes are different (feature (It looks the same when normalized by the maximum and minimum values of the parameters for each map.), (b) is a three-dimensional plot of its two-dimensional frequency characteristics (calculated from the results of normalization with the sum of parameters) .. Image for evaluation against Gaussian smoothing (σ=0.8 to 1.2) (Kodak Lossless True Color Image Suite, http://r0k.us/graphics/kodak/) This is the average ISNR [dB] of 18 restoration results and pixel The results of the normal noise level σ _N added to the values of 0.5 and 1.0 are also shown. It is a figure explaining the sine wave half-wave rectified by the ReLU activation function (reference [5, 14]).

(Outline of Invention 1)
4K/8K (Super Hi-Vision) Deblurring is performed by a convolutional neural network, which is a learning algorithm, for the purpose of correcting defocus in ultra-high definition images. Generates a smoothed input image assuming a focus shift from a true image (output expected image). The parameters of the network are estimated (learned) in advance so that the sum of squared differences between the restored image obtained by processing the smoothed input image by the network and the true image is minimized using such input/output images as learning data. .. The processing by the convolutional neural network using the parameters of the learning result can be implemented by CPU, GPU, and FPGA, and compared with the conventional method, the processing cost is low, the low frame delay by the local area processing, the high deblurring performance, the noise. Achieve resistance.

As described above, the deblurring restoration process by the convolutional neural network is an optimal nonlinear enhancer in the sense of least squares in the learning image, and realizes the deblurring precision comparable to the Wiener filter. Despite the local area processing, it is a good approximation of the Wiener filter, and is more robust to the noise contained in the image (video) than the Wiener filter. The convolutional neural network can be realized by a CPU, GPU, and FPGA, and is an effective method for performing real-time processing on high-definition video.

(Outline of Invention 2)
The present invention performs image (video) restoration processing using a convolutional neural network that is a learning algorithm. In addition, the network parameters for generating the expected output image are learned by prior learning. In determining the parameter, the parameter that maximizes the average ISNR in another evaluation image different from the learning image is used. In addition, by using an evaluation image different from such a learning image, it is possible to prevent over-learning of parameters that are excessively adapted to the learning image, and even in an image (video) other than the learning image, high deblurring performance. It is possible to obtain the generalization ability to realize.

The input convolutional layer (feature map) that forms the convolutional neural network, the non-linear activation function that follows the input convolutional layer (feature map) that forms the convolutional neural network, and the input convolutional layer (feature map that forms the convolutional neural network). ) And an output layer that integrates the results of the non-linear activation function, and an image and a learning method as learning data for estimating the parameters of each layer of the convolutional neural network. In addition, the method is such that the parameters are determined so that the average ISNR in another evaluation image different from the learning image for determining the parameters of each layer of the convolutional neural network is maximized.

As an implementation method, it can be implemented by a hardware device that processes a baseband video signal, or can be implemented by a software that processes an MXF file and a computer-based device that executes the software. However, by using a device that converts an MXF file into a baseband video signal or an inverse conversion, it can be realized by any configuration. It is also possible to perform processing on the cloud by transmitting a camera image compressed as a moving image or transmitting an MXF file by IP (Internet Protocol). Various system forms are conceivable, such as decoding a compressed image transmitted by IP into a baseband video signal, compressing the result of focus correction processing again, and distributing the stream.

Multilayer the network structure by adding the number of input convolutional layers (feature maps) in the convolutional neural network, multiple hidden layers connecting each layer between the input convolutional layer and the output layer, and a nonlinear activation function. As a result, improvement in deblurring restoration accuracy is expected. By preparing a learning image to which noise that is supposed to be included in an image (video) is added, it is expected that the learning image also has noise removal capability.

FIG. 1 is a diagram illustrating a block diagram of a convolutional neural network, which is a block diagram of a convolutional neural network that performs deblurring restoration processing for focus correction. Convolution processing is performed on the input image g _i,j with a kernel having a (2L+1)×(2L+1) pixel block size. Non-linear level operations are performed on the results of such M sets of input convolutional layers (feature maps) by non-linear activation functions. The final output image is the output clipped result of the output layer result obtained by weighted addition of the results of the nonlinear activation function.

And

As a comparison of the deblurring process by the convolutional neural network, the nonlinear enhancer process used for the 1-pass video super-resolution and the Wiener filter known as the optimum restoration filter in the meaning of least squares are shown below.

FIG. 2 is a block diagram illustrating a non-linear enhancer process in 1-pass video super-resolution. The case of one dimension will be described. An edge component of the input signal is detected by a DoG (Difference of Gaussian) filter, a harmonic component is restored by a non-linear operation regarding the level, and the edge component is added to the input signal. In order to suppress excessive emphasis, adaptive clip processing that searches for the maximum and minimum values of pixels in the input neighborhood area and sets the clip level is also used. In addition, FIG. 3 is a block diagram illustrating a deterioration process of an image (video) for explaining the Wiener filter.

Such that

The filter to find. Considering in the frequency domain,

here,

Is

Is the complex conjugate of, and for the adjustment function

Is.
Also,

Are the noise component and the power spectral density of the true image, respectively.

(Detailed description: Focus correction by convolutional neural network for ultra high definition video)

(Abstract)
4K/8K (Super Hi-Vision (Reference [15])) Performs deblurring by a convolutional neural network for the purpose of correcting defocus in ultra-high definition video. We evaluate the restoration performance and noise resistance of deblurring processing by a convolutional neural network. Furthermore, we compare the results with the nonlinear enhancer processing in 1-pass video super-resolution (reference [12]) and the Wiener filter (reference [21]) that minimizes the squared error from the true image.

(1 Introduction)
4K test broadcasting as a next-generation television broadcasting was started on CS (Communication Satellite) and cable television from June 2, 2014 (Next Generation Broadcasting Promotion Forum (NexTV-F), http://www .nextv-f.jP/). 8K (including Super Hi-Vision (reference [15]) is accelerating towards the start of practical broadcasting in 2018 (as soon as possible) (Ministry of Internal Affairs and Communications "Follow-up meeting on 4K/8K roadmap (6th Meetings) Handouts", July 2015, http://www.soumu.go.jp/main_sosiki/kenkyu/4k8kroadmap/02ryutsu11_03000046.html).

Resolution conversion is required for repurpose of HD contents in 4K/8K broadcasting. Recently, super-resolution technology has been actively researched (reference [16]). Although much of the processing is iterative, the inventor has proposed a one-pass video super-resolution with weighted averaging of local temporal and spatial interpolation of images and a multi-scaled nonlinear enhancer (reference [[ 12]). Zhao and Matsunaga (reference [23]) have accelerated the 1-pass video super-resolution processing with a GPU.

Focus adjustment is strictly required for shooting 4K/8K ultra-high-definition video, but due to the high resolution of the video, the optical size is large, the pixel size of the image sensor is small, and the depth of field becomes shallow. , Focus adjustment is much more difficult. It is not uncommon to find out of focus after shooting.

Funatsu et al. (reference [6]) have developed a low-resolution viewfinder that superimposes signals by multiplexing signals that assist focus adjustment in order to improve the visibility of the low-resolution viewfinder image for image confirmation. We proposed a method that enables focus adjustment in high-definition video.

In the present invention, deblurring processing by a convolutional neural network is performed for the purpose of correcting the focus shift in 4K/8K ultra-high definition video. We point out the similarity between deblurring by convolutional neural networks and nonlinear enhancer processing in 1-pass video super-resolution, and perform ReLU (Rectified Linear Unit) activation function that performs nonlinear operations on levels in convolutional neural networks (reference [5 , 14]) and the Fourier series expansion of the processing result reveals that harmonic components are generated. Then, the restoration performance and noise resistance of the deblurring process by the convolutional neural network are evaluated. Furthermore, we compare the results with the nonlinear enhancer processing in 1-pass video super-resolution (reference [12]) and the Wiener filter (reference [21]) that minimizes the squared error from the true image.

In recent years, convolutional neural networks (references [5, 10]) have regained attention as deep learning, but neural networks have a long history as models of information processing in the cranial nervous system, and McCulloch and We can go back to the formal neuron by Pitts (ref. [13]) and the preceptron by Rosenblatt (ref. [18]). Rumelhurt et al. (reference [19]) rediscovered the error backpropagation method (backpropagation) as a learning rule for multilayer perceptrons, which spread explosively in the 1980s (perceptrons with hidden layers due to Amari). A learning rule had already been proposed (reference [1]).The convolutional neural network that is standardly used in image recognition by deep learning (reference [9]) was also used at the NHK Broadcasting Science Laboratories (currently This is the neo-cognitron by Fukushima (reference [5]) who was a member of the NHK Broadcasting Technology Research Laboratories itself.. The ReLU activation function (reference [14] that the learning is performed faster than the sigmoid function). ]) has also been used (not called ReLU).LeCun et al. [10] used a convolutional neural network for backpropagation (backpropagation) to recognize handwritten postal codes. ).The contribution of Japanese researchers in the early research of neural networks should be more recognized.).

Although deep learning using convolutional neural networks has been actively researched for image recognition (reference [9]), it is also used for image processing such as denoising, deblurring, and super-resolution (reference [7, 4, 3, 22]). The deblurring process has a long history as an image restoration process in various fields (for details on astronomical images (reference [20])), there are many studies, but the reconstruction process is iterative. (Reference [2]).

The configuration of the present invention will be described in Chapter 2, the non-linear enhancer processing in 1-pass video super-resolution, and in Chapter 3, the configuration and learning method of the convolutional neural network will be described respectively, and in Chapter 4, the results of the image simulation will be shown. It will be summarized in Chapter 5.

(2 Non-linear enhancer in 1-pass video super-resolution)
The present inventor further improves the resolution by performing the nonlinear enhancer processing based on the edge information of the image as the post-processing of the result of the resolution conversion processing by the intra-frame spatial directional interpolation as the 1-pass video super-resolution. (Reference [12]). FIG. 4 shows the operation of the nonlinear enhancer processing.

A Gaussian difference (Difference of Gaussian, DoG) filter is used for edge detection. The processing kernel of the Gaussian smoothing filter to calculate the Gaussian difference,

Then, the DoG filter of image I(x) is

(However, in the case of one dimension). Here, * is a convolution operation, and σ1<σ2 (assuming σ1→0,

Therefore, adding a value obtained by appropriately gaining the result of Expression (2) to the original signal corresponds to so-called "Unsharp Masking". ). The DoG filter is a good approximation of a Laplacian (Laplacian of Gaussian, LoG) filter which is the second derivative of the Gaussian smoothing filter, and has high calculation efficiency. In the case of an image, it can be processed separately in the horizontal and vertical directions. Like the Laplacian filter, edge detection independent of direction is possible.

The edge component detected by the DoG filter is added to the original image by expanding the high-frequency component by a non-linear operation related to the level, but here, in order to suppress excessive emphasis due to the non-linear operation, the input pixel value in the vicinity of the pixel of interest is suppressed. The maximum value and the minimum value are searched for, and the adaptive clip processing is performed using those values. As a non-linear operation regarding the level, for example,

Here, sgn(•) is a sign function, and r is a constant of 2 or more. The present inventor further multi-scale extends such a non-linear enhancer (reference document [12]) (details omitted).

(3 convolutional neural network)
FIG. 5 is a block diagram of a convolutional neural network that performs deblurring processing for focus correction. The convolutional neural network has a minimum configuration of two layers. The layers of the convolutional neural network are as follows.

here,

Are the pixel values of the input and output images, and the activation function

Is as follows:

Xmax in the above formula (8) represents the maximum value. Then, the parameters of each layer that minimize the following objective function J are estimated.

here,

Is a pixel value in a true image expected as an output image. The slope of the parameters of each layer with respect to J and the derivative of the activation function are shown in Appendix A.

The activation function φ(x) is known as ReLU (references [5, 14]), but like the non-linear enhancer, the non-linear operation regarding the level plays an important role in the restoration of the high frequency component. The ReLU activation function that clips the negative component is equivalent to a "half-wave rectifier (diode)", and it can be seen that a harmonic component is generated in the half-wave rectified sine wave (Appendix B reference).

Serialize all the parameters whose values should be determined (2L+1) (2L+1)M+M+1-dimensional vector

Is defined as follows.

A certain initial value u ^(o) is determined, and u is determined by the following stochastic gradient descent method (reference [11]).

Where λ is a small learning coefficient. this

Repeat until When L=3 and M=8, the total number of parameters is 7×7×8+8+1=401. As the learning image, a true image is used as an output expected image, and a Gaussian-smoothed image corresponding to defocus is used as an input image. In actual learning, an appropriate number of partial images at the same position between the input and output images is randomly extracted and used for each iteration (mini-batch learning (reference [11]).

In order to accelerate the parameter update, the following momentum method (references [19, 11]) may be used.

The momentum coefficient μ is set to 0≦μ<1. This can be viewed as a recursive filter on the parameters.

In order to stabilize the learning, it is advisable to decrease the learning coefficient λ exponentially with the number of iterations. For example, when the number of iterations is 100 or more, the initial learning coefficient λ ₀ is set to 1/10, and when the number of iterations is 10,000 or more, 1/10 of the initial learning coefficient λ ₀ is exponentially reduced by the number of iterations. In order to obtain λ ₀ /lOO with 100,000 iterations,

As

And it is sufficient.

(4 image simulation)
Image simulation is performed to deblurr the Gaussian smoothed input image using a convolutional neural network. A true image is used as an output expected image, and a smoothed image with a σ=1.0 Gaussian filter is used as an input image. For each iteration, 256 partial images (33 × 33 pixel size) at the same position between the input and output images were randomly extracted for each image and used for learning (mini-batch learning (reference [11]). , Kodak color evaluation images (Kodak Lossless True Color Image Suite, http://r0k.us/graphics/kodak/) used for learning are shown.Only G (green) images are used in the experiment. The network configuration is M=8 feature maps, and the input convolution kernel size is 7×7 pixel size (L=3) The initial parameters for learning are the input convolution weight parameters.

Is mean O, normal random number with standard deviation 0.01, feature map weight parameter

Is a uniform random number of [0.0, 0.1) and the bias term b=0.0. With the initial learning coefficient λ ₀ =8×10 ⁻⁶ and the momentum coefficient μ=0.9, the learning coefficient λ was controlled by the number of iterations of the equation (14).

Further, as described above, FIG. 6 shows a learning image (Kodak Lossless True Color Image Suite, http://r0k.us/graphics/kodak/). The true image is used as the output expected image, and the smoothed image with the Gaussian filter with a=1.0 is used as the input image. For each iteration, 256 partial images (33 × 33 pixel size) at the same position between the input and output images were randomly extracted for each image and used for learning (mini-batch learning (reference [11])). Only G (green) images were used in the experiment. The frame in the image represents the size of the partial image used for learning.

Further, FIG. 7(a) shows a learning curve (enlarged display of the entire state on the upper right side), and FIG. 7(b) shows the results of restoration of the learning image (training) and the evaluation image (test) with respect to the number of iterations. 4 is a diagram illustrating the average ISNR [dB] of FIG. In each case, the horizontal axis is the number of iterations and is on a logarithmic scale. Error bars are standard deviations. The average ISNR of the restoration result of the evaluation image was the maximum at 98,500 iterations.

As described above, FIG. 7A is a plot of the learning image residual J (equation (10)) against the number of iterations (learning curve). Then, FIG. 6B shows a learning image for the number of iterations, a training image (training), and an ISNR between the restoration result of the evaluation image (test) different from the learning image and the true image. Improvement in SNR) is plotted. As the evaluation images, 18 of the 24 Kodak color evaluation images (Kodak Lossless True Color Image Suite, http://r0k.us/graphics/kodak/) other than the 6 learning images were used. .. ISNR measures the degree of improvement of the SN ratio due to restoration processing, and is calculated as follows (reference [2]).

here,

Is a true image,

Is the Gaussian smoothed input image,

Is a deblurred image.

The average ISNR of the learning image and the evaluation image both gradually increase as the learning progresses, but the average ISNR of the evaluation image then starts to decrease. This is likely due to "over-learning" as the residuals in the training images are decreasing. Therefore, the parameter that maximizes the average ISNR in the evaluation image is used as the final learning result. In FIG. 7B, the average ISNR of the restoration results of the evaluation image was the maximum at the number of iterations of 98,500. FIG. 8 is an example of a deblurring result of an evaluation image using such a learning parameter. From the left, the Gaussian smoothed input image (σ=1.0), the restored image by deblurring, and the true image (output expected image) are shown. The ISNR of the deblurred image was 5.33/5.54 [dB]. The lower part of the figure is a binarized image of frequency components by the FFT processing (threshold value 100). As for the result of the deblurring process by the convolutional neural network for the Gaussian smoothed input image in which the high frequency component is limited, it can be seen that the high frequency component is restored.

FIG. 9A shows one input convolution weight parameter of the feature map in the learning parameter.

Is visualized as an image. In any feature map

The parameter images are almost the same, but the actual sizes are different (they look the same when normalized by the maximum and minimum values of the parameters for each feature map). FIG. 3B is a three-dimensional plot of the two-dimensional frequency characteristic (calculated from the result of normalization with the sum of parameters). It can be seen from the frequency characteristics that they are high-frequency emphasis filters, but the respective high-frequency emphasis gains are different. The result in which the high frequency band is emphasized in each feature map is clipped by the ReLU activation function as the negative component, and the result of the weighted addition is clipped as the output to be the final output. It is considered that the high-frequency component is restored by integrating the clip processing results of high-frequency emphasis with a plurality of different gains. Depending on the learning result, the input convolution weight parameter in the feature map

Feature map weight parameter

Then, almost 0 existed. The parameter initial values are generated by random numbers, and the learning image is randomly extracted at each iteration, so it is considered that correct learning was not performed.

Next, the restoration performance of the deblurring process of the convolutional neural network in the learned parameters and the noise resistance are evaluated. For a convolutional neural network that learned a smoothed image by a Gaussian filter with σ=1.0, we evaluate the restoration result of a Gaussian smoothed image with σ varied from 0.8 to 1.2 in 0.1 steps. Further, a smoothed image containing no noise was used for learning, but an actual image usually contains compression noise and imaging noise. Therefore, the restoration accuracy when normal noise is added to the pixel value is also evaluated. The normal noise has an average of 0, and the standard deviation σ _N is 0.5, 1.0. Non-linear enhancer in 1-pass video super-resolution (reference [12]) and Wiener filter (reference [21]) are also used for restoration processing. The Wiener filter is shown in Appendix C.

The nonlinear enhancer uses a DoG filter for edge detection with σ=0, and uses a cubic function for the nonlinear operation regarding the level. Without using an adaptive clip, the clip level was optimized with the enhancer gain γ and the DoG filter σ ₂ as adjustment parameters so that the average ISNR in the learning image was maximized. The downhill simplex method (Nelder-Mead method) (reference [17]) was used for the optimization. For the Wiener filter, the point spread function is a Gaussian smoothing filter with σ=1.0, and the intensity ratio parameters of the true image and the power spectrum of the noise component are determined so that the average ISNR in the training image is maximized. Table 1 shows the restoration result for the normal noise σ _N added to the Gaussian smoothed image (σ=1.0), and the evaluation image (Kodak Lossless True Color Image Suite, http://r0k.us/graphics/kodak/) The average ISNR [dB] of 18 images is shown (standard deviation in parentheses).

The Wiener filter is optimal in the sense of least squares, and is strictly restored when the point spread function is true and there is no noise. Affect. Fig. 10 shows the average ISNR [dB] of 18 restored images (Kodak Lossless True Color Image Suite, http://r0k.us/graphics/kodak/) for evaluation against Gaussian smoothing (σ = 0.8 to 1.2). Is. The results of the normal noise level σ _N added to the pixel value being 0.5 and 1.0 are also shown. Error bars are standard deviations. When σ _N =0.0 and Gaussian smoothing σ=1.0, the restoration accuracy by the Wiener filter in FIG. 7C is maximum, but when Gaussian smoothing σ changes, the restoration accuracy decreases. The restoration accuracy decreases as the noise level σ _N increases to 0.5 and 1.0. It can be seen that the fluctuation of the restoration result is large and the resistance to fuller and noise is low.

The result obtained by the convolutional neural network in FIG. 9A also has the maximum restoration accuracy when σ _N =0.0 and Gaussian smoothing σ=1.0. The results for blur and noise show almost the same tendency as that of the Wiener filter, but it is found to be more resistant and less variable than the Wiener filter. On the other hand, it is hard to say that the result of the nonlinear enhancer in FIG. 6B has sufficient restoration accuracy. However, although the restoration accuracy is low, the resistance to blur and noise is highest.

The deblurring process by the convolutional neural network has a restoration accuracy comparable to that of the Wiener filter, and is a good approximation of the Wiener filter despite being the local region processing, and is more robust against the fuller and the noise. ..

(5 summary)
We performed deblurring with a convolutional neural network for the purpose of correcting the focus shift in 4K/8K ultra high definition images. We point out the similarity between the deblurring process by the convolutional neural network and the non-linear enhancer process in the 1-pass video super-resolution, and perform the Fourier series expansion of the processing result of the ReLU activation function that performs the non-linear operation on the level in the convolutional neural network. By doing so, it was clarified that a harmonic component was generated. Then, the restoration performance and noise resistance of the deblurring process by the convolutional neural network were evaluated. Furthermore, the results are compared with the results obtained by the optimal Wiener filter in the sense of nonlinear enhancer processing and least squares in 1-pass video super-resolution.

The deblurring process by the convolutional neural network is to restore the high frequency component because the edge component detected by the input convolution filter is subjected to the non-linear operation with respect to the level by the ReLU activation function, like the non-linear enhancer process in the one-pass video super-resolution. Therefore, it can be said that it is an optimal nonlinear enhancer in the sense of least squares in the training image.

Future issues include optimization of network configuration and improvement of restoration performance and noise resistance by deepening layers, use of GPU to accelerate learning, and real-time processing of 4K/8K video by FPGA implementation. Convolutional neural networks are probably the most effective method at present for real-time processing of optimization results by prior learning in 4K/8K ultra-high definition video.

(References)
[1] S. Amari, A theory of adaptive pattern classifiers, IEEE Transactions on Electronic Computers, EC-16-3 (June 1967), 299-307.
[2] J. Biemond, RL Lagendijk, and RMMersereau, Iterative methods for image deblurring, Proceedings of the IEEE, 78-5 (May 1990), 856-883.
[3] C. Dong, C.-C. Loy, K. He, and X. Tang, Learning a deep convolutional network for image superresolution, Proceedings of 13th European Conference on Computer Vision (ECCV2014), Part IV, Zurich, Switzerland , pp. 184-199 (September 2014).
[4] D. Eigen, D. Krishnan, and R. Fergus, Restoring an image taken through a window covered with dirt or rain, IEEE International Conference on Computer Vision (ICCV2013), Sydney, Australia, pp. 633-640 (December 2013).
[5] K. Fukushima, Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position, Biological Cybernetics, 36-4 (April 1980), 193-202.
[6] Ryohei Funazu, Yoshiyuki Yamashita, Koji Mitani, Yuji Nojiri, Focus Auxiliary Signal for Super Hi-Vision Camera, Journal of Image Information Media, 65-4 (April 2011), 531-539
[7] V. Jain and HS Seung, Natural image denoising with convolutional networks, Proceedings of Advances in Neural Information Processing Systems 21 (NIPS2008), pp. 769-776 (2008).
[8] Ken'ichi Kanaya, “Applied Mathematics Classes Understood by This: From Least Squares to Wavelets”, Kyoritsu Shuppan, June 2003.
[9] A. Krizhevsky, I. Sutskever, and GE Hinton, ImageNet classification with deep convolutional neural networks, Proceedings of Advances in Neural Information Processing Systems 25 (NIPS2012), pp. 1106-1114 (2012).
[10] Y. LeCun, B. Boser, JS Denker, D. Henderson, RE Howard, W. Hubbard, and LD Jackel, Backpropagation applied to handwritten zip code recognition, Neural Computation, 1-4 (December 1989), 541- 551.
[11] Y. LeCun, L. Bottou, G. Orr, and K. Miiller, Efficient BackProp, In G. Orr and K. Miiller (Eds), Neural Networks: Tricks of the trade, Springer, 1998.
[12] Riki Matsunaga, 1-pass video super-resolution using spatiotemporal directional interpolation and multiscale nonlinear enhancer, Proc. of the 20th Image Sensing Symposium (SSII2014), Yokohama (Pacifico Yokohama), June 2014.
[13] WS McCulloch and W. Pitts, A logical calculus of the ideas immanent in nervous activity, The bulletin of mathematical biophysics, 5-4 (December 1943), 115-133.
[14] V. Nair and GE Hinton, Rectified linear units improve restricted Boltzmann machines, Proceedings of the 27th International Conference on Machine Learning (ICML10), Haifa, Israel, pp. 807-814 (June 2010).
[15] Japan Broadcasting Corporation, Special Issue on "Super Hi-Vision Video Technology", NHK STRL R&D, No.137, January 2013 [16] SC Park, MK Park, and MG Kang, Super-resolution image reconstruction: A technical overview , IEEE Signal Processing Magazine, 20-3 (May 2003), 21-36.
[17] WH Press, SA Teukolsky, WT Vetterling, and BP Flannery, Numerical Recipes 3rd Edition: The Art of Scientific Computing, Cambridge University Press, September 2007.
[18] F. Rosenblatt, The Perceptron: A probabilistic model for information storage and organization in the brain, Psychological Review, 65-6 (1958), 386-408.
[19] DE Rumelhart, JL McClelland, and CORPORATE PDP Research Group, Parallel Distributed Processing: Explorations in the Micro structure of Cognition, Vol. 1: Foundations, Vol. 2: Psychological and Biological Models, MIT Press, Cambridge, MA, USA , July 1986, July 1987. Shunichi Amari (Translated), "PDP Model-Cognitive Science and Search for Neuron Networks", Industrial Book, March 1989 [20] J. -L. Starck, and F. Murtagh, Astronomical Image and Data Analysis, Springer, 2006.
[21] N. Wiener, Extrapolation, interpolation, and smoothing of stationary time series: with engineering applications, John Wiley & Sons. Inc., New York, 1949.
[22] L. Xu, JS Ren, C. Liu, and J. Jia, Deep convolutional neural network for image deconvolution, Proceedings of Advances in Neural Information Processing Systems 27 (NIPS2014), pp. 1790-1798 (2014).
[23] Zhao Yanjun, Riki Matsunaga, Acceleration of 1-pass video super-resolution up conversion for MXF files by GPU, GPU Technology Conference (GTC Japan 2014), 2014-8008, Tokyo (Tokyo Midtown Hall & Conference) ), July 2014.

(Appendix A Convolutional Neural Network Gradient)
The gradient of the objective function J of the equation (10) for each layer of the convolutional neural network is as follows.

Where the activation function

The derivative of is as follows.

(Appendix B Fourier series expansion of ReLU activation function processing results)
The periodic function f(t) repeated in the period T can be expanded into a Fourier series by the following trigonometric function (reference [8]).

Here, the angular frequency ω=2π/T, and the coefficients a _k and b _{k on the} right side are as follows.

The sine wave half-wave rectified by the ReLU activation function (references [5,14]) is

And when the Fourier coefficient is calculated,

Here, FIG. 8 is a diagram illustrating a sine wave half-wave rectified by the ReLU activation function (reference document [5, 14]). It can be seen that even-order harmonic components are generated in the sine wave half-wave rectified by the ReLU activation function.

(Appendix C Wiener filter)
Observation image g(x,y) is a true image

Is deteriorated by the point spread function h(x,y), and the noise component n(x,y) is added as follows.

Here, * is a convolution operation. In the case of defocus, h(x,y) is approximated by a two-dimensional Gaussian function.
In the frequency domain,

And the true image from the observed image G (u,v)

To

The Wiener filter W (u,v) that is estimated as

Is. Here, H ^* represents a complex conjugate of H.

_Where S _n and S _f are the noise component and the power spectral density of the true image, respectively.
K(u,v) is determined from the true image and the noise component, and an approximate one may be known, but it is usually an adjustment parameter specified as an empirical constant. When K=0, W(u,v)=1/H(u,v), which is an inverse filter. For large u and v,

Then, the high frequency component is suppressed. Wiener filter is the squared error from the true image

Is a filter that minimizes (reference [21]).

The present invention is also suitable for 4K/8K (Super Hi-Vision) ultra high definition images.

Claims (1)

In the focus correction processing method by the learning type algorithm,
In order to use the convolutional neural network that is the learning algorithm for deblurring restoration processing,
In order to generate a smoothed input image assuming defocus from the true image, such input/output images are used as learning data, and the restored image obtained by processing the smoothed input image by the network and the true image. In the step of estimating the parameters of the network such that the sum of squared differences is minimized, the learning parameter is determined so that the average ISNR of deblurring restoration results in another evaluation image different from the learning image is maximized. And the process,
And a focus correction process using the convolutional neural network based on the determined learned parameters.