KR102543451B1 - Image feature extraction and synthesis system using deep learning and its learning method - Google Patents

Abstract

The present invention relates to a system for extracting and synthesizing features of an image using deep learning and a method for learning the same. The system for extracting and synthesizing image features of the present invention includes a style encoder ( 200); the geometry encoder 100 extracting the geometry vector 110 from the second image 20; and a synthesizing decoder 300 generating a synthesized image 310 by synthesizing the geometry vector 110 and the style vector 210.

Description

Image feature extraction and synthesis system using deep learning and its learning method {Image feature extraction and synthesis system using deep learning and its learning method}

The present invention relates to a system for generating a new composite image by extracting features of a plurality of images, respectively, using deep learning, and a method for learning components constituting the system.

In addition, one of the technical features of the present invention may be to independently extract and synthesize geometry information and style information of a person image.

Patent Document 001 relates to a system and method for generating a virtual person through face image search, and the method for generating a virtual person through face image search according to the present invention is a virtual character face of various appearances based on an artificial intelligence technique by a computer system. generating an image and storing it in an image database; receiving a 2D image from the outside by an image search unit and searching an image database in which a face image of a virtual character is stored; extracting similar images from an image database through face landmark-based similarity comparison by an image search unit; and receiving, by a virtual person creation unit, the image extracted by the image search unit and generating a new virtual person based thereon.

Patent Document 002 discloses an image correction method and correction device. The image correction method includes image correction through application of filters, storage of filter information, retraining of a deep neural network model using the filter information, and output of filter information using the deep neural network model. According to the present invention, a technology capable of image correction based on image analysis using an artificial intelligence (AI) model through a 5G network is presented.

Patent Document 003 relates to an automatic image generation device and method, wherein the image automatic generation device includes at least one processor, and the at least one processor generates a second image of a different domain based on a first image. A technique of generating a second image, extracting preset parameters from the generated second image, and generating a third image based on the extracted parameter is proposed.

Patent Document 004 relates to an artificial intelligence device and method for synthesizing images, a memory for storing image data, obtaining a person object image and object object image from the image data, and a 3D person model matched with the person object image. generating, extracting coordinate information of image data, acquiring street view data matching the coordinate information, extracting reference object images in which the object object of the object object image is photographed from the street view data, and extracting a person object image and object object We present an artificial intelligence device including a processor that acquires image placement information and places a 3D person model on a street view image of street view data using the placement information and reference object image.

KR 10-2022-0011100 (Date of publication: January 27, 2022) KR 10-2021-0054907 (published date: May 14, 2021) KR 10-2021-0062274 (published date: May 31, 2021) KR 10-2021-0078813 (published date: 29 June 2021)

The present invention relates to a system for extracting and synthesizing features of images using deep learning and a method for learning the same.

In addition, it is intended to secure the completeness of the system by providing a more efficient learning method by sequentially learning each configuration of the system of the present invention.

In addition, in learning a plurality of encoders and decoders, each component of the present invention, a method of independently extracting geometry information and style information of a learning person image is performed to secure the object and effect of the present invention.

An invention according to an embodiment of the present invention is an invention for an image synthesis system for generating a virtual image through feature extraction and synthesis of images using deep learning, a style encoder for extracting a style vector from a first image; 2 A geometry encoder that extracts a geometry vector from the image; and a synthesizing decoder generating a synthesized image by synthesizing the geometry vector and the style vector.

An invention according to an embodiment of the present invention relates to a method for learning the above-mentioned invention, and the geometry encoder generates a first learning geometry vector from a first sketch extracted from the first image by the sketch encoder (a) doing; and (b) learning the geometry encoder in a direction to reduce a first error, which is an error between the first learning geometry vector and the geometry vector.

An invention according to an embodiment of the present invention is an invention for a learning method of an image synthesis system that generates a virtual image through image feature extraction and synthesis using deep learning, and in the learning method presented above, the above (a) Steps include: (a-1) generating, by a sketch decoder, a transformation sketch for first training from the first geometry vector for training; (a-2) generating, by a sketch decoder, a transformation sketch for second training from the geometry vector; And the step (b) includes (b-1) learning the geometry encoder in a direction that reduces a second error, which is an error between the first transformation sketch for learning and the second transformation sketch for learning; made up of

An invention according to an embodiment of the present invention is an invention for a learning method of an image synthesis system that generates a virtual image through image feature extraction and synthesis using deep learning, and in the learning method presented above, the above (a) and (b) are repeatedly performed to learn the geometry encoder.

An invention according to an embodiment of the present invention is an invention for a learning method of an image synthesis system that generates a virtual image through image feature extraction and synthesis using deep learning, and in the learning method presented above, the sketch encoder and The sketch decoder (c) generating a second learning geometry vector from the person sketch 12 extracted from the person image by the sketch encoder; (d) the sketch decoder performing a third learning transformation from the second learning geometry vector creating a sketch; and (e) learning the sketch encoder and the sketch decoder in a direction of reducing a third error (3), which is an error between the character sketch and the third conversion sketch for learning.

An invention according to an embodiment of the present invention is an invention for a learning method of an image synthesis system that generates a virtual image through image feature extraction and synthesis using deep learning, and in the learning method presented above, the style encoder and The synthesis decoder includes: (f) the style encoder generating a style vector for first training from the synthesized image; (g) the geometry encoder generating a third geometry vector for training from the first image; ( h) synthesizing, by the synthesis decoder, the first learning style vector and the third learning geometry vector to generate a first learning synthesis image; and (i) learning the style encoder and the synthesis decoder in a direction to reduce an error between the first synthetic image for learning and a fourth error, which is an error between the first image.

An invention according to an embodiment of the present invention is an invention for a learning method of an image synthesis system that generates a virtual image through image feature extraction and synthesis using deep learning, and in the learning method presented above, the above (f) and (i) are repeatedly performed to learn the style encoder and the synthesis decoder.

An invention according to an embodiment of the present invention is an invention for a learning method of an image synthesis system that generates a virtual image through image feature extraction and synthesis using deep learning, and in the learning method presented above, the style encoder and The synthesis decoder includes: (j) the geometry encoder generating a fourth learning geometry vector from the synthesized image; (k) the style encoder generating a second learning style vector from the second image; ( l) synthesizing, by the synthesis decoder, the fourth learning geometry vector and the second learning style vector to generate a second training synthesis image; and (m) learning the style encoder and the synthesis decoder in a direction of reducing a fifth error, which is an error between the second synthesized image for training and the second image.

An invention according to an embodiment of the present invention is an invention for a learning method of an image synthesis system that generates a virtual image through image feature extraction and synthesis using deep learning, and in the learning method presented above, the above (j) and (m) are repeatedly performed to learn the style encoder and the synthesis decoder.

An invention according to an embodiment of the present invention is an invention for a learning method of an image synthesis system that generates a virtual image through image feature extraction and synthesis using deep learning, and in the learning method presented above, the style encoder and The synthesizing decoder includes: (n) the geometry encoder generating a fourth learning geometry vector from the synthesized image; (o) the style encoder generating a second learning style vector from the second image; ( p) synthesizing, by the synthesis decoder, the fourth learning geometry vector and the second learning style vector to generate a second training synthesis image; and (q) learning the style encoder and the synthesis decoder in a direction of reducing a fifth error, which is an error between the second synthesized image for learning and the second image.

An invention according to an embodiment of the present invention is an invention for a learning method of an image synthesis system that generates a virtual image through image feature extraction and synthesis using deep learning, and in the learning method presented above, the style encoder and The synthesis decoder (r) setting the first image and the second image to a third image that is the same image; (s) generating a third learning style vector from the third image by the style encoder; , (t) the geometry encoder generating a fifth learning geometry vector from the third image;, (u) the synthesizing decoder synthesizing the third learning style vector and the fifth learning geometry vector for third learning generating a composite image; and (v) learning the style encoder and the synthesis decoder in a direction of reducing a sixth error between the third synthesized image for learning and the third image.

The present invention provides a system capable of generating a single composite image by extracting features of a plurality of images and a learning method therefor, thereby generating a virtual image containing features desired by a user.

In addition, the present invention can provide an efficient learning method by suggesting a method of sequentially learning components of an image synthesis system.

In addition, in the process of generating a composite image, a new composite image may be created through user's artificial correction.

1 is a conceptual diagram illustrating synthesis examples of multiple person images according to an embodiment of the present invention.
2 is an exemplary view of a person image and a person sketch extracted therefrom according to an embodiment of the present invention.
3 is a configuration diagram of an image synthesis system according to an embodiment of the present invention.
4 is a conceptual diagram for learning a geometry encoder according to an embodiment of the present invention.
5 is a learning conceptual diagram of a sketch encoder and a sketch decoder according to an embodiment of the present invention.
6 is a learning conceptual diagram of a style encoder and a synthesis decoder according to another embodiment of the present invention.
7 is a configuration block diagram of a computer device that can be used in an image synthesis system and learning method according to an embodiment of the present invention.

Advantages and features of the present invention, and methods of achieving them, will become clear with reference to the detailed description of the embodiments taken in conjunction with the accompanying drawings. However, it should be understood that the present invention is not limited to the embodiments presented below, but may be implemented in a variety of different forms, and includes all conversions, equivalents, and substitutes included in the spirit and scope of the present invention. .

The embodiments presented below are provided to make the disclosure of the present invention complete, and to fully inform those skilled in the art of the scope of the invention to which the present invention belongs. In describing the present invention, if it is determined that a detailed description of related known technologies may obscure the gist of the present invention, the detailed description will be omitted.

Terms used in this application are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, the terms "include" or "have" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that the presence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded.

In an embodiment of the present invention, "communication", "communication network" and "network" may be used in the same meaning. The above three terms may include wired and wireless local and wide area data transmission and reception networks capable of transmitting and receiving files between user terminals, terminals of other users, and download servers.

Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, the same or corresponding components are assigned the same reference numerals, and overlapping descriptions thereof are omitted. I'm going to do it.

(Embodiment 1-1) The present invention relates to an image synthesis system for generating a virtual image through feature extraction and synthesis of images using deep learning, and specifically, a style vector 210 from a first image 10 A style encoder 200 that extracts; a geometry encoder 100 that extracts a geometry vector 110 from a second image 20; and a synthesized image by synthesizing the geometry vector 110 and the style vector 210. A synthesis decoder 300 that generates 310;

Referring to FIG. 3 , the present invention may relate to an image synthesis system that extracts features of a plurality of images using deep learning and synthesizes the features to generate a virtual composite image. The present invention may include a geometry encoder 100, a style encoder 200, and a synthesis decoder 300.

The geometry encoder 100 may be a component that extracts a geometry vector 110 including appearance sketch information as shown in FIG. 2 excluding colors from the second image 20 that is an image of a person.

The style encoder 200 may be a component that extracts a style vector 210 including information such as style and color from the second image 20 , which is an image of a person different from the first image 10 .

The synthesis decoder 300 synthesizes the geometry vector 110 and the style vector to generate a new person image in which the style and color information of the first image 10 and the geometry information or sketch information of the second image 20 are synthesized. It may be a configuration that

As described above, the present invention may relate to a system for generating a new image by extracting a part of information from two different images and synthesizing them.

In addition, the geometry encoder 100, the style encoder 200, and the synthesis decoder 300 of the present invention can be learned by a learning method described later. According to the learning method described below, each of the components constituting the present invention is composed of a configuration combination that extracts only desired information from each image and creates a new image according to the user's intention.

In addition, the present invention can be learned by a learning method described later. One of the key features of learning is to extract geometry information and style information from a person image independently by excluding each other's influence. . Through this, it is possible to secure the predictability of extraction and synthesis.

(Embodiment 2-1) In relation to the learning method of the image synthesis system of Embodiment 1-1, the geometry encoder 100 includes (a) a first image extracted from the first image 10 by the sketch encoder 400. Generating a first learning geometry vector 410 from the sketch 11; (b) Reducing the first error 1, which is an error between the first learning geometry vector 410 and the geometry vector 110 Learning the geometry encoder 100 in a direction; A learning method of an image synthesis system that is learned by a method including;

(Example 2-2) In Example 2-1, the learning method of the image synthesis system for learning the geometry encoder 100 by repeatedly performing steps (a) and (b)

The present invention relates to a method for learning a geometry encoder (100).

The sketch encoder 400 may be a component that extracts and converts into geometry information when a sketch is input, and the sketch decoder 500 may be a component that extracts and converts geometry information into a sketch when input. The geometry vector 110 may be a computer-readable object containing geometry information.

Hereinafter, the present invention will be described in detail.

The geometry vector 110 extracted from the first image 10 by the geometry encoder 100 by learning according to the present invention and the sketch encoder 400 extracted from the first sketch 11 sketching the first image 10 The geometry encoder 100 may be learned in a direction in which an error of the first learning geometry vector 410 is reduced.

As shown in FIG. 2 , the first image 10 as a person image and the first sketch 11 as a sketch image extracted therefrom may include only sketch information of a person excluding style information such as color.

After preparing several sets of images and sketch sets for the images through general software such as Photoshop, the images are input to the geometry encoder 100, the sketches are input to the sketch encoder 400, and then the geometry extracted from each It may be a process of iteratively learning the geometry encoder 100 in a direction of reducing the error by comparing the vector 110 with the first geometry vector 410 for learning.

In the learning process of the present invention, only the geometry encoder 100 is learned, and the sketch encoder 400 is not learned, and the internal algorithm may be in a fixed state. As will be described later, the sketch encoder 400 and the sketch decoder 500 may have already been sufficiently learned through a separate learning process.

In the present invention, the geometry encoder 100 can be learned using the first image and the first sketch. The first image or the first sketch does not mean a specific image or sketch, and numerous images and their corresponding This may mean learning the geometry encoder 100 using sketches as learning data.

(Example 2-3) In Example 2-1, in the step (a), (a-1) the sketch decoder 500 converts the first learning transformation sketch 411 from the first learning geometry vector 410 generating; (a-2) generating, by the sketch decoder 500, a second training transform sketch 421 from the geometry vector 110; -1) learning the geometry encoder 100 in a direction that reduces a second error (2), which is an error between the first transformation sketch 411 for learning and the transformation sketch 421 for second learning; comprising A learning method of an image synthesis system that is trained by the method.

(Example 2-4) In Example 2-3, the learning method of the image synthesis system for learning the geometry encoder 100 by repeatedly performing steps (a) and (b)

Referring to FIG. 4 , the present invention relates to a method for learning a geometry encoder 100 . In step (a) of the present invention, the sketch decoder 500 generates the first training conversion sketch 411 from the first training geometry vector 410 and the sketch decoder 500 generates the second training sketch from the geometry vector 110. A step of generating a transformation sketch 421 for learning may be further included.

In addition, step (b) may further include comparing the first transformation sketch 411 for learning and the transformation sketch 421 for second learning, and learning the geometry encoder 100 in a direction to reduce the error.

As input values for learning the geometry encoder 100 through this, the first error (1), which is an error value between the geometry vector 110 and the first geometry vector 410 for learning, and the first learning transform sketch 411 and the second 2 may be the second error (2), which is an error value with the conversion sketch 421 for learning.

In addition, an input value for learning the geometry encoder 100 may be an error of feature maps generated in an intermediate process of generating a transformation sketch by the sketch decoder 500 . More specifically, in the process (a-1), the sketch decoder 500 generates the first training transformation sketch 411 from the first training geometry vector 410, and in the process of generating, feature values are continuously extracted. can Similarly, in the process (a-2), feature values can be extracted in the process of generating the second training transform sketch 421 from the geometry vector 110 by the sketch decoder 500, and the error values of these two feature values are characterized It can be used as an input value for learning the geometry encoder 100 as a value error.

The present invention also relates to a method for learning the geometry encoder 100, and internal algorithms of the sketch encoder 400 and the sketch decoder 500 may not be learned or changed. As will be described later, the sketch encoder 400 and the sketch decoder 500 may be sufficiently learned through a separate learning process.

(Example 3-1) In Example 2-4, the sketch encoder 400 and the sketch decoder 500 are (c) generated from the person sketch 12 extracted from the person image by the sketch encoder 400. 2 generating a learning geometry vector 420; (d) generating a third training transform sketch 431 from the second training geometry vector 420 by the sketch decoder 500; (e) learning the sketch encoder 400 and the sketch decoder 500 in a direction of reducing a third error 3, which is an error between the person sketch and the third conversion sketch 431 for learning; Learning method of an image synthesis system to be.

(Example 3-2) In Example 3-1, the image synthesis system for learning the sketch encoder 400 and the sketch decoder 500 by repeatedly performing steps (c) and (e) learning method.

In the above-described embodiments, the sketch encoder 400 and the sketch decoder 500 are not learned. More specifically, in the above-described embodiments, the sketch encoder 400 and the sketch decoder 500 have already been sufficiently learned, and the geometry encoder 100 is learned using the sketch encoder 400 and the sketch decoder 500 It can be said that it is an invention of how to do it.

Referring to FIG. 5 , the present invention relates to a method for learning a sketch encoder 400 and a sketch decoder 500 . In order to learn the geometry encoder 100, the sketch encoder 400 and the sketch decoder 500 must have already been sufficiently learned, and the present invention prioritizes the sketch encoder 400 and the decoder prior to learning the geometry encoder 100 It is about learning invention.

The sketch encoder 400 may perform a step of generating a second learning geometry vector 420 from the person sketch 12 extracted from the person image.

Then, the sketch decoder 500 may perform a step of generating a transformation sketch 431 for third training from the second geometry vector 420 for training.

Thereafter, the character sketch 12 put into the sketch encoder 400 and the third conversion sketch 431 for learning generated by the sketch decoder 500 are compared to reduce the third error (3). The encoder 400 and the sketch decoder 500 can be learned.

The above processes can be repeatedly performed while inputting a plurality of person sketches 12 to train the sketch encoder 400 and the sketch decoder 500.

If the sketch encoder 400 and the sketch decoder 500 that have been sufficiently learned through the above processes are used, the person sketch 12 is put into the sketch encoder 400 to generate a second learning geometry vector 420, When the second learning geometry vector 420 is provided and the sketch decoder 500 generates the third learning transformation sketch 431, the person sketch 12 input to the sketch encoder 400 and the third learning transformation sketch ( 431) has a negligibly small value.

A geometry vector for learning or a transform sketch for learning may not be an object in itself, but an object that is merely a means for learning a model desired by a user.

After sufficiently learning the sketch encoder 400 and the sketch decoder 500 through the above-described series of processes, the geometry encoder 100 can be learned using them.

(Example 4-1) In Example 2-1, the style encoder 200 and the synthesis decoder 300 (f) the style encoder 200 converts the first learning style vector 220 from the synthesis image ); (g) the geometry encoder 100 generates a third learning geometry vector 230 from the first image 10; (h) the synthesis decoder 300 generates the Generating a first learning synthesis image 211 by synthesizing the first learning style vector 220 and the third learning geometry vector 230; (i) the first learning synthesis image 211 and the first learning synthesis image 211 Learning the style encoder 200 and the synthesis decoder 300 in a direction of reducing a fourth error 4, which is an error of 1 image 10; Learning method of an image synthesis system that is learned by a method including.

(Example 4-2) In Example 4-1, the image synthesis system for learning the style encoder 200 and the synthesis decoder 300 by repeatedly performing steps (f) and (i) learning method.

The present invention relates to a method for learning a style encoder (200) and a synthesis decoder (300). The style encoder 200 may perform a step of generating the first style vector 220 for learning from the synthesized image.

Then, the geometry encoder 100 may generate a third geometry vector 230 for learning from the first image 10 .

Thereafter, the synthesis decoder 300 may generate the first synthesis image 211 by synthesizing the first learning style vector 220 and the third learning geometry vector 230 .

Thereafter, a step of learning the style encoder 200 and the synthesis decoder 300 in a direction of reducing the error by comparing the first synthesized image 211 for learning and the first image 10 is performed.

If the above-described learning processes are described in detail, the synthesized image input in step (f) may be an image obtained by synthesizing the style vector of the first image 10 and the geometry vector 110 of the second image 20 there is. In step (f), the style encoder 200 receives the synthesized image and generates the first learning style vector 220. If the style encoder 200 is sufficiently learned, the first learning style vector 220 is It may be a vector implying only style information of the first image 10 .

Furthermore, in step (g), the geometry encoder 100 receives the first image 10 and generates a geometry vector 230 for third learning. If the geometry encoder 100 is sufficiently learned, the geometry for third learning The vector 230 may be a vector that implies only geometry information of the first image 10 .

Also, in step (h), the synthesis decoder 300 synthesizes the first learning style vector 220 and the third learning geometry vector 230 generated above to generate the first learning synthesis image 211. Can be generated. Similarly, if the synthesis decoder 300 is sufficiently trained, the synthesis image 211 for first training may be the same image as the first image 10 .

However, before the style encoder 200 and the synthesis decoder 300 are sufficiently trained, the first training synthesis image 211 will have an error with the first image 10 .

Accordingly, the present invention may relate to a method for learning the style encoder 200 and the synthesis decoder 300 in a direction of reducing such an error as in step (i).

In addition, the present invention may relate to a method for sufficiently learning the style encoder 200 and the synthesis decoder 300 by repeatedly performing the above steps (f) to (i). It may be necessary to sufficiently secure the first image 10 and the second image 20 in order to repeatedly perform steps (f) to (i). The process of preparing the image may be performed online and/or offline, but this process may not be a key feature of the present invention, so it will be omitted.

(Example 4-3) In Example 4-2, the style encoder 200 and the synthesis decoder 300 (j) the geometry encoder 100 generates a fourth learning geometry vector 240 from the synthesis image. ); (k) The style encoder 200 generates a second learning style vector 250 from the second image 20; (l) The synthesis decoder 300 generates the second learning style vector 250 Generating a synthesized image 221 for second learning by synthesizing a fourth geometry vector 240 for learning and the second style vector 250; (m) the synthesized image 221 for second learning and the first 2 learning the style encoder 200 and the synthesis decoder 300 in a direction of reducing a fifth error 5, which is an error of the image 20; A learning method of an image synthesis system that is learned by a method including.

(Example 4-4) In Example 4-3, the image synthesis system for learning the style encoder 200 and the synthesis decoder 300 by repeatedly performing steps (j) and (m) learning method.

Other networks that learn the style encoder 200 and synthesis decoder 300 can also be performed. In the above steps (f) to (i), the method for learning the style encoder 200 and the synthesis decoder 300 using the error value between the first image 10 and the first synthesis image 211 for learning is described. If so, the present invention may be a learning method using the second image 20.

More specifically, in step (j), the geometry encoder 100 may generate a fourth learning geometry vector 240 by receiving the synthesized image. If the geometry encoder 100 is sufficiently trained, the fourth learning geometry vector 240 may be a vector that implies only geometry information of the second image 20 .

Then, in step (k), the style encoder 200 may receive the second image 20 and generate a second style vector 250 for learning. If the style encoder 200 has been sufficiently trained, the second learning style vector 250 may be a vector containing only style information of the second image 20 .

Thereafter, in step (l), the synthesized decoder 300 may generate a synthesized image 221 for second learning by synthesizing the fourth geometry vector 240 for learning and the style vector 250 for second learning. If the synthesized decoder 300 has been sufficiently trained, the synthesized image 221 for second learning may be the same image as the second image 20 .

However, before the style encoder 200 and the synthesis decoder 300 are sufficiently trained, the synthesis image 221 for second learning will have an error with the second image 20 . Such an error value is referred to as a fifth error (5).

Accordingly, the present invention may relate to a method for learning the style encoder 200 and the synthesis decoder 300 in a direction of reducing such an error as in step (m).

In addition, the present invention may relate to a method for sufficiently learning the style encoder 200 and the synthesis decoder 300 by repeatedly performing the above steps (j) to (m).

(Example 4-1) and (Example 4-2) learn the style encoder 200 and synthesis decoder 300 using the error (fourth error 4) compared with the first image 10 For how to do this, (Example 4-3) and (Example 4-4) use the fourth error (4) and the error (fifth error (5)) compared with the second image (20). It may be about a method of learning the style encoder 200 and the synthesis decoder 300.

(Example 5-1) In Example 2-1, the style encoder 200 and the synthesis decoder 300 (n) the geometry encoder 100 generates a fourth learning geometry vector 240 from the synthesis image. ); (o) the style encoder 200 generates the second learning style vector 250 from the second image 20; (p) the synthesis decoder 300 generates the Generating a synthesized image 221 for second learning by synthesizing a fourth geometry vector 240 for learning and the second style vector 250; (q) the synthesized image 221 for second learning and the first 2 learning the style encoder 200 and the synthesis decoder 300 in a direction of reducing a fifth error 5, which is an error of the image 20; A learning method of an image synthesis system that is learned by a method including.

(Example 5-2) In Example 5-1, the image synthesis system for learning the style encoder 200 and the synthesis decoder 300 by repeatedly performing steps (n) and (q) learning method.

The present invention also relates to a method for learning the style encoder (200) and synthesis decoder (300). Since the explanation for this is the same as that described in (Example 4-3) and (Example 4-4), it can be replaced with a description thereof. However, in (Example 4-3) and (Example 4-4), the error between the first image 10 and the first synthesized image 211 for learning, together with the second image 20 and the second synthesized image for learning ( 221), if it was a method of learning the style encoder 200 and the synthesis decoder 300 by reflecting all of the errors, (Example 5-1) and (Example 5-2) It may be about a method of learning the style encoder 200 and the synthesis decoder 300 by reflecting only errors with the synthetic image 221 for learning.

The present invention relates to an image synthesis system for generating a virtual image through feature extraction and synthesis of images using deep learning, and specifically

a geometry encoder (100) extracting a geometry vector (110) from the first image (10);

a style encoder 200 that extracts a style vector from the second image 20;

and a synthesizing decoder 300 generating a synthesized image by synthesizing the geometry vector 110 and the style vector.

(Example 6-1) In Example 2-1, the style encoder 200 and the synthesis decoder 300 (r) convert the first image 10 and the second image 20 to the same image. setting as a third image 30; (s) generating a third learning style vector 260 from the third image 30 by the style encoder 200; (t) the geometry encoder (100) generating a fifth learning geometry vector 270 from the third image 30; (u) the synthesis decoder 300 generates the third learning style vector 260 and the fifth learning Generating a third synthesized image 231 for learning by synthesizing geometry vectors 270; (v) a sixth error (6) which is an error between the third synthesized image 231 and the third image 30; ) learning the style encoder 200 and the synthesis decoder 300 in a direction of reducing; learning method of an image synthesis system learned by a method including.

Referring to FIG. 6 , the present invention relates to a method for learning a style encoder 200 and a synthesis decoder 300 . Specifically, it may be about a learning method by a method different from the above-described embodiments 4 and 5. In Embodiments 4 and 5, after generating a composite image using the first image 10 and the second image 20 that are different from each other, the composite image 211 for first learning and/or It may be about a method of generating a second synthetic image 221 for training and learning the style encoder 200 and the synthetic decoder 300 using the fourth error and/or the fifth error 5, which are their error values. .

However, in the present invention, the first image 10 and the second image 20 are the same image, the third image 30, and the style encoder 200 uses the third image 30 as a third learning style vector ( 260), and the geometry encoder 100 may generate a fifth learning geometry vector 270 from the third image 30.

The synthesis decoder 300 synthesizes the third learning style vector 260 and the fifth learning geometry vector 270 thus generated to generate the third learning synthesis image 231 .

The style encoder 200 and the synthesis decoder 300 may be learned in a direction in which a sixth error 6, which is an error value, is reduced by comparing the third image 30 with the third synthesized image 231 for training.

Since the above method has a relatively simple network structure compared to Example 4 and Example 5, the learning process is light and the advantage of saving the time required for the learning can be secured.

In the entire specification of the present invention, the first error (1) to the sixth error (6) are defined, and a configuration to be learned, such as a geometry encoder 100, a style encoder 200, a synthesis decoder 300, etc. The principle of learning may be a principle of performing a feedback action in a direction of reducing errors.

More specifically, it can be learned using a back propagation method, and since such a method is known to those skilled in the art, a detailed description thereof will be omitted.

In addition, the least square method may be used, and when comparing images and setting the error value, the LAB color difference calculation method may be used. In addition, the principle of VGGNet (hereinafter “VGG”), which calculates an error value by continuously comparing feature maps generated in the process of encoding, decoding, and synthesizing several vectors containing image information, may be used. The VGG error calculation method may compare values of feature maps generated in the process of increasing or decreasing the depth of a layer in the process of converting to an image or converting to a vector.

In addition, “learning” described in this specification is not a purpose in itself, but may be an object temporarily created to learn the configurations of the image synthesis system of the present invention.

Ultimately, the present invention extracts and synthesizes geometric features and style features from two arbitrary person images using the sufficiently learned geometry encoder 100, style encoder 200, and synthesis decoder 300, so that the respective features are It may be to a system that creates a new composite image that is combined.

In addition, it may be an invention for a learning method of the above-described image synthesis system, and the image synthesis system may be learned by employing a characteristic learning method according to the present invention.

Finally, FIG. 7 is an internal block diagram of a general-purpose computer device that can be employed to perform a system for extracting and synthesizing features of an image using deep learning and a learning method thereof according to the present invention. The computer device 900 includes one or more processors 910 connected to a main memory including a random access memory (RAM) 920 and a read only memory (ROM) 930. The processor 910 is also referred to as a central processing unit (CPU). The processor may be a subject of learning as a component that performs a learning method. As is widely known in the art, the ROM 930 serves to unidirectionally transmit data and instructions to the CPU, and the RAM 920 typically transmits data and instructions bidirectionally used to do RAM 920 and ROM 930 may include any suitable form of computer readable media. The mass storage device 940 is bi-directionally connected to the processor 910 to provide additional data storage capability, and may be any of the above-described computer readable recording media. The mass storage device 940 is used to store programs, data, and the like, and is typically a secondary storage device such as a hard disk that is slower than the main storage device. A special mass storage device such as CD ROM 960 may also be used. Processor 910 may include one or more input/output devices, such as a video monitor, trackball, mouse, keyboard, microphone, touch screen type display, card reader, magnetic or paper tape reader, voice or handwriting recognizer, joystick, or other known computer input/output devices. It is connected to interface 950. Finally, processor 910 may be coupled to a wired or wireless communication network via network interface 970 . The procedure of the above-described method can be performed through such a network connection. The devices and tools described above are well known to those skilled in the art of computer hardware and software.

In the specification of the present invention (particularly in the claims), the use of the term "above" and similar indicating terms may correspond to both singular and plural. In addition, when a range is described in the present invention, it includes an invention in which individual values belonging to the range are applied (unless there is a description to the contrary), and each individual value constituting the range is described in the detailed description of the invention Same as

The steps constituting the method according to the present invention may be performed in any suitable order unless an order is explicitly stated or stated to the contrary. The present invention is not necessarily limited according to the order of description of the steps. The use of all examples or exemplary terms (eg, etc.) in the present invention is simply to explain the present invention in detail, and the scope of the present invention is limited due to the examples or exemplary terms unless limited by the claims. it is not going to be In addition, those skilled in the art can appreciate that various modifications, combinations and changes can be made according to design conditions and factors within the scope of the appended claims or equivalents thereof.

Therefore, the spirit of the present invention should not be limited to the above-described embodiments and should not be determined, and all scopes equivalent to or equivalently changed from the claims as well as the claims to be described later are within the scope of the spirit of the present invention. would be said to belong to

10: first image 20: second image
30: third image 100: geometry encoder
110: geometry vector 200: style encoder
210: style vector 220: style vector for first learning
230: geometry vector for third learning 240: geometry vector for fourth learning
250: style vector for second learning 260: style vector for third learning
270: geometry vector for fifth learning 300: synthesis decoder
400: sketch encoder 410: geometry vector for first learning
420: Geometry vector for second learning 411: Transformation sketch for first learning
421: conversion sketch for second learning 431: conversion sketch for third learning

Claims (11)

delete A style encoder (200) extracting a style vector (210) from a first image (10), a geometry encoder (100) extracting a geometry vector (110) from a second image (20), and the geometry vector (110) and the above In the learning method of an image synthesis system including a synthesis decoder 300 generating a synthesis image 310 by synthesizing a style vector 210,
The geometry encoder 100,
(a) generating a first learning geometry vector 410 from the first sketch 11 extracted from the first image 10 by the sketch encoder 400; and
(b) learning the geometry encoder 100 in a direction that reduces a first error 1, which is an error between the first learning geometry vector 410 and the geometry vector 110; Learning through;
In step (a),
(a-1) generating, by the sketch decoder 500, a transform sketch 411 for first training from the geometry vector for first training; and
(a-2) generating, by the sketch decoder 500, a transformation sketch 421 for second learning from the geometry vector 110;
In step (b),
(b-1) learning the geometry encoder 100 in a direction that reduces a second error (2), which is an error between the first transformation sketch 411 for learning and the transformation sketch 421 for second learning; contains,
The style encoder 200 and the synthesis decoder 300,
(f) generating, by the style encoder 200, a first learning style vector 220 from the synthesized image;
(g) generating a third learning geometry vector 230 from the first image 10 by the geometry encoder 100;
(h) synthesizing, by the synthesis decoder 300, the first learning style vector 220 and the third learning geometry vector 230 to generate a first learning synthesis image 211; and
(i) learning the style encoder 200 and the synthesis decoder 300 in a direction of reducing a fourth error 4 that is an error between the first synthetic image 211 for learning and the first image 10 is learned through;
The geometry encoder 100 is also learned in the direction of reducing the feature value error,
The feature value error is a feature value ( feature map) and the feature value (feature map) continuously extracted in the process of generating the second learning conversion sketch 421 from the geometry vector 110 by the sketch decoder 500 in step (a-2) A learning method of an image synthesis system, defined as an error value.
delete ◈Claim 4 was abandoned when the registration fee was paid.◈ The method of claim 2,
A learning method of an image synthesis system for learning the geometry encoder 100 by repeatedly performing steps (a) and (b).
◈Claim 5 was abandoned when the registration fee was paid.◈ The method of claim 4,
The sketch encoder 400 and the sketch decoder 500 perform (c) generating a second learning geometry vector 420 from the person sketch 12 extracted from the person image by the sketch encoder 400;
(d) generating, by the sketch decoder 500, a third transform sketch 431 from the second geometry vector 420 for training; and
(e) learning the sketch encoder 400 and the sketch decoder 500 in a direction of reducing a third error 3, which is an error between the character sketch and the third conversion sketch 431 for learning; A learning method of an image synthesis system that is trained by the method.
delete The method of claim 2,
A learning method of an image synthesis system for learning the style encoder 200 and the synthesis decoder 300 by repeatedly performing steps (f) and (i).
The method of claim 7,
The style encoder 200 and the synthesis decoder 300 may include (j) the geometry encoder 100 generating a fourth learning geometry vector 240 from the synthesis image;
(k) generating a second learning style vector 250 from the second image 20 by the style encoder 200;
(l) generating a second learning synthesis image 221 by synthesizing the fourth learning geometry vector 240 and the second learning style vector 250 by the synthesis decoder 300; and
(m) learning the style encoder 200 and the synthesis decoder 300 in a direction of reducing a fifth error 5 that is an error between the second synthesized image 221 for learning and the second image 20 A learning method of an image synthesis system that is learned by a method including;
The method of claim 8,
A learning method of an image synthesis system for learning the style encoder 200 and the synthesis decoder 300 by repeatedly performing steps (j) and (m).
The method of claim 2,
The style encoder 200 and the synthesis decoder 300 include (n) the geometry encoder 100 generating a fourth learning geometry vector 240 from the synthesis image;
(o) generating a second learning style vector 250 from the second image 20 by the style encoder 200;
(p) generating a second learning synthesis image 221 by synthesizing the fourth learning geometry vector 240 and the second learning style vector 250 by the synthesis decoder 300; and
(q) learning the style encoder 200 and the synthesis decoder 300 in a direction of reducing a fifth error 5 that is an error between the second synthesized image 221 for learning and the second image 20 A learning method of an image synthesis system that is learned by a method including;

◈Claim 11 was abandoned when the registration fee was paid.◈ The method of claim 2,
The style encoder 200 and the synthesis decoder 300 step (r) setting the first image 10 and the second image 20 to a third image 30 that is the same image;
(s) generating a third learning style vector 260 from the third image 30 by the style encoder 200;
(t) generating a fifth learning geometry vector 270 from the third image 30 by the geometry encoder 100;
(u) synthesizing, by the synthesis decoder 300, the third learning style vector 260 and the fifth learning geometry vector 270 to generate a third learning synthesis image 231; and
(v) learning the style encoder 200 and the synthesis decoder 300 in a direction of reducing a sixth error 6 that is an error between the third synthesized image 231 for learning and the third image 30 A learning method of an image synthesis system that is learned by a method including;

Images