JP2026042613A - Information processing device, information processing system, information processing method, and information processing program - Google Patents

Abstract

A facial authentication system is provided that protects feature data extracted from a facial image while allowing a person to visually confirm the facial image.
[Solution] An information processing device performs a one-way transformation based on a predetermined algorithm on feature data extracted from a facial image to generate protection features indicating the protected features of the facial image and auxiliary information to be used for authentication using the protection features, generates a protection image for the facial image based on the facial image and the predetermined protection transformation, and registers the auxiliary information, the protection features, and the protection image in a database.
[Selected Figure] Figure 1

Description

The present invention relates to an information processing device, an information processing system, an information processing method, and an information processing program.

BTP (Biometric Template Protection) technology, which authenticates biometric data such as facial images while protecting it through special encryption and one-way conversion, is being researched and developed, and efforts are underway to put it into practical use and standardize it.

The standard architectures for BTP technology are known to be PIE (Pseudonomous Identifier Encoder), which realizes the registration feature protection transformation function, PIR (Pseudonomous Identifier Recorder), which realizes the authentication feature protection transformation function, and PIC (Pseudonomous Identifier Comparator), which realizes the transformation feature matching function.

The PIE converts the feature data used at the time of enrollment to generate AD (Auxiliary Data) and PI (Pseudonomous Identifier). The original feature data cannot be restored from either or both of the AD and PI. The PIR generates PI*, which is the PI for authentication, from the feature data used at the time of authentication and the AD generated by the PIE. The original feature data cannot be restored from PI*. The PIC compares the PI and PI* and outputs "Success (Similar)" or "Failure (Dissimilar)." The PIC can determine the similarity between the original feature data and the feature data used at the time of authentication without restoring the original feature data.

A specific example of BTP technology is the Fuzzy Extractor (FE) described in Non-Patent Document 1. The PIE in FE first transforms the feature data used during enrollment to generate AD data for correcting errors that may occur during authentication. Examples of transformation methods include concealing the feature data by additively applying random error-correcting code words, or removing stable components useful for authentication from the feature data and leaving only the error components. Both of these are one-way transformations that are information-theoretically irreversible. The PIE in FE then generates PI data by applying a one-way transformation, such as a hash function, to the feature data and AD, which is computationally difficult to reverse. In this way, the PIE in FE is a one-way transformation that generates AD and PI from feature data, and has the property that it is difficult to restore the original feature data from either or both of the AD and PI.

The PIR in the FE uses the AD to correct errors in the feature data used at the time of authentication, restoring the feature data used at the time of registration, and generates PI* by performing one-way transformations such as a hash function on the restored feature data and AD. The PIC in the FE compares PI with PI*, outputting a successful match if there is an exact match, or a failed match if there is not. Because PI and PI* are data that cannot be distinguished from random numbers, they can also be used as secure encryption keys.

By applying BTP technologies such as FE to facial recognition technology, even if the AD or PI registered in the system is leaked, an attacker who obtains it will not be able to restore facial feature data, making it possible to realize a secure facial recognition system.

A technology with a different purpose than protecting feature data in user authentication and identity verification is described in Non-Patent Document 2. The technology described in Non-Patent Document 2 aims to prevent attacks that violate privacy by tracking user behavior by attackers performing image searches of facial photos uploaded to Internet services such as SNS (Social Networking Service) without the user's permission. The technology described in Non-Patent Document 2 uses machine learning to add subtle perturbations to facial images, thereby hindering automatic face matching by image search systems.

Dodis Y., Reyzin L., Smith A. “Fuzzy Extractors: How to Generate Strong Keys from Biometrics and Other Noisy Data.” In: Cachin C., Camenisch J.L. (eds) Advances in Cryptology - EUROCRYPT 2004. EUROCRYPT 2004. Lecture Notes in Computer Science, vol 3027. Springer, Berlin, Heidelberg S.Shan, E.Wenger, J.Zhang,H.Li, H.Zheng and B.Y.Zhao, “Fawkes: Protecting privacy against unauthorized deep learning models,” Proc. the 29th USENIX Security Symposium, pp.1589-1604, Aug. 2020.

Depending on the use case for facial recognition, in addition to the system's automatic matching process, a human may need to visually confirm the facial image. In such use cases, the system must store the user's facial image during registration and display it on a screen for visual confirmation during authentication.

For this reason, even if feature data is protected using the technology described in Non-Patent Document 1, there is a risk that the facial image will be leaked when it is displayed on a screen for visual confirmation, and an attacker who obtains the leaked facial image can extract and misuse the feature data. Furthermore, the technology described in Non-Patent Document 2 is designed to prevent the matching of a user's facial image, and therefore has a fundamentally different purpose from user authentication, which performs feature matching based on a facial image (assuming that feature matching will sometimes be successful).

Therefore, one aspect of the present invention is to realize facial recognition that allows humans to visually confirm facial images while protecting feature data extracted from facial images.

To solve the above problem, one aspect of the present invention employs the following configuration: An information processing device includes a processor and a storage device. The storage device holds a database in which a registration facial image, registration feature data extracted from the registration facial image and indicating features of the registration facial image, protection features indicating protected features of the facial image, auxiliary information used for authentication using the protection features, and a protected image generated based on a predetermined protection transformation of the facial image are registered. The processor performs a one-way transformation on the registration feature data based on a predetermined algorithm to generate registration protection features that are the protection features for the registration feature data and the auxiliary information for the registration feature data, generates a registration protected image that is the protection image for the registration facial image based on the registration facial image and the predetermined protection transformation, and registers the generated auxiliary information, registration protection features, and registration protected image in the database.

One aspect of the present invention makes it possible to achieve face authentication that allows humans to visually confirm face images while protecting feature data extracted from face images.

Other issues, configurations, and advantages will become clear from the description of the embodiments below.

1 is a block diagram showing an example of the system configuration of a face authentication system according to a first embodiment. 2 is a block diagram illustrating an example of the hardware configuration of a computer that constitutes a registration terminal, an authentication terminal, a visual confirmation terminal, and an authentication server according to the first embodiment. FIG. FIG. 10 is a sequence diagram illustrating an example of a registration process according to the first embodiment. FIG. 10 is a sequence diagram illustrating an example of authentication processing according to the first embodiment.

Embodiments of the present invention will now be described in detail with reference to the drawings. In this embodiment, identical components will generally be assigned the same reference numerals, and repeated explanations will be omitted. It should be noted that this embodiment is merely one example for realizing the present invention, and does not limit the technical scope of the present invention.

This embodiment relates to a face recognition system that enables people to visually confirm face images while protecting feature data extracted from face images from leakage and misuse.

In this embodiment, the facial authentication system not only performs a matching process for features generated based on a facial image, but also displays the facial image for a person to visually confirm. A use case requiring both mechanical feature matching and human visual confirmation is, for example, identity verification when opening an online banking account, where a mechanical match is required between facial photo data included on an identification card and a photograph of the applicant's face, and an operator is also required to compare and visually confirm the two.

Another use case is when a store makes a payment using facial recognition, and the system displays the facial image of the registered user along with the facial recognition results on a terminal for store staff, who then visually confirms it by comparing it with the user's facial features. Another use case is when immigration control at airports and entrance/exit management at facilities use facial recognition, and if the system's facial recognition fails, an administrator may visually confirm the face.

Figure 1 is a block diagram showing an example system configuration of a facial authentication system. The facial authentication system includes, for example, a registration terminal 100, an authentication terminal 110, a visual confirmation terminal 120, and an authentication server 130, all of which are connected to one another via a network 150 such as the Internet. Note that any two or all three of the registration terminal 100, authentication terminal 110, and visual confirmation terminal 120 may be implemented in the same device (i.e., integrated). Note that if there is an integrated device, communication between the devices is omitted.

The registration terminal 100 acquires the user's ID information and facial image during user registration. The registration terminal 100 has, for example, an ID input function 101, a camera 102, a photographing function 103, and a communication function 104. The ID input function 101 accepts input of a user ID. The camera 102 photographs the user's facial image. The photographing function 103 controls the camera 102 to acquire the user's facial image. The communication function 104 controls communication with other devices connected to the registration terminal 100 via the network 150.

The authentication terminal 110 acquires the user's ID information and facial image during user authentication. The authentication terminal 110 has, for example, an ID input function 111, a camera 112, a photographing function 113, and a communication function 114. The ID input function 111 accepts the input of a user ID. The camera 112 photographs the user's facial image. The photographing function 113 controls the camera 112 to acquire the user's facial image. The communication function 114 controls communication with other devices connected to the authentication terminal 110 via the network 150.

The visual confirmation terminal 120 displays a facial image of the user for visual identity verification by the operator. The visual confirmation terminal 120 has a communication function 121, an image display function 122, and a judgment result input function 123. The communication function 121 controls communication with other devices connected to the visual confirmation terminal 120 via the network 150. The image display function 122 displays the protected image on the screen. The judgment result input function 123 accepts input of the identity verification judgment result by the operator.

The authentication server 130 performs processes such as registration and matching based on facial images. The authentication server 130 has, for example, a communication function 131, a facial feature extraction function 132, a registration feature protection conversion function 133, a facial image protection conversion function 134, a protected image determination function 135, an encryption function 136, an authentication feature protection conversion function 137, a protected feature matching function 138, a final determination function 140, and a registration search function 141. The authentication server 130 also stores an auxiliary information DB (Database) 142, a protected feature DB 143, and an encrypted protected image DB 144.

The communication function 131 controls communication with other devices connected to the authentication server 130 via the network 150. The facial feature extraction function 132 extracts feature data from a facial image. The registered feature protection transformation function 133 is realized, for example, by a PIE (Pseudonomous Identifier Encoder). The registered feature protection transformation function 133 generates auxiliary information AD (Auxiliary Data) and protected feature PI (Pseudonomous Identifier) by performing a one-way transformation using PIE on the feature data at the time of registration. The auxiliary information AD is used, for example, to correct errors in the feature data during authentication. The protected feature PI indicates features protected by performing a computationally intensive one-way transformation, such as a hash function, on the facial image feature data and AD.

The facial image protection conversion function 134 converts the facial image to generate a protected image. The protected image determination function 135 verifies the security of the protected image. The encryption function 136 encrypts the protected image based on the protection feature PI to generate an encrypted protected image.

The authentication feature protection conversion function 137 is realized, for example, by a PIR (Pseudonomous Identifier Recorder). During authentication, the authentication feature protection conversion function 137 generates authentication protection features PI* by performing one-way conversion, such as a hash function, on the feature data and auxiliary information AD.

The protection feature matching function 138 matches the protection feature PI with the authentication protection feature PI*. The protection feature matching function 138 is realized, for example, by a PIC (Pseudonomous Identifier Comparator). If the match is successful, the decryption function 139 decrypts the encrypted protected image using the authentication protection feature PI*.

The final determination function 140 makes a final determination regarding authentication based on the matching results from the protection feature matching function 138 and the information received as input by the determination result input function 123. The registration search function 141 registers the auxiliary information AD, protection feature PI, and encrypted protected image in the auxiliary information DB 142, protection feature DB 143, and encrypted protected image DB 144, respectively. The registration search function 141 searches each DB.

Figure 2 is a block diagram showing an example of the hardware configuration of the computers 400 that make up each of the registration terminal 100, authentication terminal 110, visual confirmation terminal 120, and authentication server 130. The computers 400 each include, for example, a CPU (Central Processing Unit) 401, memory 402, auxiliary storage device 403, input device 404, display device 405, and communication device 406, all of which are connected to each other via internal communication lines such as a bus.

CPU 401 is an example of a processor, and executes programs stored in memory 402. Examples of processors include CPU 401 and a GPU (Graphics Processing Unit), but other semiconductor devices may also be used as long as they are capable of executing predetermined processing.

Memory 402 includes ROM (Read Only Memory), a non-volatile storage element, and RAM (Random Access Memory), a volatile storage element. ROM stores unchanging programs (e.g., BIOS (Basic Input/Output System)). RAM is a high-speed, volatile storage element such as DRAM (Dynamic Random Access Memory), and temporarily stores programs executed by CPU 401 and data used when the programs are executed.

The auxiliary storage device 403 is a large-capacity, non-volatile storage device such as a magnetic storage device (HDD (Hard Disk Drive)) or flash memory (SSD (Solid State Drive)), and stores programs executed by the CPU 401 and data used when the programs are executed. In other words, the programs are read from the auxiliary storage device 403, loaded into memory 402, and executed by the CPU 401.

The input device 404 is a device, such as a keyboard or mouse, that receives input from the user. The display device 405 is a device, such as a display or printer, that outputs the results of program execution in a format that can be viewed by the user.

The communication device 406 is a network interface device that controls communication with other devices according to a predetermined protocol. The communication device 406 may also include a serial interface such as a USB (Universal Serial Bus).

Part or all of the programs executed by CPU 401 may be provided to computer 400 via network 150 from removable media (such as CD-ROM or flash memory), which are non-transitory storage media, or from an external computer equipped with non-transitory storage devices, and stored in non-volatile auxiliary storage device 403, which is also a non-transitory storage medium. For this reason, computer 400 should preferably have an interface for reading data from removable media.

The registration terminal 100 is a computer system configured on a single physical computer 400, or on multiple logically or physically configured computers 400, and may operate in separate threads on the same computer 400, or on a virtual computer constructed on multiple physical computer resources. The same applies to the authentication terminal 110, visual confirmation terminal 120, and authentication server 130.

The CPU 401 of the computer 400 constituting the registration terminal 100 realizes each of the functions of the registration terminal 100 shown in FIG. 1. For example, the CPU 401 of the computer 400 constituting the registration terminal 100 realizes the ID input function 101 by operating in accordance with an ID input program loaded into the memory 402 of the computer 400 constituting the registration terminal 100, realizes the photographing function 103 by operating in accordance with a photographing program loaded into the memory 402, and realizes the communication function 104 by operating in accordance with a communication program loaded into the memory 402. Similarly, the functions possessed by each of the authentication terminal 110, visual confirmation terminal 120, and authentication server 130 are realized by the CPU 401 of the computer 400 constituting the device operating in accordance with a program loaded into the memory 402 of the computer 400 constituting the device.

In addition, some or all of the functions of the registration terminal 100, authentication terminal 110, visual confirmation terminal 120, and authentication server 130 may be realized by dedicated circuits such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array).

The auxiliary storage device 403 of the computer 400 constituting the authentication server 130 provides storage areas for implementing the auxiliary information DB 142, the protection feature DB 143, and the encrypted protected image DB 144. Some or all of the information stored in the auxiliary information DB 142, the protection feature DB 143, and the encrypted protected image DB 144 may be stored in the memory 402 of the computer 400 constituting the authentication server 130, or may be stored in an external database connected to the authentication server 130.

In this embodiment, the information used by the facial recognition system does not depend on the data structure and may be expressed in any data structure. For example, the information may be stored in a data structure appropriately selected from a table, list, database, or queue.

Figure 3 is a sequence diagram showing an example of the registration process. The registration terminal 100 starts the registration process in response to a user operation on the input device 404 of the computer 400 constituting the registration terminal 100, and the ID input function 101 acquires a user ID in accordance with the user input on the input device 404 (S200). The photographing function 103 controls the camera 102 to acquire a facial image of the user for registration, and transmits the facial image and the user ID acquired in step S200 to the authentication server 130 (S201).

The authentication server 130 receives the facial image sent in step S201, and the facial feature extraction function 132 extracts feature data from the facial image using a predetermined feature extraction algorithm (S202). The registered feature protection conversion function 133 generates auxiliary information AD and protected features PI by performing a unidirectional conversion based on PIE on the feature data (S203).

The PIE, PIR, and PIC used in this embodiment are the same as the PIE, PIR, and PIC in FE (Fuzzy Extractor). Therefore, the protection feature PI and auxiliary information AD in this embodiment are the hash value and auxiliary information in FE, respectively. Furthermore, the authentication protection feature PI* in this embodiment is a hash value obtained by applying a hash function to the feature information AD obtained by error-correcting the feature data of the facial image at the time of authentication using the auxiliary information AD. However, the PIE, PIR, and PIC are not limited to the PIE, PIR, and PIC in FE, and the PIE, PIR, and PIC in any BTP (Biometric Template Protection) algorithm can be applied.

The facial image protection conversion function 134 generates a protected image from the facial image by applying protection conversion to the facial image using a predetermined algorithm (S204). Here, the protection conversion performed by the facial image protection conversion function 134 has the following two properties:

The first property is feature dissimilarity. Feature dissimilarity is the property where the similarity between the feature data extracted from the original facial image (feature data extracted in step S202) and the protected image feature data, which is feature data extracted by the facial feature extraction function 132 from the protected image generated in step S204, is sufficiently low (dissimilar based on predetermined conditions (for example, the similarity is below a predetermined value)).

By having feature dissimilarity in the protection transformation, even if the protected image is leaked due to hacking into the system or if a protected image displayed on the operator's screen is illegally photographed, it is not possible to generate an authentication protection feature PI* that will successfully authenticate from the leaked protected image or an illegally photographed protected image, thereby preventing spoofing attacks.

The second property is symmetric visual similarity. Symmetric visual similarity is the property that if any two facial images (I1, I2) have visual similarity (the property of being similar to or greater than a predetermined threshold in a facial image similarity measure that models human visual perception functions (such as SSIM (Structural Similarity))), then and only if, the two protected images (I1', I2') obtained by protecting and transforming (I1, I2) respectively also have visual similarity.

The protection transformation may have asymmetric visual similarity instead of or in addition to symmetric visual similarity. Asymmetric visual similarity is a property in which, if and only if, any two face images (I1, I2) have visual similarity (the property of being similar enough to be determined to be the same person by a human's visual perception function), the two protected images (I1', I2) also have visual similarity.

Because the protection transformation has symmetric visual similarity, an operation can be realized in which the operator of the visual confirmation terminal 120 visually confirms the user's protected image both at the time of registration and at the time of authentication on the screen of the visual confirmation terminal 120 without meeting the user face-to-face (hereinafter, this operation will also be referred to as "online identity verification").

Furthermore, because the protection transformation has asymmetric visual similarity, an operation can be realized in which the operator of the visual confirmation terminal 120 meets the user face-to-face, visually views only the protected image at the time of registration on the screen of the visual confirmation terminal 120, and compares the protected image at the time of registration with the facial features of the user they are meeting to verify the user's identity (hereinafter, this operation will also be referred to as "face-to-face identity verification").

In this embodiment, to configure a facial image protection transformation having the above properties, the facial image protection transformation function 134 uses an image transformation with preserving visual similarity. Preserving visual similarity is a property in which images (I1, I1') before and after the facial image protection transformation have visual similarity in addition to feature dissimilarity.

A specific algorithm for image transformation with preserving visual similarity is disclosed in Non-Patent Document 2. When the face image protection transformation function 134 uses image transformation with preserving visual similarity, (I1, I1') and (I2, I2') each have visual similarity. Therefore, if and only if (I1, I2) have visual similarity, it is expected that (I1', I2') also have visual similarity (symmetric visual similarity), and (I1', I2) also have visual similarity (asymmetric visual similarity).

Facial image preserving transformations with feature dissimilarity and symmetric visual similarity can also be achieved using methods other than the method using the image transformation with preserving visual similarity described above.

Specifically, for example, the facial image protection transformation function 134 may use, as the facial image protection transformation, a transformation (morphing transformation) in which the user's facial image I1 (or I2) is input, morphs it to resemble a facial image J of another person (facial image J is stored in advance, for example, by the authentication server 130), and outputs an intermediate image between I1 (or I2) and J as I1' (or I2').

Morphing transformations significantly alter the facial image itself, so neither features nor visibility are preserved. Therefore, morphing transformations do not have either preserving visual similarity or asymmetric visual similarity, but they do have symmetric visual similarity and feature dissimilarity.

Next, the protected image determination function 135 verifies the security of the protected image and determines whether protection was successful or unsuccessful. If protection was unsuccessful, the registration process is terminated (without executing the processes from step S206 onwards) (S205).

An example of a specific protection image determination method in step S205 will be described. The facial feature extraction function 132 extracts feature data of the protection image from the protection image generated in step S204. The authentication feature protection conversion function 137 generates protection features PI' of the protection image from the feature data of the protection image and auxiliary information AD using PIR. The protection feature matching function 138 uses PIC to match the protection features PI with the protection features PI' of the protection image. If the match results in failure (dissimilarity), the protection image determination function 135 determines that "protection was successful," and if the match results in success (similarity), the protection image determination function 135 determines that "protection failed."

As another method for determining a protected image in step S205, the protected image determination function 135 may evaluate the similarity between the feature data extracted in step S202 and the feature data of the protected image using a scale similar to that used in general face recognition algorithms (for example, a predetermined type of similarity such as cosine similarity), and determine that "protection was successful" if the similarity is below a predetermined threshold, or that "protection failed" if it is not.

When the method of Non-Patent Document 2 or morphing conversion is used as the method for facial image protection conversion by the facial image protection conversion function 134, feature dissimilarity is not necessarily met with 100% accuracy. Depending on the input facial image and the value of the random number used within the facial image protection conversion, there is a probability that the output protected image will not meet feature dissimilarity. In this case, there is a risk that feature data will be leaked from the protected image. Therefore, by performing a protected image determination in step S205, feature data can be reliably protected.

The encryption function 136 then encrypts the protected image generated in step S204 using the protection feature PI generated in step S203 as the encryption key, thereby generating an encrypted protected image (S206). The encrypted protected image will only be correctly decrypted if the PIC matching process in the authentication process described below is successful (if PI = PI*). While the processing of step S206 is not required, further encrypting the protected image visible to humans makes it possible to protect the user's privacy even if the data stored on the authentication server 130 is leaked.

Next, the registration search function 141 links the auxiliary information AD with the user ID and registers it in the auxiliary information DB 142, links the protection features PI with the user ID and registers it in the protection features DB 143, and links the encrypted protected image with the user ID and registers it in the encrypted protected image DB 144 (S207).

Figure 4 is a sequence diagram showing an example of authentication processing. The authentication terminal 110 starts the authentication processing in response to a user's operation on the input device 404 of the computer 400 that constitutes the authentication terminal 110, and the ID input function 111 acquires a user ID in accordance with the user's input on the input device 404 (S300). Note that "1:N authentication," which does not acquire a user ID, can also be implemented.

The photographing function 113 controls the camera 112 to acquire the user's facial image for authentication, and transmits the user ID acquired in step S300 and the facial image for authentication to the authentication server 130 (S301). Note that when 1:N authentication is performed, the photographing function 113 transmits only the facial image for authentication.

The authentication server 130 receives the authentication facial image sent in step S301, and the facial feature extraction function 132 extracts authentication feature data from the authentication facial image (S302). The registration search function 141 searches the auxiliary information DB 142 for auxiliary information AD corresponding to the user ID sent in step S301, and the authentication feature protection conversion function 137 generates authentication protection features PI* by performing a unidirectional conversion based on PIR on the authentication feature data and the auxiliary information AD (S303).

The registration search function 141 searches the protection feature DB 143 for the protection feature PI corresponding to the user ID sent in step S301, and the protection feature matching function 138 matches the protection feature PI with the authentication protection feature PI* using the PIC, and determines whether the match is successful or unsuccessful as a result of the match (S304).

When performing 1:N authentication, the authentication server 130 repeatedly executes the processes of steps S303 and S304 for all registered user IDs, and generates a list (success list) of user IDs and protection features PI for which matching was successful.

The facial image protection conversion function 134 generates a protected authentication image from the facial authentication image using the same algorithm as in step S204 (S305). The protected image determination function 135 verifies the security of the protected authentication image using a method similar to that described in step S205 (a method of comparing, using PIC, the protected features generated by PIE from the authentication feature data with the protected features of the protected authentication image generated by PIR from the feature data of the protected authentication image and auxiliary information (generated by PIE from the authentication feature data), or a method of evaluating the similarity between the feature data for authentication and the feature data of the protected authentication image), and determines whether protection was successful or unsuccessful. If protection was unsuccessful, the authentication process is terminated (without executing the processes from step S307 onwards) (S306). Note that, if only face-to-face identity verification is performed and online identity verification is not performed, the processes of steps S305 and S306 may be omitted.

The registration search function 141 searches the encrypted protected image DB 144 for the encrypted protected image corresponding to the user ID sent in step S301, and the decryption function 139 performs a decryption process on the encrypted protected image using the authentication protection feature PI* as the decryption key, and sends the authentication protected image and the protected image obtained as a result of the decryption process to the visual confirmation terminal 120 (S307).

When performing 1:N authentication, in step S307, the authentication server 130 searches for and decrypts the encrypted protected image for each user ID included in the success list described above, and transmits the authentication protected image and all protected images obtained as a result of the decryption process to the visual confirmation terminal 120.

Note that if the processing of step S206 is omitted, the decoding processing by the decoding function 139 in step S307 is omitted.

Furthermore, if the processing of step S206 is performed and a matching result indicating a matching failure is obtained in the matching processing of step S304, the generation of the authentication protection feature PI* as a decryption key has failed, and therefore the decryption processing of step S307 will also fail. Therefore, if the processing of step S206 is performed and a matching result indicating a matching failure is obtained in the matching processing of step S304, the authentication processing may end without performing the processing of step S305 and subsequent steps.

The visual confirmation terminal 120 receives the protected image and the protected image for authentication transmitted in step S307, and the image display function 122 displays the received protected image and the protected image for authentication on the display device 405 of the computer 400 constituting the visual confirmation terminal 120 (S308).

In the case of face-to-face identity verification, in step S308, the image display function 122 may display only the received protected image on the display device 405. In the case of 1:N authentication, the image display function 122 may display multiple protected images on the display device 405 (for example, in descending order of the matching score used for matching in step S304), or may not display any protected images on the display device 405 if the success list is empty.

When performing online identity verification, the operator visually compares the protected image displayed on the display device 405 with the protected image for authentication, and determines whether the user whose face image was captured by the authentication terminal 110 is the registered user. On the other hand, when performing face-to-face identity verification, the operator visually compares the protected image displayed on the display device 405 with the facial features of the user facing the user, and determines whether the user facing the user is the registered user.

In this case, even if the protected image and/or the protected image for authentication displayed on the display device 405 is photographed by an external attacker using a camera, or if the data is leaked by an unauthorized operator (internal attacker), the attacker will not be able to extract feature data that would enable successful authentication from the photographed image or leaked data. Therefore, while allowing visual confirmation by the operator, a high level of security can be ensured against attacks that attempt to impersonate someone by stealing a facial image.

The judgment result input function 123 accepts input of the operator's visual judgment result (a judgment result indicating whether visual identity verification was successful or not) via the input device 404 of the computer 400 constituting the visual confirmation terminal 120, and transmits the judgment result to the authentication server 130 (S309).

The final judgment function 140 of the authentication server 130 generates a final judgment result based on the matching result in step S304 and the judgment result sent in step S309, and sends the final judgment result to the authentication terminal 110 and the visual confirmation terminal 120 (S310).

In step S310, if the matching result in step S304 is successful and the judgment result transmitted in step S309 is successful in identity verification, the final judgment function 140 determines the final judgment result to be successful authentication. Also, if the matching result in step S304 is unsuccessful and the judgment result transmitted in step S309 is unsuccessful in identity verification, the final judgment function 140 determines the final judgment result to be unsuccessful authentication.

Furthermore, if only one of the following conditions is satisfied: the matching result in step S304 is a successful match, and the judgment result transmitted in step S309 is a successful identity verification, the final judgment function 140 may be configured to determine the final judgment result as authentication success, or may be configured to determine the final judgment result as authentication failure, or may be configured to determine the final judgment result as conditional authentication success.

Note that, for example, the processing executed after the authentication process is completed differs depending on whether the final determination result is authentication success or conditional authentication success. When payment processing is performed based on the results of the authentication process, for example, if the final determination result is conditional authentication success, only payment processing for amounts less than a specified amount can be executed, but if the final determination result is authentication success, payment processing for amounts above that specified amount can also be executed.

Next, the visual confirmation terminal 120 displays the final judgment result transmitted in step S310 on the display device 405 of the computer 400 that constitutes the visual confirmation terminal 120 (S311). The authentication terminal 110 displays the final judgment result transmitted in step S310 on the display device 405 of the computer 400 that constitutes the authentication terminal 110 (S312).

In this embodiment, the authentication server 130 converts the feature data extracted from the facial image during the registration process into auxiliary information AD and protected features PI using PIE, before registering them in the DB. It also applies a facial image protection transformation that creates feature dissimilarity to the facial image, before registering it in the DB. This prevents spoofing attacks, as even if all information were to be leaked from all DBs, it would be impossible to generate authentication protected features PI* from the leaked information that would allow successful authentication.

Furthermore, according to this embodiment, the facial image protection transformation has symmetric visual similarity (or asymmetric visual similarity), allowing the operator to visually compare the registered protected image with the authentication protected image (or the user's own facial features) and verify the identity of the user. In this way, this embodiment makes it possible to realize a highly secure facial authentication and identity verification system that allows the operator to visually verify the facial image while firmly protecting feature data for facial authentication from leakage and misuse for impersonation.

Furthermore, the above-mentioned facial image protection transformation is a technology that aims to cause (machine) matching of feature data extracted from facial images to fail, and is a technology with a different purpose from user authentication, which assumes that matching of feature data extracted from facial images may be successful. Therefore, there are obstacles to applying facial image protection transformation to BTP. However, in this embodiment, by applying facial image protection transformation to BTP, it is possible to realize a facial authentication and identity verification system that achieves the effects described above.

The present invention is not limited to the above-described embodiments, and includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

Furthermore, the above-mentioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole in hardware, for example by designing them as integrated circuits. Furthermore, the above-mentioned configurations, functions, etc. may be realized in software by a processor interpreting and executing programs that realize each function. Information such as programs, tables, and files that realize each function can be stored in memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.

Furthermore, the control and information lines shown are those deemed necessary for explanation, and do not necessarily represent all control and information lines on the product. In reality, it is safe to assume that almost all components are interconnected.

100 Registration terminal, 110 Authentication terminal, 120 Visual confirmation terminal, 130 Authentication server, 132 Facial feature extraction function, 133 Registration feature protection conversion function, 134 Facial image protection conversion function, 135 Protected image determination function, 136 Encryption function, 137 Authentication feature protection conversion function, 138 Protected feature matching function, 140 Final determination function, 141 Registration search function, 142 Auxiliary information DB, 143 Protected feature DB, 144 Encrypted protected image DB, 400 Computer, 401 CPU, 402 Memory, 403 Auxiliary storage device

Claims (15)

An information processing device,
a processor and a storage device;
The storage device
A facial image for registration;
registration feature data indicating features of the registration face image extracted from the registration face image;
a database in which protection features indicating protected features of a facial image, auxiliary information used for authentication using the protection features, and a protection image generated based on a predetermined protection transformation of the facial image are registered;
The processor:
performing a one-way transformation based on a predetermined algorithm on the feature data for registration to generate a protected feature for registration, which is the protected feature for the feature data for registration, and the auxiliary information for the feature data for registration;
generating a registration protected image, which is the protected image for the registration face image, based on the registration face image and the predetermined protection transformation;
an information processing device that registers the generated auxiliary information, the protection feature for registration, and the protection image for registration in the database; 2. The information processing device according to claim 1,
The processor:
extracting protected image characteristic data indicating characteristics of the protected image from the protected image;
generating a protected image protection feature based on the protected image feature data and the auxiliary information;
The information processing device compares the protection feature for registration with the protection feature of the protected image, and if the comparison fails, registers the protected image for registration in the database. 2. The information processing device according to claim 1,
The processor:
extracting protected image characteristic data indicating characteristics of the protected image from the protected image;
calculating a similarity between the registration feature data and the protected image feature data;
If the similarity is equal to or smaller than a predetermined threshold, the information processing device registers the protected image for registration in the database. 2. The information processing device according to claim 1,
The storage device
A face image for authentication;
authentication feature data indicating features of the authentication face image extracted from the authentication face image;
The processor:
generating an authentication protection feature based on the authentication feature data and the auxiliary information;
performing protection feature matching to match the registration protection feature registered in the database with the authentication protection feature;
outputting a protected image for registration registered in the database;
An information processing device that outputs an authentication result based on the result of the protection feature matching. 5. The information processing device according to claim 4,
The processor:
generating an authentication protected image, which is the protected image for the authentication face image, based on the authentication face image and the predetermined protection transformation;
an information processing device that outputs the authentication protected image; 6. The information processing device according to claim 5,
The processor:
extracting, from the protected image for authentication, protected image characteristic data indicating characteristics of the protected image for authentication;
performing the unidirectional transformation based on the predetermined algorithm on the authentication feature data to generate the protection feature for the authentication feature data and the auxiliary information for the authentication feature data;
generating a protected image feature for authentication based on the protected image feature data for authentication and the auxiliary information for the feature data for authentication;
an information processing device that compares the protection feature for the authentication feature data with the protection feature for the authentication protected image, and outputs the authentication protected image if the comparison fails; 6. The information processing device according to claim 5,
The processor:
extracting, from the protected image for authentication, protected image characteristic data indicating characteristics of the protected image for authentication;
calculating a similarity between the authentication feature data and the authentication protected image feature data;
When the degree of similarity is equal to or smaller than a predetermined threshold, the information processing device outputs the protected image for authentication. 5. The information processing device according to claim 4,
The processor:
obtaining a result of the identity determination made by an operator through visual confirmation of the output protected image for registration;
an information processing device that outputs the authentication result based on the determination result and the result of the protection feature matching; 5. The information processing device according to claim 4,
The processor:
generating an image by performing the predetermined protection transformation on the face image for registration, and encrypting the generated image using the protection feature for registration as an encryption key, thereby generating the protected image for registration;
decrypting the protected image for registration registered in the database using the protected feature for authentication as a decryption key;
an information processing device that outputs the decoded registration image; 2. The information processing device according to claim 1,
the predetermined protection transformation has feature dissimilarity and symmetric visual similarity;
The feature dissimilarity is a property indicating that the facial images before and after the predetermined protection transformation are not similar based on a predetermined condition,
The symmetric visual similarity is a property indicating that a combination of the protected images for the facial images included in the combination has the visual similarity only when the combination of facial images before the specified protection transformation is applied has the visual similarity. 2. The information processing device according to claim 1,
the predetermined protection transformation has feature dissimilarity and asymmetric visual similarity;
The feature dissimilarity is a property indicating that the facial images before and after the predetermined protection transformation are not similar based on a predetermined condition,
The asymmetric visual similarity is a property indicating that one facial image included in a combination of facial images before the specified protection transformation has visual similarity with the protected image for the other facial image included in the combination, only when the combination has visual similarity. 2. The information processing device according to claim 1,
the predetermined protection transformation comprises feature dissimilarity and visual similarity preserving;
The feature dissimilarity is a property indicating that the facial images before and after the predetermined protection transformation are not similar based on a predetermined condition,
The preserved visual similarity is a property indicating that, only when a combination of face images before the specified protection transformation has visual similarity, one face image included in the combination and the protection image for that face image have the visual similarity, and also the other face image included in the combination and the protection image for the other face image have the visual similarity. an information processing device according to any one of claims 1 to 12;
a registration terminal that captures the face image for registration and transmits it to the information processing device;
a visual confirmation terminal that receives the registration protected image registered in the database from the information processing device and displays it. An information processing method by an information processing device,
the information processing device includes a processor and a storage device;
The storage device
A facial image for registration;
registration feature data indicating features of the registration face image extracted from the registration face image;
a database in which protection features indicating protected features of a facial image, auxiliary information used for authentication using the protection features, and a protection image generated based on a predetermined protection transformation of the facial image are registered;
The information processing method includes:
the processor performs a one-way transformation on the registration feature data based on a predetermined algorithm to generate a registration protection feature that is the protection feature for the registration feature data and the auxiliary information for the registration feature data;
the processor generates a registration protected image, which is the protected image for the registration face image, based on the registration face image and the predetermined protection transformation;
The information processing method, wherein the processor registers the generated auxiliary information, the registration protection feature, and the registration protection image in the database. An information processing program that causes the information processing device to execute the information processing method described in claim 14.