KR20260032119A - Electronic device, method and non-volatile computer readable strorage medium for performing user authentication - Google Patents

Abstract

An electronic device includes at least one processor including a memory storing instructions, a communication circuit, and a processing circuit, wherein the instructions, when individually or collectively executed by the at least one processor, cause the electronic device to, when first user information is collected, obtain a user authentication result based on the first user information using a first artificial intelligence model, quantify the user authentication result to obtain a first score, identify a user authentication level based on the first score, and, if the user authentication level is the first level, obtain first user status information based on the first user information using a second artificial intelligence model, and, if the user authentication level is a second level higher than the first level, transmit the first user status information to the outside of the electronic device.

Description

ELECTRONIC DEVICE, METHOD AND NON-VOLATILE COMPUTER READABLE STRORAGE MEDIUM FOR PERFORMING USER AUTHENTICATION

The present disclosure relates to an electronic device, a method, and a non-transitory computer-readable storage medium for performing user authentication.

Recently, the spread of various types of portable electronic devices such as smartphones, tablet PCs, wireless earphones, and smart watches is increasing.

The Internet of Things (IoT) is a technology that connects various objects to the Internet by embedding sensors and communication capabilities into them. In other words, it connects various objects via wireless communication. "Objects" here can include various embedded systems, such as home appliances, mobile electronic devices, and wearable devices. Objects connected to the IoT must have a unique IP address that identifies them and be connected to the Internet. They can also incorporate sensors to acquire data from the external environment.

The recent expansion of IoT has expanded connectivity between users and their surroundings, and has provided users with greater convenience through interaction between objects. To empower users with control over IoT, portable electronic devices have traditionally been used as control controllers.

The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art related to the present disclosure.

According to one embodiment, an electronic device includes: a memory storing instructions; a communication circuit; and at least one processor including a processing circuit; wherein the instructions, when individually or collectively executed by the at least one processor, cause the electronic device to, when first user information is collected, obtain a user authentication result based on the first user information using a first artificial intelligence model, quantify the user authentication result to obtain a first score, identify a user authentication level based on the first score, and, if the user authentication level is the first level, obtain first user status information based on the first user information using a second artificial intelligence model, and, if the user authentication level is a second level higher than the first level, transmit the first user status information to the outside of the electronic device.

According to one embodiment, a method for controlling an electronic device includes: when first user information is collected, obtaining a user authentication result based on the first user information using a first artificial intelligence model; obtaining a first score by digitizing the user authentication result; identifying a user authentication level based on the first score; if the user authentication level is the first level, obtaining first user status information based on the first user information using a second artificial intelligence model; and if the user authentication level is a second level higher than the first level, transmitting the first user status information to the outside of the electronic device.

In one embodiment, a non-transitory computer-readable medium storing instructions that, when executed by a processor of an electronic device, cause the electronic device to perform an operation, the operations include: when first user information is collected, obtaining a user authentication result based on the first user information using a first artificial intelligence model; obtaining a first score by digitizing the user authentication result; identifying a user authentication level based on the first score; when the user authentication level is the first level, obtaining first user status information based on the first user information using a second artificial intelligence model; and when the user authentication level is a second level higher than the first level, transmitting the first user status information to the outside of the electronic device.

The above and other aspects, features and advantages of specific embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings.
FIG. 1 is a diagram for explaining an IoT (internet of things) system according to one embodiment.
FIG. 2 illustrates an example of a block diagram of an electronic device according to one embodiment.
Figure 3 is a flowchart illustrating the operation of an electronic device according to one embodiment.
FIG. 4A is a block diagram illustrating in detail the operation of an electronic device according to one embodiment.
FIG. 4b is a diagram illustrating an example of a generative artificial intelligence model according to one embodiment.
Figure 5 is a flowchart for explaining a method for designing an optimal IoT environment according to one embodiment.
FIG. 6 is a diagram according to an example for explaining a method for designing an optimal IoT environment according to one embodiment.
FIG. 7 is a diagram according to an example for explaining a method for designing an optimal IoT environment according to one embodiment.
FIGS. 8A to 8E are drawings according to an example for explaining a method for designing an optimal IoT environment according to one embodiment.
FIG. 9 is a diagram according to an example for explaining a first score calculation method according to one embodiment.
FIG. 10 is a drawing according to an example for explaining a method for designing an optimal IoT environment according to one embodiment.
FIG. 11 is a diagram according to an example for explaining a method for designing an optimal IoT environment according to one embodiment.
FIGS. 12A to 12D are drawings for explaining examples of UI (user interface) screens provided in an electronic device according to one embodiment.
FIG. 13 is a drawing for explaining a method for providing an IoT environment in a vehicle according to one embodiment.
FIG. 14 is a block diagram of an electronic device within a network environment according to various embodiments.

Hereinafter, the present disclosure will be described in detail with reference to the attached drawings.

The terms used in the embodiments of this disclosure have been selected from widely used, current terms, taking into account the functions of this disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, or the emergence of new technologies. Furthermore, in certain cases, the applicant may arbitrarily select terms, in which case their meanings will be described in detail in the description of this disclosure. Therefore, the terms used in this disclosure should not be defined simply as names (analyzing calls, messages, and schedules), but rather based on the meanings of the terms and the overall content of this disclosure.

In this specification, expressions such as “has,” “can have,” “includes,” or “may include” indicate the presence of a feature (e.g., a component such as a number, function, operation, or part), and do not exclude the presence of additional features.

The expression "at least one of A and/or B" should be understood to mean either "A" or "B" or "A and B".

As used herein, expressions such as “first,” “second,” “first,” or “second,” can describe various components, regardless of order and/or importance, and are only used to distinguish one component from another, but do not limit said components.

When it is said that a component (e.g., a first component) is “(operatively or communicatively) coupled with/to” or “connected to” another component (e.g., a second component), it should be understood that the component may be directly coupled to the other component, or may be coupled via another component (e.g., a third component).

Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as "comprise" or "consist of" are intended to specify the presence of a feature, number, operation, operation, component, part, or combination thereof described in the specification, but should be understood not to preclude the possibility of the presence or addition of one or more other features, numbers, operations, operations, components, parts, or combinations thereof.

In the embodiments, a "module" or "part" performs at least one function or operation and may be implemented in hardware or software, or a combination of hardware and software. Furthermore, multiple "modules" or multiple "parts" may be integrated into at least one module and implemented as at least one processor, excluding any "modules" or "parts" that need to be implemented as specific hardware.

In this disclosure, the term user may refer to a person using an electronic device or a device (e.g., an artificial intelligence electronic device) using an electronic device.

The various elements and areas in the drawings are schematically drawn. Therefore, the technical concept of the present invention is not limited by the relative sizes or spacings drawn in the attached drawings.

Hereinafter, embodiments of the present disclosure will be described in more detail with reference to the attached drawings.

FIG. 1 is a diagram illustrating an IoT system according to one embodiment.

According to FIG. 1, the electronic system may include a plurality of devices (10, 20, 30), a user terminal (40), a server (50), and an AP (access point) device (60).

The plurality of devices (10, 20, 30) may be various IoT (Internet of Things) devices managed by a server (50). For example, the plurality of devices (10, 20, 30) may be implemented as various home appliances such as an air conditioner, a television, an air purifier, a washing machine, a refrigerator, a dryer, and an oven. According to an example, the plurality of devices (10, 20, 30) may be controlled and/or managed through an application installed on a user terminal (40).

For example, the plurality of devices (10, 20, 30) may be Internet of Things (IoT) devices supporting a WiFi module. The WiFi module may include WiFi firmware and a WiFi driver that allows the operating system of the electronic device (100) to recognize the WiFi firmware. Here, the WiFi firmware (or wireless network card) may be hardware and may be mounted inside the main body of the plurality of devices (10, 20, 30) or may be mounted to the plurality of devices (10, 20, 30) using an interface (for example, a universal serial bus (USB). For example, the WiFi firmware may be implemented in at least one of a peripheral component interconnect (PCI), a PCI-Express, a personal computer memory card international association (PCMCIA), or a universal serial bus (USB).

The user terminal (40) can download and install an application from a server that provides the application. For example, the application may be an IoT application for registering and managing multiple devices (10, 20, 30) on the server (50). The application is software that the user directly uses on the operating system (OS) and may be provided in the form of an icon interface on the screen of the user terminal (40). For example, the user can execute an application (hereinafter referred to as an IoT application) on the user terminal (40), enter a user account, and log in to the server (50) using the entered user account, and the user terminal (40) can communicate with the server (50) based on the logged-in user account.

The server (50) can manage multiple devices (10, 20, 30) registered to a user account. Here, the server (50) can be implemented as a cloud server, but is not limited thereto.

According to an example, a user terminal (40) can use Easy Setup technology to connect multiple devices (10, 20, 30) to communicate and register them with a server (50). The "Easy Setup" technology may be a function that generally simplifies the process of initially setting up an electronic device, thereby helping a user to use the device quickly and easily. According to the Easy Setup technology, multiple devices (10, 20, 30) can be operated in AP mode, and a beacon signal including identification information (e.g., SSID (service set identifier)) transmitted by the multiple devices (10, 20, 30) can be scanned by the user terminal (40). In this case, the user terminal (40) can provide a UI screen including identification information corresponding to the scanned beacon signal, and when one of the identification information is selected, the user terminal (40) can communicate and connect with a device corresponding to the selected identification information. The user terminal (40) transmits information necessary for onboarding to multiple devices (10, 20, 30) via a communication connection, thereby facilitating Internet connection and server (50) registration of the multiple devices (10, 20, 30). Onboarding may be a process in which a user using an electronic device for the first time becomes familiar with the functions and usage methods of the service.

When multiple devices (10, 20, 30) are registered to a user account through Easy Setup technology, the server (50) can transmit data received from the multiple devices (10, 20, 30) to a user terminal (40) that communicates with the server (50) based on the user account in which the multiple devices (10, 20, 30) are registered. In addition, when the server (50) receives a control command for the multiple devices (10, 20, 30) from the user terminal (40), the server (50) can transmit a control signal corresponding to the received control command to the multiple devices (10, 20, 30). In this case, a user terminal (40) implemented as a smartphone, a mobile phone, a tablet, or a laptop can access a server (50) through a mobile communication network such as LTE (long term evolution) or 5G NR (new radio), or can access the server (50) through an AP device (60) or another AP device, and can transmit control commands for multiple devices (10, 20, 30) to the server (50) through an IoT application installed on the user terminal (40).

For example, a server (50) can manage multiple devices (10, 20, 30) located in spaces (memorized places) preset (or pre-registered) by a user. The preset spaces can be various spaces such as a home, a car, a work space, or a user's desk within a company. For example, the preset spaces can be registered to the server (50) via a user terminal (40).

For example, when the server (50) receives user status information from the user terminal (40), the server (50) can control the IoT device (10, 20, 30) based on the received user status information and information about the IoT device (10, 20, 30). For example, the server (50) can obtain IoT device control information using a predefined algorithm, a predefined formula, and/or an artificial intelligence model. For example, the user status information can include at least one of fatigue, exercise volume, discomfort index, and regularity.

Below, various embodiments of obtaining user status information from a user terminal (40) to automatically control a preset space into an environment optimal for the user's status will be described.

FIG. 2 illustrates an example of a block diagram of an electronic device (100) according to one embodiment.

According to various embodiments, the electronic device (100) of FIG. 2 may be at least partially similar to the electronic device (1401) of FIG. 14, or may include other embodiments of the electronic device.

In one embodiment, in terms of being owned by a user, the electronic device (100) may be referred to as a terminal (or user terminal). The terminal may include, for example, a personal computer (PC) such as a laptop or desktop. The terminal may include, for example, a smartphone, a smartpad, and/or a tablet PC. The terminal may include a smart accessory such as a smartwatch and/or a head-mounted device (HMD). According to one embodiment, the electronic device (100) may include a deformable housing. Based on the deformability, the housing of the electronic device (100) may be divided into a plurality of parts. According to one example, the electronic device (100) may be implemented as the user terminal (40) illustrated in FIG. 1.

According to one embodiment, the electronic device (100) may include at least one of a processor (110), a memory (120), a communication circuit (130), a sensor (140), or a display (150). The processor (110), the memory (120), the communication circuit (130), the sensor (140), and the display (150) may be electronically and/or operably coupled with each other by an electronic component, such as a communication bus.

In one embodiment, the hardware of the electronic device (100) being operatively coupled may mean that a direct connection or an indirect connection is established between the hardwares, either wired or wireless, so that the second hardware is controlled by the first hardware among the hardwares. Although illustrated based on different blocks, the embodiment is not limited thereto, and some of the hardware of FIG. 2 (e.g., at least a portion of the processor (110), the memory (120), and the communication circuit (130)) may be included in a single integrated circuit such as a system on a chip (SoC). The type and/or number of hardware included in the electronic device (100) is not limited to that illustrated in FIG. 2. For example, the electronic device (100) may include only some of the hardware components illustrated in FIG. 2.

According to one embodiment, the processor (110) of the electronic device (100) may include hardware for processing data based on one or more instructions. The hardware for processing data may include, for example, an arithmetic and logic unit (ALU), a floating point unit (FPU), a field programmable gate array (FPGA), a central processing unit (CPU), and/or an application processor (AP). The number of processors (110) may be one or more. For example, the processor (110) may have a multi-core processor structure such as a dual core, a quad core, or a hexa core.

The processor (110) can control the operations of the electronic device (100) by executing instructions stored in the memory (120). For example, the processor (110) can correspond to a plurality of processors that collectively perform a plurality of operations by dividing them among the processors.

A CPU (central processing unit) is a general-purpose processor capable of performing not only general calculations but also artificial intelligence calculations. Its multi-layer cache structure allows for the efficient execution of complex programs. CPUs are advantageous for serial processing, enabling organic linking of previous and subsequent calculation results through sequential calculations. A general-purpose processor is not limited to the examples described above, except where specifically designated as a CPU.

A GPU (graphics processing unit) is a processor designed for large-scale computations, such as floating-point operations used in graphics processing. It integrates a large number of cores to perform large-scale computations in parallel. In particular, GPUs may be advantageous over CPUs in parallel processing methods, such as convolution operations. Furthermore, GPUs can be used as coprocessors to supplement the functions of CPUs. Processors for large-scale computations are not limited to the examples described above, except in cases where the aforementioned GPU is specifically mentioned.

An NPU (Neural Processing Unit) is a processor specialized in artificial intelligence operations using artificial neural networks, and each layer of the artificial neural network can be implemented in hardware (e.g., silicon). Since the NPU is designed specifically according to the company's specifications, it has less freedom than a CPU or GPU, but can efficiently process the artificial intelligence operations requested by the company. Meanwhile, as a processor specialized in artificial intelligence operations, an NPU can be implemented in various forms, such as a TPU (Tensor Processing Unit), an IPU (Intelligence Processing Unit), and a VPU (Vision Processing Unit). Except for cases where the NPU is specifically mentioned above, the artificial intelligence processor is not limited to the examples described above.

According to one embodiment, the memory (120) of the electronic device (100) may include a hardware component for storing data and/or instructions input and/or output to the processor (110). The memory (120) may include, for example, a volatile memory such as a random-access memory (RAM) and/or a non-volatile memory such as a read-only memory (ROM). The volatile memory may include, for example, at least one of a dynamic RAM (DRAM), a static RAM (SRAM), a cache RAM, and a pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), a flash memory, a hard disk, a compact disc, a solid state drive (SSD), and an embedded multimedia card (eMMC).

According to one embodiment, one or more instructions (or commands) representing operations and/or actions to be performed on data by the processor (110) may be stored in the memory (120) of the electronic device (100). A set of one or more instructions may be referred to as firmware, an operating system, a process, a routine, a sub-routine, and/or an application. For example, the electronic device (100) and/or the processor (110) may perform various operations when a set of a plurality of instructions distributed in the form of an operating system, firmware, a driver, and/or an application is executed. Hereinafter, the fact that an application is installed in an electronic device (100) may mean that one or more instructions provided in the form of an application are stored in the memory (120) of the electronic device (100), and that the one or more applications are stored in a format executable by the processor (110) of the electronic device (100) (e.g., a file having an extension specified by the operating system of the electronic device (100).

One or more processors (110) control input data to be processed according to predefined operation rules or artificial intelligence (AI) models stored in memory (120). The predefined operation rules or AI models are characterized by being created through learning. Being created through learning means that a predefined operation rule or AI model with desired characteristics is created by applying a learning algorithm to a plurality of learning data. This learning may be performed in the device itself on which the artificial intelligence according to the present disclosure is performed, or may be performed through a separate server/system.

An AI model may be composed of multiple neural network layers. At least one layer has at least one weight value and performs its own operation through the operation result of the previous layer and at least one defined operation. Examples of neural networks include a convolutional neural network (CNN), a recurrent neural network (RNN), a deep neural network (DNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), deep Q-networks, and a Transformer. The neural networks in the present disclosure are not limited to the above-described examples unless otherwise specified.

A learning algorithm is a method for training a target device (e.g., a robot) using a large amount of learning data, enabling the target device to make decisions or predictions on its own. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. Unless otherwise specified, the learning algorithms in this disclosure are not limited to the aforementioned examples.

A communication circuit (130) of an electronic device (100) according to an embodiment may include hardware for supporting transmission and/or reception of electrical signals between the electronic device (100) and an external device (e.g., a server). The communication circuit (130) may include, for example, at least one of a modem (modulator and demodulator), an antenna, and an optical/electronic (O/E) converter. The communication circuit (130) may support transmission and/or reception of electrical signals based on various types of protocols, such as Ethernet, a local area network (LAN), a wide area network (WAN), wireless fidelity (WiFi), near field communication (NFC), Bluetooth, Bluetooth low energy (BLE), ZigBee, long term evolution (LTE), 5G NR (new radio), and/or 6G.

In one example, the electronic device (100) may be connected to a server based on a wired network and/or a wireless network. The wired network may include a network such as the Internet, a local area network (LAN), a wide area network (WAN), Ethernet, or a combination thereof. The wireless network may include a network such as long term evolution (LTE), 5G new radio (NR), wireless fidelity (WiFi), Zigbee, near field communication (NFC), Bluetooth, Bluetooth low-energy (BLE), or a combination thereof. In one example, the electronic device (100) and the server may be indirectly connected via an intermediate node within the network.

A sensor (seonsor) (140) of an electronic device (100) according to an embodiment can sense various user information. The sensor (140) can be implemented with various types of sensors capable of user sensing. For example, the sensor (140) can include at least one sensor from among a ToF (time of flight) sensor, an ultrasonic sensor, a RADAR (radio detection and ranging) sensor, a photodiode sensor, a proximity sensor, a PIR (passive infrared sensor) sensor, a pin hole sensor, a pin hole camera, an infrared human body detection sensor, a CMOS (complementary metal oxide semiconductor) image sensor, a heat detection sensor, a light sensor, and a motion detection sensor.

The sensor (140) may include a touch sensor that detects a touch action in the form of a touch film, a touch sheet, or a touch pad.

The sensor (140) may include at least one of a camera, a microphone, a _CO2 sensor, and a barometric pressure sensor. The camera may convert a captured image into an electrical signal and generate image data based on the converted signal. For example, the camera may include at least one of a general (or basic) camera, a depth camera, and an ultra-wide-angle camera. The microphone is configured to receive user voice or other sounds and convert them into audio data. The _CO2 sensor is a sensor for measuring carbon dioxide concentration. The barometric pressure sensor is a sensor for sensing the surrounding pressure.

The sensor (140) may further include at least one sensor capable of sensing ambient illuminance, ambient temperature, and the direction of incident light. In this case, the sensor (140) may be implemented as an illuminance sensor, a temperature detection sensor, a light intensity sensing layer, and a camera.

The sensor (140) may further include at least one of an acceleration sensor (or gravity sensor), a geomagnetic sensor, and a gyro sensor. For example, the acceleration sensor may be a three-axis acceleration sensor. The three-axis acceleration sensor may measure gravitational acceleration for each axis and provide raw data to the processor (140). The geomagnetic sensor or gyro sensor may be used to obtain attitude information. Here, the attitude information may include at least one of roll information, pitch information, and yaw information.

According to one embodiment, the display (150) of the electronic device (100) can output visualized information to the user. For example, the display (150) can be controlled by a controller such as a GPU (graphic processing unit) to output visualized information to the user. The display (150) can include an OLED (organic light emitting diodes) display, LED (light emitting diodes), micro LED (micro LED), mini LED, PDP (plasma display panel), QD (quantum dot) display, QLED (quantum dot light-emitting diodes), and/or an e-ink display and/or an e-paper display. According to one example, the display (150) can be implemented as a flat display, a curved display, a foldable and/or rollable flexible display.

FIG. 3 is a flowchart for explaining the operation of an electronic device (100) according to one embodiment.

In the following examples, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

According to one embodiment, operations 310 to 370 may be understood to be performed in the processor (110) of the electronic device (100).

According to FIG. 3, in operation 310, an electronic device (100) according to one embodiment can collect first user information.

For example, the electronic device (100) may collect first user information from the time a preset event occurs and/or from a specific time point. User information may be collected in real time, and user information collected at the current time point is referred to as first user information to distinguish it from user information collected at a later time point. For example, the later time point may include at least one of a time point after a preset time from the current time point or a time point after a preset event occurs from the current time point. For example, the preset event may include an event in which the first user information collected at the current time point is processed (e.g., processed for authentication) by the electronic device (100).

The first user information may include at least one of behavioral information, biometric information, voice information, image information, touch information, app execution information, health information, environmental information, time information, task information, feedback information, and message/call information. For example, behavioral information may include information on various actions and activities performed by the user in a digital environment. For example, behavioral information may include information such as clickstream data, search history, session data, user interaction data, content consumption data, and social media activity. For example, biometric information may include health-related data or physiological information such as heart rate, sleep patterns, and activity level through a wearable device. For example, voice information may include voice assistant usage information, voice call information, and voice recording information. For example, image information may include body image information including facial image information. For example, touch information may include information on touch location, touch pressure, and touch size. For example, app execution information may include the type of app being used, the execution time, and operating system information. For example, time information may include time-related information such as the current time, day of the week, and season. For example, environmental information may include information about the surrounding environment, such as weather, temperature, and lighting. For example, work information may analyze calls, messages, and schedules, and may include workload. For example, feedback information may include feedback on apps, websites, products, or services.

According to one embodiment, the electronic device (100) may preprocess the collected first user information. Preprocessing may be a process of organizing and transforming raw data to analyze the data using an artificial intelligence model. For example, the electronic device (100) may preprocess the first user information through at least one of data cleaning, data transformation, feature selection, data splitting, data augmentation, and data integration. For example, in the case of a facial image, preprocessing may be performed by cropping the image centered on facial features, histogram equalization, image gamma correction, and Gaussian filtering. However, this is merely an example, and various preprocessing may be performed depending on the type of user information. The first user information input into the first artificial intelligence model below may be preprocessed user information, but for the convenience of explanation, it is collectively referred to as first user information.

For example, the electronic device (100) may collect first user information outside of a preset space. The preset space may be a space preset by the user for IoT environment control. The preset space (or memorized place) may be various spaces such as a home, a car, a work space, or a user's desk within a company. Information about the preset space may be stored in the memory (120). For example, information about the preset space, including at least one of location information, barometric pressure information, geomagnetic information, and Wi-Fi information, may be stored in the memory (120).

In operation 320, an electronic device (100) according to an embodiment may obtain a first user authentication result based on first user information using a first artificial intelligence model.

According to one example, the electronic device (100) may input user authentication information among the first user information into the first artificial intelligence model to obtain a first user authentication result. For example, the user authentication information may include at least one of behavioral information, biometric information, voice information, image information, touch information, or app execution information. For example, the electronic device (100) may obtain a prompt based on the user authentication information and input the obtained prompt into the first artificial intelligence model. The prompt may be a text input that instructs a specific algorithm or AI model to perform a desired task. According to one example, the electronic device (100) may obtain a prompt based on the user authentication information according to a preset rule. Rule-based prompt generation may be a method of generating a prompt using at least one of predefined rules, patterns, formats, and templates.

For example, the first artificial intelligence model may be trained to compare previously authenticated (or previously registered or previously stored) user information with information for user authentication, and output a first user authentication result including a user match. The previously authenticated user information may be information confirmed to correspond to a user of the electronic device (100). For example, the previously authenticated user information may include at least one of behavioral information, biometric information, voice information, image information, touch information, or app execution information corresponding to the user of the electronic device (100).

For example, the first AI model may compare a prompt corresponding to the input user authentication information with previously authenticated user information to output a user match (or user accuracy). For example, if the gait pattern of a previously authenticated user is stored as a number, the first AI model may convert the input gait information into a number and compare it with the previously authenticated gait pattern. The user match may be output in the form of probability information (e.g., a probability value) or correlation information (e.g., a correlation value), but is not necessarily limited thereto.

For example, a first AI model may include multiple first AI models for outputting user matching scores for each type of information for user authentication. Because it includes multiple first AI models, it may also be referred to as an AI package model.

In one example, multiple first AI models may be trained to output user matches for each type. For example, the multiple first AI models may include a first AI model trained to output a first user match by comparing first type information (e.g., behavioral information) with previously authenticated user information, and a first AI model trained to output a second user match by comparing second type information (e.g., biometric information) with previously authenticated user information.

In operation 330, the electronic device (100) according to an embodiment may digitize the first user authentication result to obtain a first score. According to an example, the electronic device (100) may initiate an operation of digitizing the first user authentication result based on at least one of an event in which the electronic device (100) is located within a preset space, an event in which biometric authentication (e.g., fingerprint, iris, facial recognition) is performed, and an event in which the first user information collected in the memory (120) is greater than or equal to a preset amount. For example, the preset amount may be set based on at least one of the capacity of the memory (120), the processing performance of the electronic device (100) (e.g., the processor (110)), and the processing performance of the first artificial intelligence model. However, this is merely an example, and the digitization operation may also be performed according to a preset time cycle.

For example, the electronic device (100) may convert the user agreement (e.g., probability value or correlation value) output from the first artificial intelligence model into a numerical value within a preset range to obtain a first score. For example, the electronic device (100) may convert the user agreement into a preset quantitative value.

For example, the electronic device (100) may obtain a first score by quantifying a plurality of user matches output by type of information for user authentication from a plurality of first artificial intelligence models. For example, the electronic device (100) may convert a plurality of user matches output from a plurality of first artificial intelligence models into numerical values within a preset range and add up the converted numerical values to obtain a first score. For example, the electronic device (100) may apply the same weight to the plurality of user matches to convert them into numerical values within a preset range. For example, the electronic device (100) may apply different weights to the plurality of user matches to convert them into numerical values within a preset range.

According to another example, the user agreement (e.g., probability value or correlation value) output from the first artificial intelligence model of the electronic device (100) may be in the form of a predefined quantitative value. In this case, the electronic device (100) may obtain a first score based on the numerical value output from the first artificial intelligence model. For example, a plurality of first artificial intelligence models may be trained to output a value digitized as a predefined quantitative value for each type of information for user authentication. In this case, the electronic device (100) may obtain a first score by calculating (e.g., adding) the numerical values output from the plurality of first artificial intelligence models.

In operation 340, the electronic device (100) according to one embodiment can identify a user authentication level based on the first score.

User authentication levels can be various security actions established to verify a user's identity and/or authorize a system or service. User authentication levels are divided into multiple authentication levels (or actions), and each action can represent a different level of security.

For example, the electronic device (100) may identify one of a plurality of authentication levels based on the first score. For example, the plurality of authentication levels may include a 0th level, a 1st level, and a 2nd level. The reliability of authentication may increase in the order of 0th level, 1st level, and 2nd level. Level 0 may be a level where basic user authentication is difficult and user authentication must be re-performed. Level 1 may be a level where user information can be utilized within the electronic device (100). Level 2 may be a level where user information can be transmitted outside the electronic device (100).

According to one embodiment, the electronic device (100) may perform different operations depending on the user authentication level. However, at a relatively higher authentication level, at least some of the operations performed at a relatively lower authentication level may be performed redundantly. According to one example, if the user authentication level is level 0, the electronic device (100) may identify the current user authentication as a failure and re-perform user authentication based on second user information collected thereafter.

In operation 350, the electronic device (100) according to one embodiment can identify whether the user authentication level is the first level. Whether the user authentication level is the first level may be whether the user authentication level satisfies the first level.

In operation 350, the electronic device (100) according to one embodiment can obtain first user status information based on first user information using a second artificial intelligence model if the user authentication level is the first level (or higher) (350:Y), in operation 360.

For example, the electronic device (100) may acquire first user status information by inputting information for analyzing the user's status among the first user information into a second artificial intelligence model. The information for analyzing the user's status may include at least one of health information, environmental information, time information, work information, feedback information, and message/call information. For example, the electronic device (100) may acquire a prompt based on the information for analyzing the user's status and input the acquired prompt into the second artificial intelligence model.

For example, the second artificial intelligence model may be trained to output user status information including multiple status items by applying weights according to the type of information for analyzing the user's status. For example, the second artificial intelligence model may apply different weights according to the type of information for analyzing the user's status. For example, the multiple status items may include numerical values for each status item. For example, the multiple status items may include at least one of fatigue, exercise volume, discomfort index, and regularity. For example, the second artificial intelligence model may be implemented as an LLM (large language model), but is not limited thereto.

LLM refers to an artificial neural network-based language model that has learned from a large amount of text data through pre-training. LLMs can contain significantly more parameters (e.g., over 10 billion) than typical language models.

In operation 350, the electronic device (100) according to one embodiment may not proceed with subsequent operations if the user authentication level does not satisfy the first level (is lower than the first level) (350:N). For example, the electronic device (100) may not acquire the first user status information.

In operation 370, the electronic device (100) according to one embodiment can identify whether the user authentication level is the second level. Whether the user authentication level is the second level may be determined by whether the user authentication level satisfies the second level.

In operation 370, if the user authentication level is the second level (or higher than the second level) (370:Y), the electronic device (100) according to one embodiment may transmit the first user state information to the outside of the electronic device (100) in operation 380. In one example, the electronic device (100) may transmit the first user state information to a server that manages an IoT (internet of things) device located in a preset space. For example, since the user authentication level is the second level, it also satisfies the condition of the first level, so the first user state information may be acquired. For example, the server may be implemented as the server (50) of FIG. 1, and thus, for the convenience of explanation, the following description will assume that the server is implemented as the server (50) of FIG. 1.

In operation 370, the electronic device (100) according to one embodiment may not transmit the first user status information to the server if the user authentication level does not satisfy the second level (is lower than the second level) (370:N).

In one example, a threshold value for distinguishing between multiple authentication levels based on a first score may be preset. For example, a first threshold value for distinguishing between level 0 and level 1 and a second threshold value for distinguishing between level 1 and level 2 may be preset. In one example, if the first score is less than the first threshold value, the electronic device (100) may identify it as level 0 and re-perform user authentication based on second user information collected thereafter. In one example, if the first score is equal to or greater than the first threshold value, the electronic device (100) may identify the user authentication level (or trustworthiness for the user) as level 1. In one example, if the first score is equal to or greater than the second threshold value, the electronic device (100) may identify the user authentication level as level 2.

The first threshold value and/or the second threshold value may be set at the time of manufacturing the electronic device (100) in consideration of at least one of the type of the electronic device (100), the hardware performance of the electronic device (100), the performance of the first artificial intelligence model and/or the second artificial intelligence model, update information of the first artificial intelligence model and/or the second artificial intelligence model, and the numerical conversion rule of the first artificial intelligence model. In one example, the first threshold value and/or the second threshold value may be settable and/or changeable by the user. In one example, the first threshold value and the second threshold value may be automatically adjusted based on at least one of the user's usage history of the electronic device (100) (e.g., usage time of the electronic device (100), usage habits), user feedback, and update information of the first artificial intelligence model and/or the second artificial intelligence model.

According to one embodiment, the electronic device (100) may transmit first user status information to the server (50) and then identify whether the user continues to use the electronic device (100). According to one example, if the use of the electronic device (100) is maintained, the electronic device (100) may maintain second level user authentication and transmit second user status information acquired based on second user information collected thereafter to the server (50). According to one example, if the use of the electronic device (100) is terminated, the electronic device (100) may maintain first level user authentication and store second user status information acquired based on second user information in the memory (120). Whether the user continues to use the electronic device (100) may be identified based on an activity timeout. For example, the electronic device (100) can identify whether the user is continuing to use the device through at least one of screen status detection, biometric signal detection, app or program usage pattern analysis, background activity detection, location information-based detection, and session hit detection. Session hit detection may be a technology that tracks requests (hits) coming into a web server to detect and analyze requests that occur during a session in which a user interacts with a website or application.

According to one embodiment, if the first score is greater than or equal to a first threshold and less than a second threshold in operation 330, the electronic device (100) may identify whether the electronic device (100) is located within a preset space based on information corresponding to the preset space and sensing information acquired through the sensor (140). If the electronic device (100) is identified as being located within the preset space, the electronic device (100) may transmit first user status information to the server (50). For example, the sensing information may include at least one of location information, atmospheric pressure information, geomagnetic information, and WiFi information. In this way, if the electronic device (100) is located within the preset space, even if the first score is less than the second threshold that satisfies the second level, reliability can be secured, and thus the first user status information can be shared with an external device, i.e., the server (50).

According to one embodiment, when information about an IoT device is received from the server (50), the electronic device (100) may obtain control information for controlling the IoT device based on first user status information and information about the IoT device, and transmit the obtained control information to the server (50). For example, the electronic device (100) may obtain control information corresponding to each IoT device based on information about the IoT device in order to provide an IoT environment that is optimal for the user's status. According to one example, the electronic device (100) may obtain IoT device control information using a predefined algorithm, a predefined formula, and/or an artificial intelligence model. For example, the electronic device (100) may input the first user status information and information about the IoT device into a third artificial intelligence model to obtain control information corresponding to each IoT device. However, as described above, the electronic device (100) transmits the first user status information to the server (50), and the server (50) can obtain control information for controlling the IoT device based on information about the IoT device managed by the server (50). The server (50) can obtain IoT device control information using a predefined algorithm, a predefined formula, and/or an artificial intelligence model.

FIG. 4a is a block diagram for explaining in detail the operation of an electronic device (100) according to one embodiment.

According to FIG. 4A, the electronic device (100) may include a sensor module (410), a sub-device module (420), an application module (430), an AI package module (440), a user authentication service module (450), a biometric authentication AI core module (460), a reliability module (470), a user status analysis module (480), and an IoT composition service module (490). For example, each module may be implemented with at least one software, at least one hardware, and/or a combination thereof.

For example, at least one of the sensor module (410), the sub-device module (420), the application module (430), the AI package module (440), the user authentication service module (450), the biometric authentication AI core module (460), the reliability module (470), the user state analysis module (480), and the IoT composition service module (490) may be implemented to use a predefined algorithm, a predefined formula, and/or an artificial intelligence model. The sensor module (410), the sub-device module (420), the application module (430), the AI package module (440), the user authentication service module (450), the biometric authentication AI core module (460), the reliability module (470), the user state analysis module (480), and the IoT composition service module (490) may be included within the electronic device (100), but may be distributed to at least one external device according to an example.

The sensor module (410) can collect user information based on sensing values from various sensors. For example, the sensor module (410) can collect user information from sensors that sense information about the movement of the electronic device (100) or the user's actions, such as a 6-axis sensor, a geomagnetic sensor, a barometric pressure sensor, or a global positioning system (GPS).

The sub-device module (420) can collect user information from various sub-devices. For example, the sub-device module (420) can collect user information received from wearable devices such as XR (extended reality), rings, and watches. For example, the sub-device module (420) can receive user information from the sub-device using a communication driver.

The application module (430) can control the processing of user information acquired through a specific app received from the sensor module (410) and/or the sub-device module (420), and the processing of user authentication in a specific app. For example, the application module (430) can process user information acquired through a voice app (e.g., a call app), a camera app, a user motion analysis app (e.g., a health app), and a wearable app. For example, the application module (430) can control on-device AI processing of user information acquired through a specific app (e.g., a voice app, a camera app). For example, the application module (430) can perform user authentication processing (e.g., a public certificate, i-PIN, 3rd party app authentication) within a specific app.

For example, the application module (430) may include at least one of a Voice App module (431), a Camera App module (432), a User Motion Analysis App module (433), a Wearable App module (434), and a public certificate, i-PIN, or 3rd party app authentication module (435).

When a user's voice is input through a microphone, such as a phone call or recording, the Voice App module (431) can distinguish an area containing a human voice among the sound signals through VAD (voice activity detection) and transmit it to the Speaker verification model (442) of the on-device AI of the AI package module (440).

The camera app module (432) can transmit the captured image to the On-device LVM package (443) when the captured image contains a human face (e.g., face recognition) when the user executes a camera function (including a 3rd party function).

The user behavior analysis App module (433) collects data for authenticating users through behavior analysis, and can learn the user's unique behavior patterns by analyzing touch patterns, user typing habits, and gait.

The Wearable App module (434) can process the user's biometric signals (e.g., heart rate, breathing pattern, brain waves, health information) transmitted from the sub-device module (420) to correspond to the input features of the biometric signal-based authentication model (462).

The public certificate, i-PIN, and 3rd party app authentication module (435) can receive information when a user performs i-PIN, public certificate, or phone authentication through a 3rd party app and transmit the information to the user authentication service module (450) to use it to maintain user authentication.

The AI package module (440) can analyze user information acquired through a specific app using an artificial intelligence model. For example, the AI package module (440) can perform feature-processing on user information acquired through a specific app, such as a voice app or a camera app, using an artificial intelligence model specialized for that app. Features represent characteristics or properties of data that the artificial intelligence model can learn, and feature-processing can include feature processing such as feature generation, feature transformation, and feature selection.

For example, the AI package module (440) may include at least one of an On-device LLM model (441), a Speaker verification model (442), and an On-device LVM package (443).

The on-device LLM model (441) is an on-device AI model implemented in a phone call or recording, and may be an AI model that features the input information of the Biometrics Authentication AI core module (460) by analyzing the user's tone, expression, and response to the other party.

The Speaker verification model (442) may be an AI model that features the input information of the Biometrics Authentication AI core through voice analysis in a situation where micro input is input.

The on-device LVM package (443) may be an AI model that features input information of the Biometrics Authentication AI core module (460) by checking whether a user is among the recognized human faces when the camera function is in operation.

The user authentication service module (450) can perform user authentication renewal, maintenance, and management. For example, the user authentication service module (450) requests analysis of user information from the biometric authentication AI core module (460), transmits the AI analysis results to the scoring module (451) for digitization, and determines whether to use the user information and transmit it outside the device to configure an IoT environment.

For example, the user authentication service module (450) may include at least one of a Scoring module (451), a UX module (452), and a Data manager (453).

Scoring module (451) may be a module that quantifies and scores the degree of user authentication through 3rd party app authentication, on-device AI package, biometric signal authentication AI core, and biometric authentication.

The UX module (452) may be a module that displays a pop-up on the display (150) to notify the user when user authentication fails.

The data manager (453) may be a managing module that transmits data featured in the application to the biometric authentication AI core module (460) or transmits the output of the biometric authentication AI core module (460) to the scoring model (451).

The biometric authentication AI core module (460) may include an artificial intelligence model that determines whether a user is using the electronic device (100) based on the user's biometric information. For example, the biometric authentication AI core module (460) may include an AI core model that inputs a biometric signal and outputs a user judgment.

According to an example, the biometric authentication AI core module (460) may include at least one of a Behavior Biometrics model (461) and a Biometric signal-based authentication model (462).

The Behavior Biometrics model (461) may be an AI model that infers a user based on the user's unique behavioral patterns learned by analyzing the user's touch patterns, typing habits, gait, repetitive movements, and slide/swipe movements when using an electronic device (100).

A biometric signal-based authentication model (462) may be an AI model that infers a user by using the user's heart rate, breathing pattern, and brain waves extracted from input information of a sub-device (e.g., a wearable device) as input for user authentication analysis.

The reliability module (470) stores information on user biometric authentication (e.g., fingerprint, iris, facial recognition), and when an input for user biometric authentication comes in, the user can be judged based on the degree of user matching based on the stored information.

The user status analysis module (480) may include a second artificial intelligence model that obtains digitized user status information based on user information. For example, the user status analysis module (480) may digitize fatigue, regularity, amount of exercise, and discomfort index for each body part based on user information such as health information, weather information, sleep information, user workload, number of steps, app execution information, and user feedback.

For example, the user state analysis module (480) may utilize the LLM (large language model) package. For example, the LLM package may be trained through transfer learning, which sets initial weights using a base model (e.g., a model previously trained on a large data set) and then performs fine-tuning based on new data. For example, the LLM package may analyze user state information by applying self-attention through a transformer encoder.

The IoT composition service module (490) can verify user authentication information when an electronic device (100) accesses an environment where IoT devices are built, and then transmit information to the IoT hub for performing operations of the IoT devices based on the type of IoT device transmitted by the IoT hub and the digitized user status information (e.g., fatigue, regularity, amount of exercise, discomfort index) analyzed by the user status analysis module (480). The IoT hub can be a platform (e.g., server (50)) that helps centrally manage and communicate IoT devices. The IoT hub can play a central role in allowing multiple IoT devices to collect data, transmit the data to the cloud, or communicate with other devices.

According to an example, the IoT composition service module (490) may include a Matching manager module (491) and a Certification module (492).

The matching manager module (491) may be a module that designs the operation of an IoT device to configure an environment suitable for the user based on numerical values.

The certification module (492) may be a module that determines whether a user is authenticated before transmitting data to the IoT hub.

FIG. 4b is a diagram for explaining a generative artificial intelligence model according to an embodiment of the present disclosure.

According to one embodiment, at least one of the first artificial intelligence model or the second artificial intelligence model may be implemented as a Generative AI Model (1005) as illustrated in FIG. 4b.

According to FIG. 4b, the User Query/Response Interface (1001) can receive user input. The user input may be in the form of natural language, images, and/or videos.

For example, the user input may include the user's voice received through a microphone. However, the present invention is not limited thereto, and the user input may also include text corresponding to the voice generated by a speech-to-text (STT) model in addition to the voice. In addition, context information may also be transmitted when the user input is transmitted. The context information may include various additional information at the time of the user input. For example, information about the application the user is currently using or information about the user's location. Furthermore, the user input may also be in a form that mixes the natural language, images, sounds, and context information described above. Furthermore, the user input may also be in a non-natural language form, such as selecting a menu.

The User Query/Response Interface (1001) can output the results of a generative artificial intelligence system to the user. The output can be in the form of natural language or specific content, and can also be provided in the form of an action requested by the user. The User Query/Response Interface (1001) can output the results of a generative artificial intelligence system to the user. The output can be in the form of natural language or specific content, and can also be provided in the form of an action requested by the user. For example, the User Query/Response Interface (1001) can output content generated by a Generative AI Model (1005) based on voice input received from the user.

The AI framework (1002) can receive user input and coordinate and control each component necessary to perform the user's intention based on the user's query.

User input received from the User Query/Response Interface (1001) can be transmitted to the Prompt design component (1002-1). The Prompt design component (1002-1) can be used to generate a prompt suitable for inputting the user input into an LLM (large language model) or LMM (large multimodal model). The Prompt design component (1002-1) can be an AI component that uses a machine learning algorithm or a neural network to develop better prompts over time. The Prompt design component (1002-1) can access knowledge repositories (1003) containing user preference data, a prompt library, and prompt examples based on the user input to generate a prompt, and transmit the generated prompt to the LLM or LMM.

The API/Plug-in management component (1002-2) can communicate with external information when there is a request for additional information when passing user input as input to the generative model. The API/Plug-in management component (1002-2) establishes a channel for communication with the outside of the AI Interface through the API, and can enable access to various data sources through the established channel. In addition, the API/Plug-in management component (1002-2) can request an action through the API if the application or service needs to perform an action that ultimately performs the user input, rather than an intermediate result. Information obtained from an external source (e.g., the Applications/service component (1004)) can be used to generate a prompt in the Prompt design component (1002-1) along with the user input, or can be passed as input to the generative model.

The Output Modification Component (1002-3) (or Refiner component) can fine-tune the output from the generative model. For example, the Output Modification Component (1002-3) can verify that the content generated through the LLM and/or LMM is not irrelevant, does not contain biased content, or does not contain harmful content. In addition, the Output Modification Component (1002-3) can determine the degree to which the result matches the user's desired result and, if necessary, can perform additional processing. The Output Modification Component (1002-3) can additionally configure and provide hints to the user to avoid undesired output.

Generative AI Model (1005) can generally refer to an artificial intelligence neural network that generates new types of data based on user input information. Generative AI Model (1005) can include an image-generating model and/or a language-generating model. Representative models for generating images include a generative adversarial network (GAN) and a variational autoencoder (VAE), and examples include a diffusion-based generative model that uses a VAE and a transformer structure. A language-generating model is a model trained to statistically output the most appropriate output based on input values, and representative examples include models such as CHAT-GPT 3 and CHAT-GPT 4 (e.g., including CHAT-GPT 4o). There is also an LMM (large multimodal model) that can recognize various types of data input, such as text, images, and voice, and generate new data corresponding to them.

According to an embodiment, when a prompt is input from a prompt design component (1002-1), the Generative AI Model (1005) generates content corresponding to the prompt based on the instruction, and can output the content through the User Query/Response Interface (1001).

Figure 5 is a flowchart for explaining a method for designing an optimal IoT environment according to one embodiment.

Hereinafter, a method for designing an optimal IoT environment will be described in detail with reference to FIGS. 6, 7, 8a to 8e, 9, and 10. FIG. 6 is a diagram according to an example for explaining a method for designing an optimal IoT environment according to an embodiment. FIG. 7 is a diagram according to an example for explaining a method for designing an optimal IoT environment according to an embodiment. FIGS. 8a to 8e are diagrams according to an example for explaining a method for designing an optimal IoT environment according to an embodiment. FIG. 9 is a diagram according to an example for explaining a first score calculation method according to an embodiment. FIG. 10 is a diagram according to an example for explaining a method for designing an optimal IoT environment according to an embodiment.

In the following examples, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

According to one embodiment, operations 505 to 555 may be understood to be performed by a processor (110) of an electronic device (100).

According to FIG. 5, in operation 505, an electronic device (100) according to an embodiment can register a memorized place based on a user input.

According to an example, a memorized place can be registered through a UI screen provided through an electronic device (100) as illustrated in FIG. 6. For example, when a memorized place is registered through a UI screen at a specific location, the electronic device (100) can obtain at least one of location information, barometric pressure information, geomagnetic information, and WiFi information of the specific location through various sensors (e.g., GPS, barometric pressure sensor, geomagnetic sensor, WiFi module), and store the obtained information in a memory (120). According to an example, a user can set a specific range as a memorized place based on GPS location information through the UI screen, and the range of the memorized place can be changed according to a user's selection.

In operation 510, the electronic device (100) according to one embodiment may collect user information. For example, the electronic device (100) may collect various user information from an area outside of a memorized place, as illustrated in FIG. 6.

For example, user information may include user information for user authentication and user information for user status analysis. For example, user information for user authentication may include voice signals, image data, sensor values, touch information, and app execution information. User information for user status analysis may include health information, time information, weather information, work information, user feedback, message/call content, and history information. However, user information for user authentication and user status analysis are not necessarily distinct, and at least some of the information may be used in duplicate for user authentication and user status analysis.

In operation 515, the electronic device (100) according to one embodiment may preprocess the collected user information. For example, the electronic device (100) may feature the collected user information according to input features that can be processed by the artificial intelligence model within the AI package module (440). For example, only a human voice can be extracted from a voice signal through voice activity detection (VAD), and only an area corresponding to a human face can be extracted from an image signal.

In operation 520, the electronic device (100) can identify whether user authentication is complete.

For example, as illustrated in FIG. 7, the electronic device (100) can perform user authentication when the user's location information comes within a certain distance (hereinafter, collectively referred to as a memorized region) (713) from a pre-registered memorized place (712). For example, the memorized place (712) can be set within a certain range based on pre-established location information (711).

User authentication may be the process of verifying the identity of a user attempting to access a system or service. Completion of user authentication in operation 520 may mean that the user's identity has been verified with a level of reliability sufficient to allow analysis of the collected user information. For example, the electronic device (100) may verify the user's identity through biometric authentication. However, this is not limited to this method, and if comparative information has been accumulated to the extent that the user's identity can be verified through other methods other than biometric authentication, the user's identity may be verified through other methods.

In operation 520, when user authentication is completed (520:Y), the electronic device (100) according to one embodiment can analyze the preprocessed user information in operation 525 to obtain a user authentication result (or user authentication degree).

In operation 520, if user authentication is not completed (520:N), the electronic device (100) may re-perform operation (510) of collecting user information. Incomplete user authentication may mean that the user's identity has not been confirmed with a level of reliability sufficient to analyze the collected user information.

For example, a user authentication result can be obtained through user authentication performed in various ways as illustrated in FIGS. 8a to 8e.

For example, as illustrated in Fig. 8a, a user authentication result can be obtained through biometric authentication such as fingerprint recognition, iris recognition, or facial recognition.

For example, as illustrated in FIG. 8b, a user authentication result can be obtained through biosignal analysis based on biosignals (e.g., heart rate, breathing pattern, brain waves, menstrual cycle, blood pressure) collected from wearable sub-devices including XR, ring, and watch.

For example, as illustrated in FIG. 8c, a user authentication result can be obtained through behavioral signal analysis of touch patterns, typing habits, gait patterns, and App usage patterns collected from an electronic device (100) and/or a sub-device.

For example, as illustrated in FIG. 8d, if an on-device AI model specialized for a specific application is installed in the electronic device (100), a user authentication result can be obtained based on information analyzed by the AI model. For example, in the case of phone calls/recordings, a user authentication result can be obtained through analysis using the LLM model, in the case of voice signals, through voiceprint analysis, and in the case of the camera function, through user face authentication analysis using the LVM model.

For example, as illustrated in Figure 8e, user authentication performed by a third-party app can yield nationally accepted or socially recognized authentication results. For example, a public certificate, one-time password (OTP) authentication, or phone authentication can be obtained as user authentication results.

In one example, the electronic device (100) can analyze preprocessed user information using the AI package module (440). For example, the AI package module (440) can select an appropriate artificial intelligence model from among a plurality of artificial intelligence models according to the type of user information (or the type of input feature) to process the user information. For example, the plurality of artificial intelligence models can include at least one of a behavioral biometrics authentication model, a biometric signal-based authentication model, a speaker verification model, and a face recognition model. In one example, the user analysis information of the AI package module (440) can only be used to utilize user information and/or maintain biometric authentication within the electronic device (100) and cannot be used to determine whether to transmit user information outside the electronic device (100). In other words, the user analysis information of the AI package module (440) can be used auxiliaryly.

In one example, the input features of the behavioral analysis authentication model may include sensor data of the electronic device (100), touch patterns, typing habits, gait, app usage patterns, slide/swipe/grip patterns, and one-handed usability. The output features of the behavioral analysis authentication model may include user consistency and/or accuracy compared to existing user information (e.g., unique behavior patterns).

For example, input features of a biometric authentication model may include biometric signals collected from wearable devices, including wearable devices such as XRs, rings, and watches (e.g., heart rate, breathing patterns, brain waves, menstrual cycles, blood pressure, blood sugar, stress, and sleep information), sensor data of sub-devices, and regularities. Output features of the biometric authentication model may include user consistency and/or accuracy compared to existing user information.

In one example, the input features of the speaker analysis model may include a human voice signal input through the microphone of the electronic device (100), a human voice signal input through the input module of the sub-device, and a call recording. The output features of the speaker analysis model may include a user match rate and/or accuracy compared to existing user information.

For example, the input features of the appearance analysis model may include information including a human face among image information input through the camera of the electronic device (100), a mirror image when wearing an XR (extended reality) device, and a user image stored in the user's gallery App. The output features of the appearance analysis model may include a user match and/or accuracy compared to the user's existing information. In an XR environment, a mirror image may mean a virtual avatar reflecting the user's self-expression in the virtual world appearing through a mirror or reflective surface when the user wears an XR device (e.g., an augmented reality (AR), virtual reality (VR), or mixed reality (MR) headset).

In operation 530, the electronic device (100) according to one embodiment may calculate a score based on the user authentication result of the AI package module (440). In addition to the artificial intelligence model included in the aforementioned AI package module (440), authentication of a third-party app may be used to obtain the user authentication result. For example, authentication that can be socially and publicly recognized as a user, such as a public certificate, OTP authentication, phone authentication, or Pass authentication, may be used to obtain the user authentication result.

For example, the electronic device (100) can quantify the results of various authentication methods, including user biometric authentication, to calculate a first score and perform an operation according to the user authentication level based on the calculated score.

According to one embodiment, the electronic device (100) may calculate numerical values for multiple types of authentication results as illustrated in FIG. 9 and integrate the calculated numerical values to calculate a first score.

For example, the first authentication result, the second authentication result, ..., n-th authentication result may be authentication results obtained through a plurality of first artificial intelligence models that process different types of information. For example, the first authentication result may be a first user matching value obtained by comparing a gait pattern of a previously authenticated user with a gait pattern included in the currently acquired user information. In this case, the electronic device (100) may calculate a first value (e.g., a value 9) based on the first user matching value (e.g., a probability of 90%). For example, the second authentication result may be a second user matching value (e.g., a probability of 80%) obtained by comparing a previously authenticated breathing pattern with a currently acquired breathing pattern. In this case, the electronic device (100) may calculate a second value (e.g., a value 8) based on the second user matching value. For example, the electronic device (100) may calculate each user matching value as a number within a preset range (e.g., 0 to 10).

For example, the electronic device (100) may calculate a first score by integrating a first value, a second value, ..., and an n-th value. For example, the electronic device (100) may calculate a first score by adding the first value to the n-th value. For example, the electronic device (100) may calculate a first score by applying a weight corresponding to each information type to each of the first value to the n-th value and then adding them.

In operation 535, the electronic device (100) according to an embodiment can identify whether the score is greater than or equal to a first threshold value. For example, the electronic device (100) can identify that the user authentication level satisfies the first level if the score is greater than or equal to the first threshold value.

For example, if the score is greater than or equal to a first threshold value (535:Y), the electronic device (100) may obtain user status information based on user information using the user status analysis module (480) in operation 540. For example, the electronic device (100) may input user information for user status analysis into an artificial intelligence model to obtain digitized user status information. If the score is identified as being less than the first threshold value (535:N) in operation 535, the electronic device (100) may renew (or re-perform) user authentication in operation 530.

According to an embodiment, when the electronic device (100) acquires digitized user status information, it can identify whether the score is equal to or greater than a second threshold value in operation 545. For example, if the score is equal to or greater than the second threshold value, the electronic device (100) can identify that the user authentication level satisfies the second level. However, this is merely an example, and it is also possible to identify whether the score is equal to or greater than the second threshold value in operation 535.

For example, points awarded may vary depending on the user authentication method. For example, biometric authentication (e.g., fingerprint/iris) may have relatively high reliability, and thus may be awarded relatively more points. For example, when biometric authentication is performed, points may be awarded to satisfy the second level of user authentication. When biometric authentication is performed in this manner, the second level (or the second level of reliability) is maintained during continuous use of the electronic device (100), and the authentication score may gradually decrease (or drop) when the use of the electronic device (100) is terminated. For example, when biometric authentication is performed, authentication may be maintained by maintaining level 1 even if the authentication score decreases, but this is not limited thereto.

For example, once a certain level of reliability is achieved through biometric authentication, the user authentication level can be maintained or adjusted by referencing the user match rate based on additional authentication methods until subsequent user biometric authentication occurs. For example, if the user match rate based on additional authentication methods is low, the user authentication level can be lowered.

For example, if the score falls below the first threshold, the user status analysis using the user status analysis module (480) may be stopped and user re-authentication (e.g., biometric authentication) may be waited for.

For example, when a certain level of user information is collected in a state where the score is higher than the first threshold value (maintained at or above the first level), the user status analysis module (480) may operate.

For example, the user status analysis module (480) may analyze user information through the LLM and output user status information. For example, when input features such as exercise information, weather information, work information, App execution information, number of steps, user feedback, location information, and health information are input, the LLM may output user status information such as fatigue (e.g., fatigue by body part), regularity, discomfort index, and exercise amount. For example, the input feature may include data less than a certain amount in order to reduce the burden on the memory (120). For example, the input feature may include data for a preset period (e.g., 1 day, 2 days) or information between specific event triggers (e.g., from leaving a previously memorized place to entering a current memorized place).

For example, user status information may be calculated by applying relatively greater weight to certain types of user information than to other types. For example, health information may be weighted more heavily than other types of information. For example, if a user has a cold, fatigue levels may increase based on the user's exercise level, and the discomfort index may increase sharply depending on the weather.

For example, user status information acquired through the user status analysis module (480) may be adjusted by the user. For example, the user may adjust user status information through the UI screen, such as increasing fatigue levels, increasing exercise volume, or reducing remaining workload.

For example, power consumption can be reduced by having LLM operate only within a memorized region, at specific times, or when specific conditions are met.

For example, if the score is equal to or greater than the second threshold value (545:Y), the electronic device (100) may, in operation 550, design (or build) an optimal IoT environment of the memorized place based on the digitized user status information. For example, in operation 555, the electronic device (100) may transmit the digitized user status information to the IoT hub (or IoT server (50)). Alternatively, the electronic device (100) may obtain control information for the IoT device based on the digitized user status information and the IoT device information in the memorized place and transmit the control information to the IoT hub (or IoT server (50)). The IoT device information may include information on at least one of a device type, a device function, a device specification, a device performance, a device model, or a relative position between devices.

In operation 545, if the score is identified as being greater than or equal to the first threshold but less than the second threshold (545:N), in operation 560, the electronic device (100) according to one embodiment may identify whether the electronic device (100) is located within the memorized region.

According to one embodiment, when the electronic device (100) is identified as being located within the memorized region (560:Y), the electronic device (100) may transmit digitized user status information to the IoT hub (or IoT server (50)). For example, even if the score is less than the second threshold, if it is greater than or equal to the first threshold and the electronic device (100) is located within a preset space, the electronic device (100) may determine that the second level is satisfied and transmit digitized user status information or control information for the IoT device to the IoT hub (or IoT server (50)). That is, the requirement for user authentication for establishing an IoT environment may be relatively low within the memorized region.

According to one embodiment, if the electronic device (100) is identified as not being located within the memorized region (560:N), an operation (510) of collecting user information may be performed.

For example, as illustrated in FIG. 10, the electronic device (100) may transmit user state information or control information for IoT devices to the IoT hub (1010) (or IoT server) to control the memorized place (1020) into an optimal environment based on user state information. In this case, the IoT hub (1010) may obtain control information for IoT devices (1021, 1022, 1023) to establish an optimal IoT environment based on the user state information received from the electronic device (100), and may automatically control the operations of the IoT devices (1021, 1022, 1023). Alternatively, the IoT hub (1010) may automatically control the operations of the IoT devices (1021, 1022, 1023) based on the control information for the IoT devices (1021, 1022, 1023) received from the electronic device (100). Accordingly, when a user enters a memorized place (1020), an optimal IoT environment can be provided.

FIG. 11 is a diagram for explaining a method for obtaining IoT control information according to one embodiment.

According to one embodiment, when the server (50) receives user status information from the electronic device (100), the server (50) may input the user status information and IoT device information into the third artificial intelligence model (1110) to obtain control information of the IoT device. According to one example, the server (50) may input user status information (e.g., fatigue A, exercise amount B, discomfort index C, and regularity D) and IoT device information (e.g., refrigerator, washing machine, air conditioner, air purifier) into the third artificial intelligence model (1110) to obtain control information of the IoT device (e.g., refrigerator 3°C, washing machine ON, air conditioner 20°C, air purifier auto). The IoT device information may include at least one of type information, specification information, current status information, function information, connection status information, power status information, location information, firmware information, or software information of the IoT device.

For example, the third artificial intelligence model (1110) can be trained using user status information and IoT device information as input data and IoT device control information as output data.

Here, the learning of the artificial intelligence model means that a basic artificial intelligence model (e.g., an artificial intelligence model including any random parameters) is learned using a plurality of training data by a learning algorithm, thereby creating a predefined operation rule or artificial intelligence model set to perform a desired characteristic (or purpose). Such learning may be performed through a separate server and/or system, but is not limited thereto, and may also be performed in the electronic device (100). Examples of the learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

Here, the artificial intelligence model can be implemented as, for example, a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or deep Q-networks, but is not limited thereto.

However, control information of IoT devices is not necessarily obtained using an artificial intelligence model, but may be obtained using a pre-defined algorithm, a pre-defined formula, and/or a pre-stored lookup table.

FIGS. 12A to 12D are drawings for explaining examples of UI screens provided in an electronic device according to one embodiment.

According to an example, the electronic device (100) may provide a UI screen (1210) related to registering a memorized place, as illustrated in FIG. 12A. For example, when a user registers a specific location as a memorized place, location information and additional information of the location may be registered as the memorized place. For example, the additional information may be acquired through a sensor (140) and may include various information related to the space. For example, the additional information may include at least one of barometric pressure information, geomagnetic information, height information, indoor/outdoor information, and WiFi information. According to an example, when a memorized place is registered, the UI screen (1210) may be provided including guide information (1211) indicating that the memorized place has been registered and information (1212) about the memorized place. Accordingly, the user can check the information determined by the electronic device (100) to confirm access, entry, and exit to a memorized place (or memorized region).

According to an example, the electronic device (100) may provide a UI screen (1220) including user status information as illustrated in FIGS. 12B and 12C . According to an example, the UI screen (1220) may include an adjustment UI (1221) for adjusting the level of at least one status item among fatigue, amount of exercise, discomfort index, and regularity. For example, as illustrated in FIG. 12B , the adjustment UI (1221) may be displayed as a numerical value of a preset range of positions (e.g., 0 to 100) that a user can check and adjust.

For example, if a user adjusts some status items through the adjustment UI (1221), the electronic device (100) may initiate reanalysis of the user status information. In this case, the electronic device (100) may provide a UI (1222) that inquires whether to reanalyze the user status, as illustrated in FIG. 12C . For example, if the user selects reanalysis, the user status information may be reanalyzed, and the reanalyzed information may be provided again and/or transmitted to the server (50) through the UI. For example, if the user selects not to reanalyze, the adjusted values may be transmitted to the server (50) without reanalysis.

For example, a user can check in real time how the settings of an IoT device change as he or she directly adjusts his or her status through the adjustment UI (1221).

According to one example, the electronic device (100) may provide a UI screen (1230) including current operation setting information of an IoT device (e.g., a refrigerator, a washing machine, an air conditioner) located in a memorized place (e.g., our house) as illustrated in FIG. 12d. For example, when the operation of an IoT device is automatically set by the server (50) to provide an optimal IoT environment, the electronic device (100) may receive the set information and provide a UI screen (1230) including the received information. For example, a user may also manually change the setting information through the UI screen (1230).

Meanwhile, in FIGS. 12A to 12D, it is assumed that the electronic device (100) is implemented as a bar-type phone and is described as providing information that matches the screen size of the smart phone. However, depending on the shape and/or size of the screen according to the implementation example of the electronic device (100), the information that can be provided simultaneously may be diversified. For example, if the electronic device (100) is implemented as a foldable type phone, the screen size is larger than that of a bar-type phone, so that the user's manual operation process can be smoother, and thus a relatively large number of menus for manual operation can be provided.

FIG. 13 is a drawing for explaining a method for providing an IoT environment in a vehicle according to one embodiment.

In one embodiment, a memorized place may be a fixed location, such as a home, work space, or office space, but may also be an unfixed location, such as a vehicle (or automobile). In one example, as illustrated in FIG. 13 , a user's vehicle (1310) may be set as a memorized place, and a preset distance range based on the vehicle (1310) may be set as a memorized region (1320).

For example, the electronic device (100) can control the IoT environment within the vehicle (1310) based on user status information. For example, the electronic device (100) can adjust the seat angle, temperature, recommended navigation route (e.g., recommended route that passes many rest areas for resting), and engine preheating of the vehicle (1310) based on the user's fatigue level, discomfort index, and amount of exercise. For example, when the user enters a memorized region (1320), the electronic device (100) can adjust the environment of the vehicle (1310), which is a memorized place, based on the user status information. For example, when the electronic device (100) determines that the user has entered a memorized region (1320), which is a preset distance range from the location information of the vehicle (1310), based on the location information of the electronic device (100), the electronic device (100) can adjust the environment of the vehicle (1310) to an optimal environment based on the user status information.

For example, if the IoT environment is adjusted (e.g., seat adjustment) after the vehicle (1310) has started, an accident may occur, so additional control may be performed based on the user's status information based on information in the memorized place, such as adjusting a specific function (e.g., seat adjustment) to a non-changeable item.

In this way, since user information is collected/transmitted through an electronic device (100) (e.g., a portable terminal), an optimal environment can be provided based on user status information regardless of location.

In one embodiment, user status information (e.g., status values) may be utilized externally. For example, if a user visits a hospital, recent user status information may be provided to the hospital to help determine the cause of the illness. Alternatively, a personal trainer at a gym may provide appropriate exercise methods or advice based on the user's status values. In one example, a second user authentication level (i.e., a value higher than a second threshold) may be required to maintain a high level of security in order to externally transmit user status information or for verification by other users.

FIG. 14 is a block diagram of an electronic device (1401) within a network environment (1400) according to various embodiments. The electronic device (1401) may be implemented as the electronic device (100) illustrated in FIG. 1, for example.

Referring to FIG. 14, in a network environment (1400), an electronic device (1401) may communicate with an electronic device (1402) via a first network (1498) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (1404) or a server (1408) via a second network (1499) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (1401) may communicate with the electronic device (1404) via the server (1408). According to one embodiment, the electronic device (1401) may include a processor (1420), a memory (1430), an input module (1450), an audio output module (1455), a display module (1460), an audio module (1470), a sensor module (1476), an interface (1477), a connection terminal (1478), a haptic module (1479), a camera module (1480), a power management module (1488), a battery (1489), a communication module (1490), a subscriber identification module (1496), or an antenna module (1497). In some embodiments, the electronic device (1401) may omit at least one of these components (e.g., the connection terminal (1478)), or may have one or more other components added. In some embodiments, some of these components (e.g., sensor module (1476), camera module (1480), or antenna module (1497)) may be integrated into one component (e.g., display module (1460)).

The processor (1420) may control at least one other component (e.g., hardware or software component) of the electronic device (1401) connected to the processor (1420) by executing, for example, software (e.g., program (1440)), and may perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (1420) may store commands or data received from other components (e.g., sensor module (1476) or communication module (1490)) in volatile memory (1432), process the commands or data stored in volatile memory (1432), and store result data in non-volatile memory (1434). According to one embodiment, the processor (1420) may include a main processor (1421) (e.g., a central processing unit or an application processor) or an auxiliary processor (1423) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (1421). For example, when the electronic device (1401) includes the main processor (1421) and the auxiliary processor (1423), the auxiliary processor (1423) may be configured to use less power than the main processor (1421) or to be specialized for a given function. The auxiliary processor (1423) may be implemented separately from the main processor (1421) or as a part thereof.

The auxiliary processor (1423) may control at least a portion of functions or states associated with at least one component (e.g., the display module (1460), the sensor module (1476), or the communication module (1490)) of the electronic device (1401), for example, on behalf of the main processor (1421) while the main processor (1421) is in an inactive (e.g., sleep) state, or together with the main processor (1421) while the main processor (1421) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (1423) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (1480) or a communication module (1490)). In one embodiment, the auxiliary processor (1423) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, in the electronic device (1401) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (1408)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include a plurality of artificial neural network layers. The artificial neural network can be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), deep Q-networks, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model can additionally or alternatively include a software structure.

The memory (1430) can store various data used by at least one component (e.g., the processor (1420) or the sensor module (1476)) of the electronic device (1401). The data can include, for example, software (e.g., the program (1440)) and input data or output data for commands related thereto. The memory (1430) can include volatile memory (1432) or non-volatile memory (1434).

The program (1440) may be stored as software in memory (1430) and may include, for example, an operating system (1442), middleware (1444), or an application (1446).

The input module (1450) can receive commands or data to be used in a component of the electronic device (1401) (e.g., a processor (1420)) from an external source (e.g., a user) of the electronic device (1401). The input module (1450) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

The audio output module (1455) can output audio signals to the outside of the electronic device (1401). The audio output module (1455) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. According to one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

The display module (1460) can visually provide information to an external device (e.g., a user) of the electronic device (1401). The display module (1460) may include, for example, a display, a holographic device, or a projector, and a control circuit for controlling the device. According to one embodiment, the display module (1460) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

The audio module (1470) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (1470) can acquire sound through the input module (1450), output sound through the sound output module (1455), or an external electronic device (e.g., electronic device (1402)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (1401).

The sensor module (1476) can detect the operating status (e.g., power or temperature) of the electronic device (1401) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (1476) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

The interface (1477) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (1401) with an external electronic device (e.g., the electronic device (1402)). In one embodiment, the interface (1477) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal (1478) may include a connector through which the electronic device (1401) may be physically connected to an external electronic device (e.g., the electronic device (1402)). According to one embodiment, the connection terminal (1478) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

The haptic module (1479) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. In one embodiment, the haptic module (1479) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module (1480) can capture still images and videos. According to one embodiment, the camera module (1480) may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module (1488) can manage power supplied to the electronic device (1401). According to one embodiment, the power management module (1488) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

A battery (1489) may power at least one component of the electronic device (1401). In one embodiment, the battery (1489) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

The communication module (1490) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (1401) and an external electronic device (e.g., electronic device (1402), electronic device (1404), or server (1408)), and the performance of communication through the established communication channel. The communication module (1490) may operate independently from the processor (1420) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (1490) may include a wireless communication module (1492) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (1494) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the above-mentioned communication module can communicate with an external electronic device (1404) via a first network (1498) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (1499) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (1492) can verify or authenticate the electronic device (1401) within a communication network such as the first network (1498) or the second network (1499) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (1496).

The wireless communication module (1492) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimizing terminal power and connecting multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency communications (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (1492) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (1492) may support various technologies for securing performance in high-frequency bands, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (1492) may support various requirements specified in the electronic device (1401), an external electronic device (e.g., the electronic device (1404)), or a network system (e.g., the second network (1499)). According to one embodiment, the wireless communication module (1492) may support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

The antenna module (1497) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (1497) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (1497) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (1498) or the second network (1499), may be selected from the plurality of antennas by, for example, the communication module (1490). A signal or power may be transmitted or received between the communication module (1490) and an external electronic device via the selected at least one antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (1497).

According to various embodiments, the antenna module (1497) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band.

At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

According to one embodiment, commands or data may be transmitted or received between the electronic device (1401) and an external electronic device (1404) via a server (1408) connected to a second network (1499). Each of the external electronic devices (1402 or 1404) may be the same or a different type of device as the electronic device (1401). According to one embodiment, all or part of the operations executed in the electronic device (1401) may be executed in one or more of the external electronic devices (1402, 1404, or 1408). For example, when the electronic device (1401) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (1401) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (1401). The electronic device (1401) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (1401) may provide an ultra-low latency service using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (1404) may include an Internet of Things (IoT) device. The server (1408) may be an intelligent server utilizing machine learning and/or a neural network. According to one embodiment, the external electronic device (1404) or the server (1408) may be included in a second network (1499). The electronic device (1401) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

According to one embodiment, an electronic device includes: a memory storing instructions; a communication circuit; and at least one processor including a processing circuit; wherein the instructions, when individually or collectively executed by the at least one processor, cause the electronic device to, when first user information is collected, obtain a user authentication result based on the first user information using a first artificial intelligence model, quantify the user authentication result to obtain a first score, identify a user authentication level based on the first score, and, if the user authentication level is the first level, obtain first user status information based on the first user information using a second artificial intelligence model, and, if the user authentication level is a second level higher than the first level, transmit the first user status information to the outside of the electronic device.

According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to transmit the first user status information to a server managing an IoT device located in a preset space through the communication circuit, if the user authentication level is the second level.

According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to re-perform user authentication based on second user information collected thereafter if the first score is less than the first threshold value, identify the user authentication level as the first level if the first score is greater than or equal to the first threshold value, and identify the user authentication level as the second level if the first score is greater than or equal to a second threshold value greater than the first threshold value.

According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to transmit the first user state information to an outside of the electronic device, and then, if use of the electronic device is maintained, to transmit second user state information obtained based on the second user information to an outside of the electronic device while maintaining the second level of user authentication, and when use of the electronic device is terminated, to store the second user state information obtained based on the second user information in the memory while maintaining the first level of user authentication.

According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to initiate an operation of digitizing the user authentication result based on at least one of an event in which the electronic device is located within the preset space, an event in which biometric authentication is performed, and an event in which the first user information collected in the memory is greater than or equal to a preset amount.

According to one embodiment, the electronic device further includes a sensor; and the instructions, when individually or collectively executed by the at least one processor, cause the electronic device to identify whether the electronic device is located within the preset space based on information corresponding to the preset space and sensing information acquired through the sensor, if the acquired first score is greater than or equal to the first threshold value and less than the second threshold value, and transmit the first user status information to the outside of the electronic device if the electronic device is located within the preset space, and the sensing information may include at least one of location information, air pressure information, geomagnetic information, and Wi-Fi information.

According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, cause the electronic device to input the information for user authentication among the first user information into the first artificial intelligence model to obtain the user authentication result, wherein the information for user authentication includes at least one of behavioral information, biometric information, voice information, image information, touch information, or app execution information, and the first artificial intelligence model may be trained to compare the authenticated user information and the information for user authentication and output the user authentication result including a user match rate.

According to one embodiment, the first artificial intelligence model may include a first artificial intelligence model trained to compare first type information among the information for user authentication with the previously authenticated user information and output a first user match, and a first artificial intelligence model trained to compare second type information among the information for user authentication with the previously authenticated user information and output a second user match.

According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, cause the electronic device to input information for analyzing the user's status among the first user information into the second artificial intelligence model to obtain the first user status information, wherein the information for analyzing the user's status includes at least one of health information, environmental information, time information, work information, feedback information, and message/call information, and the second artificial intelligence model may be trained to output the first user status information including a plurality of status items that are digitized by applying different weights to each type of information for analyzing the user's status.

According to one embodiment, the electronic device further includes a display; and the instructions, when individually or collectively executed by the at least one processor, cause the electronic device to display a user interface (UI) including the first user status information on the display, wherein the UI may include a UI for adjusting a level of at least one status item among fatigue, amount of exercise, discomfort index, and regularity.

According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to, when information about the IoT device is received from the server, obtain control information for controlling the IoT device based on the first user state information and the information about the IoT device, and transmit the obtained control information to the server through the communication circuit.

According to one embodiment, a control method of an electronic device includes: when first user information is collected, an operation of obtaining a user authentication result based on the first user information using a first artificial intelligence model; an operation of quantifying the user authentication result to obtain a first score; an operation of identifying a user authentication level based on the first score; an operation of obtaining first user status information based on the first user information using a second artificial intelligence model when the user authentication level is the first level; and an operation of transmitting the first user status information to the outside of the electronic device when the user authentication level is a second level higher than the first level.

According to one embodiment, the operation of transmitting the electronic device to the outside may include an operation of transmitting the first user status information to a server that manages an IoT (Internet of Things) device located in a preset space.

According to one embodiment, the control method may further include: an operation of re-performing user authentication based on second user information collected thereafter if the first score is less than the first threshold value; an operation of identifying the user authentication level as the first level if the first score is greater than or equal to the first threshold value; and an operation of identifying the user authentication level as the second level if the first score is greater than or equal to a second threshold value that is greater than the first threshold value.

According to one embodiment, the control method may further include: transmitting the first user state information to the outside of the electronic device, and then, if use of the electronic device is maintained, transmitting the second user state information obtained based on the second user information to the outside of the electronic device by maintaining the second level of user authentication; and storing the second user state information obtained based on the second user information by maintaining the first level of user authentication when use of the electronic device is terminated.

According to one embodiment, the operation of obtaining the first score may include an operation of initiating an operation of quantifying the user authentication result based on at least one of an event in which the electronic device is located within the preset space, an event in which biometric authentication is performed, and an event in which the first user information collected in the memory is greater than or equal to a preset amount.

According to one embodiment, the control method further includes: an operation of identifying whether the electronic device is located within the preset space based on information corresponding to the preset space and sensing information acquired through the sensor when the acquired first score is greater than or equal to the first threshold value and less than the second threshold value; and an operation of transmitting the first user status information to the outside of the electronic device when the electronic device is located within the preset space; wherein the sensing information may include at least one of location information, air pressure information, geomagnetic information, and Wi-Fi information.

According to one embodiment, the operation of obtaining the user authentication result includes an operation of inputting the information for user authentication among the first user information into the first artificial intelligence model to obtain the user authentication result; wherein the information for user authentication includes at least one of behavioral information, biometric information, voice information, image information, touch information, or app execution information, and the first artificial intelligence model may be trained to compare the previously authenticated user information and the information for user authentication and output the user authentication result including a user match rate.

According to one embodiment, the first artificial intelligence model may include a first artificial intelligence model trained to compare first type information among the information for user authentication with the previously authenticated user information and output a first user match, and a first artificial intelligence model trained to compare second type information among the information for user authentication with the previously authenticated user information and output a second user match.

According to one embodiment, the operation of obtaining the first user status information includes an operation of obtaining the first user status information by inputting information for analyzing the user's status among the first user information into the second artificial intelligence model, wherein the information for analyzing the user's status includes at least one of health information, environmental information, time information, work information, feedback information, and message/call information, and the second artificial intelligence model may be trained to output the first user status information including a plurality of status items that are digitized by applying different weights to each type of information for analyzing the user's status.

According to one embodiment, the control method further includes an operation of displaying a user interface (UI) including the first user status information; wherein the UI may include a UI for adjusting a level of at least one status item among fatigue, amount of exercise, discomfort index, and regularity.

According to one embodiment, a non-transitory computer-readable medium storing instructions that, when executed by a processor of an electronic device, cause the electronic device to perform an operation, the operations include: when first user information is collected, obtaining a user authentication result based on the first user information using a first artificial intelligence model; obtaining a first score by digitizing the user authentication result; identifying a user authentication level based on the first score; if the user authentication level is the first level, obtaining first user status information based on the first user information using a second artificial intelligence model; and if the user authentication level is a second level higher than the first level, transmitting the first user status information to the outside of the electronic device.

According to the various embodiments described above, the problem of authentication being repeatedly requested during the user information transmission process or the operation of on-device AI utilizing personal information can be solved by maintaining the authentication status through user information acquired through portable electronic devices and peripheral devices.

Additionally, user information is automatically analyzed and matched to the type of IoT device, contributing to the optimal environment configuration through integrated/comprehensive control.

Additionally, on-device AI can become more personalized to users by maintaining authentication based on user information beyond fingerprints and irises (e.g., breathing patterns, walking habits, typing habits), which can significantly reduce the risk of identity theft.

Additionally, by reflecting user information through real-time information exchange, it is possible to predict the user's next action and analyze the user's satisfaction with the environment to control the IoT environment, thereby providing a better IoT environment compared to when the user operates it manually.

Although the various embodiments described above have been described as utilizing multiple individual neural network models, the operations of at least two of the multiple neural network models may be implemented in a single neural network model.

Each operation according to the various embodiments described above may be performed by the processor (110), but if necessary, a module for each operation may be utilized. For example, each module may be implemented using at least one software, at least one hardware, and/or a combination thereof. Each module may be implemented to utilize a predefined algorithm, a predefined formula, and/or a learned artificial intelligence model to perform the operation. However, at least some modules may be distributed to an external device.

The methods according to the various embodiments of the present disclosure described above may be implemented in the form of applications installable on existing electronic devices. Alternatively, the methods according to the various embodiments of the present disclosure described above may be performed using a deep learning-based artificial neural network (or deep artificial neural network), i.e., a learning network model.

The methods according to the various embodiments of the present disclosure described above can be implemented only with a software upgrade or a hardware upgrade for an existing electronic device.

The various embodiments of the present disclosure described above can also be performed through an embedded server provided in an electronic device, or an external server of the electronic device.

According to an exemplary embodiment of the present disclosure, the various embodiments described above may be implemented as software including instructions stored in a machine-readable storage medium that can be read by a machine (e.g., a computer). The device may include an electronic device (e.g., electronic device A) according to the disclosed embodiments, which is a device that can call instructions stored in the storage medium and operate according to the called instructions. When an instruction is executed by a processor, the processor may directly or under the control of the processor perform a function corresponding to the instruction using other components. The instruction may include code generated or executed by a compiler or interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' means that the storage medium does not contain a signal and is tangible, but does not distinguish between data being stored semi-permanently or temporarily in the storage medium.

Furthermore, according to one embodiment of the present disclosure, the method according to the various embodiments described above may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or online through an application store (e.g., Play Store™). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

In addition, each of the components (e.g., modules or programs) according to the various embodiments described above may be composed of a single or multiple entities, and some of the corresponding sub-components described above may be omitted, or other sub-components may be further included in various embodiments. Alternatively or additionally, some components (e.g., modules or programs) may be integrated into a single entity, which may perform the same or similar functions as those performed by each of the corresponding components prior to integration. Operations performed by modules, programs or other components according to various embodiments may be executed sequentially, in parallel, iteratively or heuristically, or at least some operations may be executed in a different order, omitted, or other operations may be added.

Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person having ordinary skill in the art to which the present disclosure pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present disclosure.

100: Electronic device 110: Processor
120: Memory 130: Communication circuit
140: Sensor 150: Display

Claims (20)

In electronic devices,
Memory (120) for storing instructions;
Communication circuit (130); and
At least one processor (110) comprising processing circuitry;
The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to:
When the first user information is collected, the first artificial intelligence model is used to obtain a user authentication result based on the first user information,
The above user authentication result is quantified to obtain a first score,
Identifying the user authentication level based on the first score,
If the user authentication level is the first level, the second artificial intelligence model is used to obtain first user status information based on the first user information,
An electronic device that transmits the first user status information to the outside of the electronic device through the communication circuit when the user authentication level is a second level higher than the first level. In the first paragraph,
The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to:
An electronic device that transmits the first user status information to a server that manages an IoT (iInternet of tThings) device located in a preset space through the communication circuit when the user authentication level is the second level. In the first paragraph,
The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to:
If the first score is less than the first threshold, user authentication is re-performed based on the second user information collected thereafter,
If the first score is greater than or equal to the first threshold value, the user authentication level is identified as the first level,
An electronic device that identifies the user authentication level as the second level if the first score is greater than or equal to a second threshold value that is greater than the first threshold value. In the third paragraph,
The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to:
After transmitting the first user status information to the outside of the electronic device, if the use of the electronic device is maintained, the second level of user authentication is maintained and the second user status information obtained based on the second user information is transmitted to the outside of the electronic device,
An electronic device that, when use of the electronic device is terminated, maintains the first level of user authentication and stores the second user status information obtained based on the second user information in the memory. In the first paragraph,
The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to:
An electronic device that initiates an operation of digitizing the user authentication result based on at least one of an event in which the electronic device is located within the preset space, an event in which biometric authentication is performed, and an event in which the first user information collected in the memory is greater than or equal to a preset amount. In the first paragraph,
The above electronic device,
including sensors;
The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to:
If the acquired first score is greater than or equal to the first threshold value and less than or equal to the second threshold value, the electronic device is identified as being located within the preset space based on information corresponding to the preset space and sensing information acquired through the sensor,
When the electronic device is located within the preset space, the first user status information is transmitted to the outside of the electronic device,
The above sensing information is,
An electronic device comprising at least one of location information, barometric pressure information, geomagnetic information, and wifi information. In the first paragraph,
The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to:
The user authentication result is obtained by inputting the user authentication information among the first user information into the first artificial intelligence model.
Information for the above user authentication is:
Contains at least one of behavioral information, biometric information, voice information, image information, touch information, or app execution information;
The above first artificial intelligence model is,
An electronic device trained to compare authenticated user information and information for user authentication and output the user authentication result including the user match. In paragraph 7,
The above first artificial intelligence model is,
An electronic device comprising a first artificial intelligence model trained to compare first type information among the information for user authentication with the previously authenticated user information and output a first user match, and a first artificial intelligence model trained to compare second type information among the information for user authentication with the previously authenticated user information and output a second user match. In the first paragraph,
The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to:
The first user status information is obtained by inputting information for analyzing the user's status among the first user information into the second artificial intelligence model,
Information for analyzing the status of the above user is:
Contains at least one of health information, environmental information, time information, work information, feedback information, and message/call information.
The above second artificial intelligence model is,
An electronic device that is trained to output the first user status information including a plurality of status items that are digitized by applying different weights to each type of information for analyzing the status of the user. In the first paragraph,
The above electronic device,
including display;
The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to:
To display a UI (user interface) including the first user status information on the display,
The above UI is,
An electronic device comprising a UI for adjusting the level of at least one of the following status items: fatigue, exercise volume, discomfort index, and regularity. In the second paragraph,
The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to:
When information about the IoT device is received from the server, control information for controlling the IoT device is obtained based on the first user status information and information about the IoT device,
An electronic device that transmits the acquired control information to the server through the communication circuit. In a method for controlling an electronic device,
When first user information is collected, an operation of obtaining a user authentication result based on the first user information using a first artificial intelligence model;
An action of obtaining a first score by quantifying the above user authentication result;
An operation for identifying a user authentication level based on the first score;
If the user authentication level is the first level, an operation of obtaining first user status information based on the first user information using a second artificial intelligence model; and
A control method, comprising: an operation of transmitting the first user status information to the outside of the electronic device when the user authentication level is a second level higher than the first level. In Article 11,
The operation of transmitting to the outside of the above electronic device is:
A control method, comprising: an operation of transmitting the first user status information to a server managing an IoT (Internet of Things) device located in a preset space through the communication circuit. In Article 12,
If the first score is less than the first threshold value, an operation of re-performing user authentication based on the second user information collected thereafter;
An operation for identifying the user authentication level as the first level if the first score is greater than or equal to the first threshold value; and
A control method further comprising: an operation for identifying the user authentication level as the second level if the first score is greater than or equal to a second threshold value that is greater than the first threshold value. In Article 14,
After transmitting the first user status information to the outside of the electronic device, if the use of the electronic device is maintained, an operation of transmitting the second user status information obtained based on the second user information to the outside of the electronic device by maintaining the second level of user authentication; and
A control method further comprising: an operation of storing the second user state information obtained based on the second user information by maintaining the first level of user authentication when use of the electronic device is terminated; In paragraph 12,
The action of obtaining the above first score is:
A control method, comprising: an operation of initiating an operation of digitizing the user authentication result based on at least one of an event in which the electronic device is located within the preset space, an event in which biometric authentication is performed, and an event in which the first user information collected in the memory is greater than or equal to a preset amount. In Article 12,
If the acquired first score is greater than or equal to the first threshold value and less than or equal to the second threshold value, an operation of identifying whether the electronic device is located within the preset space based on information corresponding to the preset space and sensing information acquired through the sensor; and
Further comprising an operation of transmitting the first user status information to the outside of the electronic device when the electronic device is located within the preset space;
The above sensing information is,
A control method comprising at least one of location information, barometric pressure information, geomagnetic information, and wifi information. In Article 12,
The action of obtaining the above user authentication result is:
An operation of inputting the user authentication information among the first user information into the first artificial intelligence model to obtain the user authentication result;
The information for the above user authentication is:
Contains at least one of behavioral information, biometric information, voice information, image information, touch information, or app execution information;
The above first artificial intelligence model is,
A control method that is learned to compare authenticated user information and information for user authentication and output the user authentication result including the user match rate. In Article 12,
The operation of obtaining the above first user status information is:
An operation of obtaining the first user status information by inputting information for analyzing the status of the user among the first user information into the second artificial intelligence model;
Information for analyzing the status of the above user is:
An electronic device comprising at least one of health information, environmental information, time information, work information, feedback information, and message/call information.
The above second artificial intelligence model is,
A control method learned to output the first user status information including a plurality of status items digitized by applying different weights to each type of information for analyzing the status of the user. A non-transitory computer-readable medium storing instructions that, when executed by a processor of an electronic device, cause the electronic device to perform an operation,
The above action is,
When first user information is collected, an operation of obtaining a user authentication result based on the first user information using a first artificial intelligence model;
An action of obtaining a first score by quantifying the above user authentication result;
An operation for identifying a user authentication level based on the first score;
If the user authentication level is the first level, an operation of obtaining first user status information based on the first user information using a second artificial intelligence model; and
A non-transitory computer-readable storage medium, comprising: an operation of transmitting the first user status information to the outside of the electronic device if the user authentication level is a second level higher than the first level.