WO2025178476A1 - Head-mounted device, operating method thereof, and non-transitory storage medium - Google Patents

Abstract

A head-mounted device is disclosed. The head-mounted device comprises: a first camera sensor corresponding to a first angle of view; a second camera sensor corresponding to the first angle of view; at least one third camera sensor corresponding to a second angle of view; an acceleration sensor; a memory for storing instructions; and a processor, wherein the instructions, when executed individually or collectively by the processor, cause the head-mounted device to: acquire a first image corresponding to a first time point through at least one of the first camera sensor and the second camera sensor; acquire a second image corresponding to the first time point through the at least one third camera sensor; and identify a third image corresponding to a second time point after the first time point on the basis of the first image corresponding to the first time point, the second image corresponding to the first time point, and sensing data acquired through the acceleration sensor.

Description

Head mounting device and its operating method, and non-transitory storage medium

The present disclosure relates to a head-mounted device for providing an image according to one embodiment, a method of operating the same, and a non-transitory storage medium.

As electronics and communication technologies advance, electronic devices can be miniaturized and lightweight enough to be worn on the body without significant discomfort. For example, wearable electronic devices such as head-mounted devices (HMDs), smartwatches (or bands), contact lenses, rings, gloves, shoes, and clothing are becoming increasingly commercialized. Because wearable electronic devices are worn directly on the body, they can offer enhanced portability and accessibility.

A visual see-through head-mounted display (VST-HMD) is a head-mounted electronic device with a goggle-like shape. A head-mounted electronic device (or head-mounted device) is a device worn on the user's head or face, capable of providing information about objects in the form of images or text within at least a portion of the user's field of vision.

Users can experience pure virtual reality (VR) rendered through an internal display while physically disconnected from the outside world (closed-view) by wearing a VST-HMD type wearable electronic device. Furthermore, this type of electronic device can transmit live video captured through a front-mounted camera to the internal display in real time, providing users with augmented reality (AR) or mixed reality (MR) experiences based on actual space.

The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art related to the present disclosure.

A head mounted device according to one embodiment of the present disclosure may include a first camera sensor corresponding to a first angle of view, a second camera sensor corresponding to the first angle of view, at least one third camera sensor corresponding to the second angle of view, an acceleration sensor, a memory storing instructions, and a processor.

In one embodiment, the instructions, when individually or collectively executed by the processor, may cause the head mounted device to acquire a first image corresponding to a first point in time through at least one of the first camera sensor and the second camera sensor.

In one embodiment, the instructions, when individually or collectively executed by the processor, may cause the head mounted device to acquire a second image corresponding to the first point in time via the at least one third camera sensor.

In one embodiment, the instructions, when individually or collectively executed by the processor, may cause the head mounted device to identify a first image corresponding to the first time point, a second image corresponding to the first time point, and a third image corresponding to a second time point after the first time point based on sensing data acquired through the acceleration sensor.

A method of operating a head mounted device according to one embodiment of the present disclosure may include an operation of acquiring a first image corresponding to a first viewpoint through at least one of a first camera sensor of the head mounted device corresponding to a first angle of view and a second camera sensor of the head mounted device corresponding to the first angle of view.

The above operating method may include an operation of acquiring a second image corresponding to the first viewpoint through at least one third camera sensor of the head-mounted device corresponding to the second angle of view.

The above operating method may include an operation of confirming a third image corresponding to a second time point after the first time point based on the first image corresponding to the first time point acquired, the second image corresponding to the first time point acquired, and sensing data acquired through an acceleration sensor of the head mounting device.

In a non-transitory storage medium storing computer-readable instructions according to one embodiment of the present disclosure, the instructions, when executed by a processor of a head mounted device, can cause the head mounted device to acquire a first image corresponding to a first viewpoint through at least one of a first camera sensor corresponding to a first angle of view and a second camera sensor corresponding to the first angle of view.

The above instructions, when executed by a processor of the head-mounted device, may cause the head-mounted device to acquire a second image corresponding to the first viewpoint through at least one third camera sensor corresponding to the second angle of view.

The above instructions, when executed by a processor of a head-mounted device, may cause the head-mounted device to identify a first image corresponding to the first time point, a second image corresponding to the first time point, and a third image corresponding to a second time point after the first time point based on sensing data acquired through an acceleration sensor.

In connection with the description of the drawings, the same or similar reference numerals may be used for the same or similar components.

FIG. 1 is a block diagram of an electronic device within a network environment according to one embodiment.

FIG. 2 is a block diagram of head mounting device configurations according to one embodiment.

FIG. 3 is a flowchart illustrating an operation method of a head mounting device according to one embodiment.

FIG. 4A is a diagram illustrating a camera sensor and generated image according to one embodiment.

FIG. 4b is a diagram illustrating a camera sensor and generated image according to one embodiment.

FIG. 4c is a diagram illustrating a camera sensor and generated image according to one embodiment.

FIG. 4D is a diagram illustrating a camera sensor and generated image according to one embodiment.

FIG. 5A is a flowchart illustrating a method for confirming an image corresponding to a second point in time according to one embodiment.

FIG. 5b is a diagram for explaining a method for confirming an image corresponding to a second point in time according to one embodiment.

FIG. 6 is a flowchart illustrating a method of post-processing a confirmed image according to one embodiment.

FIG. 7a is a flowchart illustrating a method of displaying an image according to one embodiment.

FIG. 7b is a drawing for explaining a method of displaying an image according to one embodiment.

FIG. 8a is a flowchart illustrating a method of displaying an image according to one embodiment.

FIG. 8b is a drawing for explaining a method of displaying an image according to one embodiment.

FIG. 9 is a flowchart illustrating a method of displaying an image according to one embodiment.

FIG. 10 is a diagram illustrating a generative artificial intelligence system according to one embodiment.

FIGS. 11A and 11B are drawings showing the front and rear views of a head mounting device according to one embodiment.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and conciseness.

FIG. 1 is a block diagram of an electronic device (101) within a network environment (100), according to one embodiment. Referring to FIG. 1 , in the network environment (100), the electronic device (101) may communicate with the electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of the electronic device (104) or the server (108) via a second network (199) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, in the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include a plurality of artificial neural network layers. The artificial neural network can be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), deep Q-networks, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model can additionally or alternatively include a software structure.

The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

According to various embodiments, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In one embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and/or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199). The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

In the detailed description below, reference numerals in the drawings may be used interchangeably or omitted for components that can be easily understood through the preceding embodiments, and their detailed descriptions may also be omitted. An electronic device according to an embodiment disclosed in this document may be implemented by selectively combining components of different embodiments, and components of one embodiment may be replaced by components of another embodiment. For example, it should be noted that the present disclosure is not limited to any specific drawing or embodiment.

FIG. 2 is a block diagram of configurations of a head mounting device (200) according to one embodiment.

Referring to FIG. 2, in one embodiment, a head mounted device (200, e.g., electronic device (101) of FIG. 1) may include a first camera sensor (210), a second camera sensor (220), at least one third camera sensor (230), an acceleration sensor (240), a memory (250, e.g., memory (130) of FIG. 1), and a processor (260). In one embodiment, the head mounted device (200) may have a configuration that is at least partially identical or similar to the configuration of the head mounted device (1100) illustrated in FIG. 11.

In one embodiment, the first camera sensor (210) may have at least a portion of the same or similar configuration as the camera module (180) of FIG. 1. In one embodiment, the first camera sensor (210) may correspond to a pass-through camera. A pass-through camera is a camera that acquires an image for a field of view that a user (e.g., a human) wearing a head-mounted device (200) can see without the head-mounted device (200). In one embodiment, the first camera sensor (210) may be a camera corresponding to either the user's left eye or the user's right eye. For example, when the first camera sensor (210) is implemented as a camera corresponding to the user's left eye, the first camera sensor (210) may acquire an image corresponding to the surrounding environment viewed by the user with the left eye. In one embodiment, the first camera sensor (210) may correspond to a first angle of view (AOV) or field of view (FOV).

In one embodiment, the second camera sensor (220) may have at least a portion identical or similar to the camera module (180) of FIG. 1. In one embodiment, the second camera sensor (220) may correspond to a pass-through camera. In one embodiment, the second camera sensor (220) may be a camera corresponding to either the left eye or the right eye of the user. For example, the first camera sensor (210) may be implemented as a camera corresponding to the left eye, and the second camera sensor (220) may be implemented as a camera corresponding to the right eye of the user. The second camera sensor (220) may acquire (or output) an image corresponding to the surrounding environment viewed by the user with the right eye. In one embodiment, the second camera sensor (220) may correspond to a first angle of view (field of view or FOV). In one embodiment, the first camera sensor (210) and the second camera sensor (220) may be implemented as RGB (red, green, blue) cameras.

In one embodiment, at least one third camera sensor (230) may have at least a portion identical or similar to the camera module (180) of FIG. 1. In one embodiment, at least one third camera sensor (230) may correspond to a second angle of view, and the second angle of view may be different from the first angle of view. For example, the second angle of view may be larger than or equal to the first angle of view. In one embodiment, when the second angle of view is larger than or equal to the first angle of view, an image acquired through at least one third camera sensor (230) may be an image corresponding to a relatively wider field of view than an image acquired through the first camera sensor (210) or the second camera sensor (220).

In one embodiment, at least one third camera sensor (230) can acquire an image corresponding to a relatively wider range than the user's field of view. In one embodiment, at least one third camera sensor (230) can be placed at a designated location within the head-mounted device (200). In one embodiment, at least one third camera sensor (230) can be implemented as an RGB camera or a black-and-white camera. The first camera sensor (210), the second camera sensor (220), and the at least one third camera sensor (230) will be described in detail with reference to FIGS. 4A to 4D .

In one embodiment, the acceleration sensor (240) may have at least a portion of the same or similar configuration as the sensor module (176) of FIG. 1. In one embodiment, the acceleration sensor (240) may be implemented as an inertial measurement unit (IMU) sensor, but is not limited thereto. In one embodiment, the head-mounted device (200) may obtain sensing data through the acceleration sensor (240). In one embodiment, the sensing data may be sensing information corresponding to the movement of a user wearing the head-mounted device (200). For example, when the user turns his/her head to the left, the head-mounted device (200) may obtain sensing data including sensing information corresponding to the movement of the user through the acceleration sensor (240).

In one embodiment, the memory (250) may have at least a portion of the same or similar configuration as the memory (130) of FIG. 1. For example, the memory (250) may be configured to temporarily or permanently store digital data and may include at least a portion of the configuration and/or functions of the memory (130) of FIG. 1.

The memory (250) according to one embodiment can store various instructions that can be executed by the processor (260). In addition, the memory (250) can store at least a portion of the program (140) of FIG. 1. Such instructions can include control commands such as logical operations and data input/output that can be recognized and executed by the processor (260). There is no limitation on the type and/or amount of data that the memory (250) can store, but this document will describe the configuration and function of the memory related to the method for providing an image according to various embodiments and the operation of the processor (260) that performs the method. The memory (250) can store various information, and the various information stored by the memory (250) will be described in detail below.

In one embodiment, the processor (260) may have at least a portion of the same or similar configuration as the processor (120) of FIG. 1. In one embodiment, the processor (260) may include one or more processors.

In one embodiment, the processor (260) may perform various operations by executing instructions stored in the memory (250).

In one embodiment, the processor (260) may acquire a first image corresponding to a first point of view through at least one of the first camera sensor (210) and the second camera sensor (220). In one embodiment, the first point of view may be a point of view (or a current point of view) at which an image is acquired through at least one of the first camera sensor (210) and the second camera sensor (220). In one embodiment, the first image may be an image provided to the user, and may be an image corresponding to the user's field of view. For example, the first image may be an image corresponding to the surrounding environment viewed by the user through the eyes.

In one embodiment, it is assumed that the first camera sensor (210) is a camera sensor corresponding to the user's left eye, and the second camera sensor (220) is a camera sensor corresponding to the user's right eye. The processor (260) can obtain a first image corresponding to the first viewpoint by obtaining an image corresponding to the left eye through the first camera sensor (210) and an image corresponding to the right eye through the second camera sensor (220). In this case, the first image corresponding to the first viewpoint can include an image corresponding to the left eye and an image corresponding to the right eye, respectively.

In one embodiment, the processor (260) may acquire an image from either the first camera sensor (210) or the second camera sensor (220). In this case, the first image corresponding to the first viewpoint may include only one of the image corresponding to the left eye or the image corresponding to the right eye. In one embodiment, only one of the first camera sensor (210) or the second camera sensor (220) may perform an operation of acquiring (or outputting) an image, but is not limited thereto. Each of the first camera sensor (210) and the second camera sensor (220) may perform an image acquisition operation, and the processor (260) may acquire only one of them.

In one embodiment, the processor (260) may acquire a second image corresponding to the first viewpoint through at least one third camera sensor (230). In one embodiment, the second image may be an image different from the first image. In one embodiment, when the second angle of view corresponding to at least one third camera sensor (230) is relatively larger than the first angles of view corresponding to the first camera sensor (210) and the second camera sensor (220), the second image may be an image corresponding to a relatively wider field of view than the first image. In one embodiment, the second image may be utilized as a reference image for predicting an image corresponding to the user's field of view. This will be described in detail with reference to FIGS. 4A to 4D.

In one embodiment, the processor (260) may use sensing data corresponding to the user's movement, together with a first image corresponding to a first angle of view (e.g., the user's field of view), a second image corresponding to a second angle of view (e.g., a reference image), to predict an image corresponding to the user's field of view at a point in time after a specified period of time has elapsed from the present. In one embodiment, the processor (260) may identify a third image corresponding to a second point in time after the first point in time based on the first image corresponding to the first point in time, the second image corresponding to the first point in time, and sensing data acquired through an acceleration sensor.

In one embodiment, the second point of view may be a point of view after the first point of view and may be a point of view before a third point of view at which an image is re-acquired through the first camera sensor (210) (and/or the second camera sensor (220)). In one embodiment, the third image may be an image of a future point of view corresponding to the user's field of view. For example, the third image may be a predicted image corresponding to the surrounding environment viewed by the user at a point of time after a specified period of time has elapsed from the present. The third image may be an image corresponding to the first angle of view in one embodiment. In one embodiment, the third image may be an image corresponding to the user's field of view and may include an image corresponding to the left eye and an image corresponding to the right eye, respectively.

In one embodiment, the processor (260) may use the learned artificial intelligence model to identify a third image corresponding to the second viewpoint. In one embodiment, the memory may include a first artificial intelligence model that outputs a third image corresponding to the second viewpoint. In one embodiment, the first artificial intelligence model may be implemented as a generative artificial intelligence model. In one embodiment, the first artificial intelligence model may be a model learned to output an image of a second viewpoint corresponding to the first angle of view when an image of a first viewpoint (e.g., a first image corresponding to a first angle of view and a first image corresponding to a second angle of view) and sensing data corresponding to a user's movement are input. In one embodiment, the processor (260) may input the first image corresponding to the first viewpoint, the second image corresponding to the first viewpoint, and the sensing data into the first artificial intelligence model. In one embodiment, the first artificial intelligence model may output a third image corresponding to the second viewpoint, and the processor (260) may identify the third image corresponding to the second viewpoint based on the output.

In one embodiment, the first artificial intelligence model may be a pre-learned model, but is not limited thereto, and of course, it may be learned in real time based on data input while the head mounting device (200) is operating.

In one embodiment, the sensing data may be six degrees of freedom (6DoF) acceleration information corresponding to a camera sensor included in the head-mounted device (200). In one embodiment, the processor (260) may also identify a third image corresponding to a second time point using a moving average value of the sensing data over a specified time period.

In one embodiment, the processor (260) may input information about characteristic parameters (e.g., at least one of a camera angle of view, a relative position between a plurality of cameras, and a camera shooting direction) corresponding to each of the first camera sensor (210), the second camera sensor (220), and at least one third camera sensor (230) into the first artificial intelligence model to identify a third image corresponding to the second time point. For example, the processor (260) may input information about sensing data and characteristic parameters, along with the first image of the first time point and the second image of the first time point, into the first artificial intelligence model to identify the third image corresponding to the second time point.

In one embodiment, the first artificial intelligence model may have a configuration at least partially identical or similar to the configuration of the generative artificial intelligence system illustrated in FIG. 10. In one embodiment, the first artificial intelligence model may be implemented as a generative adversarial network (GAN) model that generates images based on input data.

FIG. 3 is a flowchart for explaining an operation method of a head mounting device (e.g., the head mounting device (200) of FIG. 2) according to one embodiment.

Hereinafter, a method of operating a head-mounted device according to various embodiments will be described in detail. According to various embodiments, the operations performed by the head-mounted device described below may be executed by a processor (e.g., processor (260) of FIG. 2) including at least one processing circuitry of the head-mounted device. According to one embodiment, the operations performed by the head-mounted device may be stored in a memory (e.g., memory (250) of FIG. 2) and, when executed, executed by instructions that cause the processor (260) to operate. In the embodiments below, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel. Depending on the implementation, certain operations may be omitted.

Referring to FIG. 3, according to one embodiment, in operation 301, the head mounted device may acquire a first image (e.g., the first image of FIG. 2) corresponding to a first viewpoint through at least one of a first camera sensor (e.g., the first camera sensor (210) of FIG. 2) corresponding to a first angle of view and a second camera sensor (e.g., the second camera sensor (220) of FIG. 2) corresponding to the first angle of view. In one embodiment, the first image may be an image provided to a user, and may be an image corresponding to a field of view (or, the first angle of view) of the user. In one embodiment, the first image acquired through the first camera sensor and/or the second camera sensor may be an image corresponding to a surrounding environment viewed through the eyes of the user.

In one embodiment, when the first camera sensor is a camera sensor corresponding to the left eye, the head mounted device can obtain an image corresponding to the left eye from the first camera sensor as the first image. In one embodiment, when the second camera sensor is a camera sensor corresponding to the right eye, the head mounted device can obtain an image corresponding to the right eye from the second camera sensor as the first image. In one embodiment, the head mounted device can also obtain images (an image corresponding to the left eye and an image corresponding to the right eye) obtained from each of the first camera sensor and the second camera sensor as the first image.

According to one embodiment, in operation 303, the head-mounted device may acquire a second image (e.g., the second image of FIG. 2) corresponding to the first viewpoint through at least one third camera sensor (e.g., at least one third camera sensor (230) of FIG. 2) corresponding to the second viewpoint. In one embodiment, the second image may be an image corresponding to a different viewpoint from the first image. For example, when the second viewpoint is relatively larger than the first viewpoint, the second image may be an image corresponding to a relatively wider field of view than the first image. In one embodiment, when a plurality of third camera sensors are implemented, a plurality of second images corresponding to the first viewpoint may be provided. This will be described in detail with reference to FIGS. 4A and 4B.

In one embodiment, in operation 305, the head-mounted device can identify a third image corresponding to a second time point after the first time point based on a first image corresponding to the acquired first time point, a second image corresponding to the acquired first time point, and sensing data (e.g., sensing data of FIG. 2) acquired through an acceleration sensor (e.g., acceleration sensor (240) of FIG. 2).

In one embodiment, the head-mounted device may include an acceleration sensor, and may obtain sensing data corresponding to the user's movement through the acceleration sensor. In one embodiment, the head-mounted device may use the obtained sensing data, a first image corresponding to a first time point, and a second image corresponding to the first time point, to predict a third image corresponding to a second time point after the first time point. The third image may be an image corresponding to the first angle of view, in one embodiment. In one embodiment, the third image is an image corresponding to the user's field of view, and may include an image corresponding to the left eye and an image corresponding to the right eye, respectively.

FIGS. 4A to 4D are drawings for explaining a camera sensor and generated images according to one embodiment.

Referring to FIGS. 4A to 4D , according to one embodiment, a head mounting device (400, e.g., head mounting device (200) of FIG. 2 ) may include a plurality of camera sensors (401, 402, 403, 404, 410, and 420).

In one embodiment, the head mounted device (400) may include a first camera sensor (410, e.g., the first camera sensor (210) of FIG. 2) and a second camera sensor (420, e.g., the second camera sensor (220) of FIG. 2). In one embodiment, the first camera sensor (410) may be a camera sensor corresponding to the left eye of the user (40). In one embodiment, the first camera sensor (410) may be a camera sensor corresponding to a first angle of view (e.g., the first angle of view of FIG. 2) and may acquire an image corresponding to the left eye of the user (40). In one embodiment, the second camera sensor (420) may be a camera sensor corresponding to the right eye of the user (40). In one embodiment, the second camera sensor (420) may be a camera sensor corresponding to a first angle of view and may acquire an image corresponding to the right eye of the user (40). In one embodiment, the first image (e.g., the first image of FIG. 2) may include at least one of an image corresponding to the left eye of the user (40) and an image corresponding to the right eye of the user (40).

In one embodiment, the head mounted device (400) may include at least one third camera sensor (401, 402, 403, and 404). In one embodiment, the at least one third camera sensor (401, 402, 403, and 404) may be disposed at a designated location within the head mounted device (400), but is not limited thereto, and of course, may be disposed at a different location than the location illustrated in FIG. 4A. In one embodiment, although four third camera sensors are depicted as being disposed in the head mounted device (400) in FIG. 4A, the present invention is not limited thereto, and of course, a different number of third camera sensors may be disposed in the head mounted device (400).

In one embodiment, at least one third camera sensor (401, 402, 403, and 404) can acquire a second image (e.g., the second image of FIG. 2) corresponding to a different angle of view than the first image. In one embodiment, referring to FIG. 4B, the first camera sensor (410), the second camera sensor (420), and the at least one third camera sensor (401, 402, 403, and 404) can have different angles of view. In one embodiment, the first angle of view (43) corresponding to the first camera sensor (410) and the second camera sensor (420) can be less than the second angle of view (41) (e.g., the second angle of view of FIG. 2) corresponding to the at least one third camera sensor (401, 402, 403, and 404). Accordingly, the second image acquired through at least one third camera sensor (401, 402, 403 and 404) may be an image having a relatively wide field of view compared to the first image acquired through the first camera sensor (410) or the second camera sensor (420).

In one embodiment, as shown in FIGS. 4A and 4B, the first camera sensor (410) may be positioned at a position corresponding to the left eye of the user (40), and the second camera sensor (420) may be positioned at a position corresponding to the right eye of the user (40), but is not limited thereto. It is to be understood that the first camera sensor (410) may be positioned at a position corresponding to the right eye of the user (40), and the second camera sensor (420) may be positioned at a position corresponding to the left eye of the user (40).

In one embodiment, the second angle of view (41) corresponding to at least one of the third camera sensors (401, 402, 403, and 404) may be relatively larger than the first angle of view (43) corresponding to the first camera sensor (410) and the second camera sensor (420). In one embodiment, referring to FIG. 4C, among the fields of view (430), the fields of view (432) corresponding to the second image acquired through at least one of the third camera sensors (401, 402, 403, and 404) may be relatively wider than the fields of view (431) corresponding to the first image. Here, the fields of view (432) corresponding to the second image may be fields of view corresponding to images acquired from each of the at least one of the third camera sensors (401, 402, 403, and 404).

In one embodiment, the head mounted device (400) may generate a third image (e.g., the third image of FIG. 2) of a second viewpoint using movement information of the user (40). For example, when sensing data (e.g., sensing data of FIG. 2) corresponding to a movement of the user (40) raising his/her head is acquired, the head mounted device (400) may, based on the sensing data, identify an image of a field of view (433) corresponding to the movement of the user (40) among a field of view (432) corresponding to the second image, and identify this as a third image of the second viewpoint. In this case, the third image may include at least one of an image of the second viewpoint corresponding to the left eye and an image of the second viewpoint corresponding to the right eye.

In one embodiment, as illustrated in FIG. 4D, it is assumed that at least one third camera sensor (401, 402, 403, and 404) is positioned at a designated location of the head mounting device (400). Referring to FIG. 4D, each of the second images (401-1, 402-1, 403-1, and 404-1) acquired through each of the third camera sensors may be images corresponding to different field of view ranges from each of the image (410-1) acquired through the first camera sensor (410) and the image (420-1) acquired through the second camera sensor (420). For example, in the case of the second image (401-1) acquired through the third camera sensor (401) located relatively at the upper left of the head-mounted device (400), the image may have a relatively wide field of view (or, imaging range) for the left and upper regions compared to the image (410-1) acquired from the first camera sensor (410) corresponding to the left eye. Accordingly, as illustrated in FIG. 4C, the field of view (432) corresponding to the second images (401-1, 402-1, 403-1, and 404-1) may be relatively wider than the field of view (431) corresponding to the first image (e.g., at least one of the image (410-1) and the image (420-1)).

In one embodiment, if at least one of the third camera sensors (401, 402, 403 and 404) is implemented as an RGB camera, the second image may be a color image, but is not limited thereto, and at least one of the third camera sensors (401, 402, 403 and 404) may be implemented as a black and white camera, in which case the second image may be a black and white image.

In one embodiment, the head mounted device (400) may provide the identified third image to the user (40) when a third image corresponding to the second time point is identified. In one embodiment, the head mounted device (400) may further include a display (e.g., the display module (160) of FIG. 1). In one embodiment, the head mounted device (400) may display the third image corresponding to the identified second time point through the display. For example, after the first image corresponding to the first time point is displayed through the display, when the third image corresponding to the second time point is identified, the head mounted device (400) may display the third image through the display after the display of the first image is finished.

FIG. 5A is a flowchart illustrating a method for identifying an image corresponding to a second point in time (e.g., the second point in FIG. 2 ) according to one embodiment. FIG. 5B is a diagram illustrating a method for identifying an image corresponding to a second point in time (e.g., the second point in FIG. 2 ) according to one embodiment.

In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel. Depending on the implementation, certain operations may be omitted.

Referring to FIGS. 5A and 5B , in one embodiment, at operation 501, a head mounted device (e.g., head mounted device (200) of FIG. 2 ) may acquire at least one of a left-eye image corresponding to a first viewpoint (e.g., the first viewpoint of FIG. 2 ) and a right-eye image corresponding to the first viewpoint, via at least one of a first camera sensor (e.g., the first camera sensor (210) of FIG. 2 ) and a second camera sensor (e.g., the second camera sensor (220) of FIG. 2 ). In one embodiment, the first image may include at least one of a left-eye image corresponding to the first camera sensor and a right-eye image corresponding to the second camera sensor.

In one embodiment, the left-eye image corresponding to the first viewpoint may be an image corresponding to the left eye of the user (e.g., the user (40) of FIG. 4). In one embodiment, when the first camera sensor is implemented as a camera sensor corresponding to the left eye, the head-mounted device can obtain an image (or left-eye image) corresponding to the user's left eye through the first camera sensor. In one embodiment, when the second camera sensor is implemented as a camera sensor corresponding to the right eye, the head-mounted device can obtain an image (or right-eye image) corresponding to the user's right eye through the second camera sensor. However, the present invention is not limited thereto, and it is of course possible for the first camera sensor to correspond to the user's right eye and the second camera sensor to correspond to the user's left eye.

In one embodiment, the head mounted device may acquire images from either the first camera sensor or the second camera sensor, but is not limited thereto, and in one embodiment, the head mounted device may acquire images from each of the first camera sensor and the second camera sensor.

In one embodiment, in operation 503, the head-mounted device inputs at least one acquired image into a first artificial intelligence model (e.g., the first artificial intelligence model of FIG. 2) to identify a left-eye image corresponding to a second time point (e.g., the second time point of FIG. 2) and a right-eye image corresponding to the second time point. That is, even if the head-mounted device does not input the left-eye image corresponding to the first time point and the right-eye image corresponding to the first time point to the first artificial intelligence model, the head-mounted device can identify the left-eye image corresponding to the second time point and the right-eye image corresponding to the second time point.

In one embodiment, when either the left-eye image or the right-eye image (510, or the first image) corresponding to the first time point is acquired, the head-mounted device can input the acquired first image (510) into the first artificial intelligence model to confirm a third image (540) corresponding to the second time point. In one embodiment, the third image (540) may each include a left-eye image corresponding to the second time point and a right-eye image corresponding to the second time point, and the third image (540) may be an image of a plurality of frames. In one embodiment, the head-mounted device can display the confirmed third image (540) after the first image (510) corresponding to the first time point is provided.

In one embodiment, the head-mounted device can alternately acquire a left-eye image corresponding to the first camera sensor and a right-eye image corresponding to the second camera sensor. For example, when the head-mounted device acquires a left-eye image as a first image (510) corresponding to a first time point, the head-mounted device can acquire a right-eye image as a fourth image (520) corresponding to a third time point through the second camera sensor after a specified time has elapsed. After the fourth image (520) is acquired, the head-mounted device can acquire a left-eye image as a sixth image (530) corresponding to a fifth time point through the first camera sensor after a specified time has elapsed.

In one embodiment, the head-mounted device can input the acquired fourth image (520) into the first artificial intelligence model to confirm the fifth image (550) corresponding to the fourth point of view. In one embodiment, the fifth image (550) may include a left-eye image corresponding to the fourth point of view and a right-eye image corresponding to the fourth point of view, and the fifth image (550) may be an image of a plurality of frames.

In one embodiment, the head mounted device may end display of the third image (540) and display the fourth image (520) when the fourth image (520) is acquired after displaying a portion of the identified third image (540). In one embodiment, the head mounted device may end display of the identified fifth image (550) when display of the fourth image (520) is terminated. However, the present invention is not limited thereto, and in one embodiment, when display of the fourth image (520) is terminated, the head mounted device may generate a new image based on the remaining portion of the identified third image (540) and the identified fifth image (550). This will be described later.

In one embodiment, the head-mounted device may acquire the first image through only one of the first camera sensor and the second camera sensor. For example, the head-mounted device may acquire the first image through only the first camera sensor and input the acquired first image into the first artificial intelligence model to acquire the third image. However, the present invention is not limited thereto, and the head-mounted device may acquire the first image through only the second camera sensor and input the acquired second image into the first artificial intelligence model to acquire the third image. In one embodiment, when the first image is acquired through only one of the first camera sensor and the second camera sensor, the image acquired through the other one may be used as a reference image.

According to the above example, the head-mounted device can not only predict an image corresponding to the user's field of view using the left-eye and right-eye images respectively acquired via the camera sensor, but can also predict an image corresponding to the user's field of view using only one of the acquired left-eye and right-eye images. Accordingly, the power consumption and generation time for generating the predicted image can be reduced.

According to the example described above, since the predicted images acquired through the AI model are provided alongside the images acquired through the camera sensor, the user can be provided with images at a relatively high frame rate (FPS) compared to images acquired through a conventional camera sensor. Since the head-mounted device can provide highly responsive pass-through images, user satisfaction is enhanced.

According to the above-described example, the head-mounted device can generate a prediction image using only one of the plurality of pass-through cameras (e.g., the first camera sensor and the second camera sensor), and thus can provide the user with an image having relatively low latency compared to the case where the prediction image is generated using each of the plurality of pass-through cameras.

FIG. 6 is a flowchart illustrating a method of post-processing a confirmed image according to one embodiment.

In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel. Depending on the implementation, certain operations may be omitted.

Referring to FIG. 6, in one embodiment, at operation 601, a head-mounted device (e.g., head-mounted device (200) of FIG. 2) may input a left-eye image corresponding to the identified second point in time (e.g., left-eye image of FIG. 5) and a right-eye image corresponding to the identified second point in time (e.g., right-eye image of FIG. 5) into a second artificial intelligence model.

In one embodiment, the memory (e.g., memory (250) of FIG. 2) may further include a second artificial intelligence model. In one embodiment, the second artificial intelligence model may be a supervised-learning artificial intelligence model. In one embodiment, the second artificial intelligence model may be a model trained to output, when a confirmed third image (e.g., a left-eye image corresponding to the second viewpoint and a right-eye image corresponding to the confirmed second viewpoint) is input, a post-processed left-eye image corresponding to the second viewpoint and a post-processed right-eye image corresponding to the second viewpoint. In one embodiment, the post-processed left-eye image and the post-processed right-eye image may be images in which a viewpoint corresponding to the images is corrected.

In one embodiment, assume that a left-eye image is acquired through a first camera sensor (e.g., the first camera sensor (210) of FIG. 2) and a right-eye image is acquired through a second camera sensor (e.g., the second camera sensor (220) of FIG. 2). If the left-eye image and the right-eye image are acquired through different camera sensors, the head-mounted device can perform image correction based on the view point (or point of view (POV)) corresponding to each camera sensor, thereby improving the accuracy of the image provided to the user.

In one embodiment, the viewpoint may be obtained through an extrinsic parameter corresponding to each camera sensor. In one embodiment, the extrinsic parameter may be information about the position of the camera sensor in a 3D (dimensional) space and information about the direction the camera sensor is looking, and the viewpoint may be information obtained based on the extrinsic parameter. In one embodiment, the intrinsic parameters of the first camera sensor and the second camera sensor may be the same. The intrinsic parameters of the camera sensor may include information about how much the image panel moves, how much it is enlarged, and how much it is tilted. In one embodiment, when the intrinsic parameters of the first camera sensor and the second camera sensor are the same as described above, the head mounted device may perform correction on the image based on the viewpoint associated with the extrinsic parameter, thereby obtaining a highly accurate image.

In one embodiment, the head-mounted device may obtain a post-processed image by comparing an image acquired through a camera sensor with an image acquired through a first artificial intelligence model. In one embodiment, the second artificial intelligence model may be supervised and learned by comparing an image acquired through a camera sensor with an image identified through the first artificial intelligence model. In one embodiment, a third image identified through the first artificial intelligence model, together with a first image acquired through at least one of the first camera sensor and the second camera sensor, may be input as learning data to the second artificial intelligence model.

For example, a left-eye image corresponding to a first viewpoint, acquired through a first camera sensor, and a right-eye image corresponding to the first viewpoint, verified through a first artificial intelligence model, may be input as learning data to a second artificial intelligence model. In this case, a left-eye image corresponding to the first viewpoint, acquired through a first camera sensor, and a right-eye image acquired through a second camera sensor (e.g., the second camera sensor (220) of FIG. 2) may be input together as labels.

In one embodiment, in operation 603, the head-mounted device can identify a post-processed left-eye image corresponding to the second time point and a post-processed right-eye image corresponding to the second time point. In one embodiment, the head-mounted device can identify the image output from the second artificial intelligence model as a post-processed left-eye image corresponding to the second time point and a post-processed right-eye image corresponding to the second time point.

FIG. 7a is a flowchart illustrating a method for displaying an image according to one embodiment. FIG. 7b is a diagram illustrating a method for displaying an image according to one embodiment.

In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel. Depending on the implementation, certain operations may be omitted.

Referring to FIGS. 7A and 7B , according to one embodiment, at operation 701, a head mounted device (e.g., head mounted device (200) of FIG. 2 ) may determine whether a fourth image (730) corresponding to a third point in time after a second point in time (e.g., the second point in time of FIG. 2 ) has been acquired after at least a portion of a third image (740, e.g., the third image of FIG. 2 ) has been displayed.

In one embodiment, when a first image (720, for example, at least one of a left-eye image corresponding to the first time point (e.g., the left-eye image of FIG. 5A) and a right-eye image corresponding to the first time point (e.g., the right-eye image of FIG. 5A)) corresponding to the first time point is acquired, the head mounted device can identify a third image (740) corresponding to the second time point. The head mounted device can display the identified third image (740) through a display module (e.g., the display module (160) of FIG. 1). In one embodiment, the third image (740) can include images of a plurality of frames. In one embodiment, the head mounted device can identify whether a fourth image (730) corresponding to the third time point is acquired after at least a portion of the identified third image (740) is displayed. In one embodiment, the head-mounted device may acquire a fourth image (730) corresponding to a third viewpoint through at least one of a first camera sensor (e.g., the first camera sensor (210) of FIG. 2) or a second camera sensor (e.g., the second camera sensor (220) of FIG. 2).

In one embodiment, in operation 703, the head mounted device may terminate display of the third image (740) and display the fourth image (730) through the display based on the acquisition of the fourth image (730). In one embodiment, if the fourth image (730) is acquired after at least a portion of the third image (740) has been displayed, the head mounted device may display the fourth image (730) through the display and not display the remaining portion (750) of the third image (740) that is not displayed.

In one embodiment, a third image (740) acquired through a first artificial intelligence model (e.g., the first artificial intelligence model of FIG. 2) may be temporarily stored in a designated space (e.g., a frame buffer) within a head-mounted device. When a fourth image (730) is acquired after at least a portion of the third image (740) is displayed, the head-mounted device may delete the remaining portion (750) stored in the designated space and temporarily store the fourth image (730) in the designated space.

According to the example described above, even while the AI model is providing the predicted image to the user, if a new image is acquired through the camera sensor, the head-mounted device can terminate the provision of the predicted image and provide the user with the image acquired through the camera sensor. This can improve user satisfaction.

FIG. 8a is a flowchart illustrating a method for displaying an image according to one embodiment. FIG. 8b is a diagram illustrating a method for displaying an image according to one embodiment.

In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel. Depending on the implementation, certain operations may be omitted.

Referring to FIGS. 8A and 8B, in one embodiment, at operation 801, a head-mounted device (e.g., head-mounted device (200) of FIG. 2) can determine whether a fourth image (e.g., fourth image (730) of FIG. 7B) corresponding to a third point in time after the second point in time has been acquired after a third image (810, e.g., third image of FIG. 2) is displayed.

For example, let's assume that the number of frames of the third image acquired through the first artificial intelligence model is 6. The head-mounted device can display the third image (810) corresponding to the second time point through a display module (e.g., the display module (160) of FIG. 1). The head-mounted device can determine whether the fourth image is acquired until the display of the third image (810) is completed. However, the present invention is not limited thereto, and in one embodiment, the head-mounted device can determine whether the fourth image is acquired until the display of a part of the third image (810) is completed.

In one embodiment, at operation 803, the head mounted device may acquire updated sensing data (e.g., the sensing data of FIG. 2) via an acceleration sensor (e.g., the acceleration sensor (240) of FIG. 2) based on the fourth image not being acquired. In one embodiment, the head mounted device may acquire the updated sensing data when it is determined that the fourth image is not acquired after the third image (810) is displayed. For example, the head mounted device may acquire the sensing data at the time when it is determined that the fourth image is not acquired. However, the present invention is not limited thereto.

In one embodiment, at operation 805, the head mounted device may input a first image (e.g., the first image of FIG. 2), a second image corresponding to a first time point (e.g., the second image of FIG. 2), and updated sensing data into a first artificial intelligence model (e.g., the first artificial intelligence model of FIG. 2), thereby additionally generating a third image corresponding to the second time point.

In one embodiment, the head-mounted device can input updated sensing data, along with the first image and the second image used to generate the third image (810), into the first artificial intelligence model to confirm the added third image (820).

In one embodiment, the head mounted device may display the added third image (820) through the display after the display of the existing third image (810) is completed. In one embodiment, after a portion of the added third image (820) is displayed, if it is confirmed that a fourth image is acquired, the head mounted device may delete the remainder of the added third image (820) stored in a designated storage space (e.g., the frame buffer of FIGS. 7A and 7B) and temporarily store the fourth image (830). The head mounted device may display the stored fourth image (830) through the display.

According to the example described above, since 10 additional images are generated compared to the images acquired through the camera sensor, an image with an FPS 10 times larger than before can be provided to the user.

FIG. 9 is a flowchart illustrating a method of displaying an image according to one embodiment.

In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel. Depending on the implementation, certain operations may be omitted.

Referring to FIG. 9, according to one embodiment, a head-mounted device (e.g., the head-mounted device (200) of FIG. 2) may, in operation 901, determine whether a fourth image (e.g., the fourth image of FIGS. 7A and 7B) corresponding to a third point in time after the second point in time has been acquired after at least a portion of a third image (e.g., the third image of FIG. 2) has been displayed.

In one embodiment, when a first image corresponding to a first time point (e.g., a left-eye image corresponding to the first time point (e.g., a left-eye image of FIG. 5A) and a right-eye image corresponding to the first time point (e.g., a right-eye image of FIG. 5A)) is acquired, the head mounted device can identify a third image corresponding to a second time point. The head mounted device can display the identified third image through a display module (e.g., the display module (160) of FIG. 1). In one embodiment, the third image may include images of a plurality of frames. In one embodiment, the head mounted device can identify whether a fourth image corresponding to the third time point is acquired after at least a portion of the identified third image is displayed. In one embodiment, the head mounted device can acquire the fourth image corresponding to the third time point through at least one of a first camera sensor (e.g., the first camera sensor (210) of FIG. 2) or a second camera sensor (e.g., the second camera sensor (220) of FIG. 2).

In one embodiment, the head-mounted device, based on the acquisition of the fourth image in operation 903, may acquire a fifth image corresponding to the fourth time point using the remaining portion of the third image and the fourth image. In one embodiment, the head-mounted device may blend the acquired fourth image with the remaining portion of the third image that is not yet displayed using a designated algorithm. In one embodiment, the head-mounted device may identify the blended image as the fifth image.

According to one embodiment, the head mounted device may display the acquired fifth image via a display at operation 905.

According to the example described above, the head-mounted device can obtain and provide to the user an image blended with a predicted image obtained through an artificial intelligence model and an image obtained through a camera sensor. This allows for an image with improved accuracy to be provided to the user.

FIG. 10 is a diagram for explaining a generative artificial intelligence system (1000) according to one embodiment.

According to one embodiment, a user query/response interface (1010) can receive user input. The user input may be in the form of natural language, images, and/or videos, but is not limited thereto. Furthermore, context information may also be transmitted when the user input is transmitted. The context information may include various additional information at the time of the user input. For example, the additional information may include information about the application currently being used by the user or information about the user's location. Furthermore, the user input may be in a form that combines the aforementioned natural language, images, sounds, and context information. Furthermore, the user input may also be in a non-natural language form, such as selecting a menu. The user query/response interface (1010) can output the results of the generative artificial intelligence system to the user. The output may be in the form of natural language or specific content, and may also be provided in the form of an action requested by the user. The user query/response interface (1010) can output the results of the generative artificial intelligence system to the user. The output can be in natural language form, in the form of specific content, or in the form of an action requested by the user.

The AI framework (1040) can receive user input and coordinate and control each component necessary to perform the user's intention based on the user's query.

User input received from the user query/response interface (1010) can be transmitted to a prompt design component (1041). The prompt design component (1041) can be used to generate prompts suitable for inputting user input into a large language model (LLM) or a large multimodal model (LMM). The prompt design component (1041) can be an AI component that uses a machine learning algorithm or a neural network to develop better prompts over time. The prompt design component (1041) can access a knowledge component including user preference data, a prompt library, and prompt examples based on the user input to generate prompts and transmit the generated prompts to the LLM or LMM.

The API/Plug-in management component (1042) can communicate with external information when there is a request for additional information when passing user input as input to a generative model. The API/Plug-in management component (1042) can establish a channel for communicating with the outside of the AI Interface through the API, and can enable access to various data sources (e.g., knowledge repositories (1020)) through the established channel. In addition, the API/Plug-in management component (1042) can request the application/service component (1030) through the API for an action that ultimately performs the user input, rather than an intermediate result, when the action needs to be performed in the application or service. Information obtained from the outside can be used to generate a prompt in the prompt design component (1041) together with the user input, or can be passed as input to the generative model.

The output modification component (also called a refiner component) (1043) can fine-tune the output results from the generative model. For example, the output modification component (1043) can verify that the content generated through the LLM and/or LMM is not irrelevant, does not contain biased content, or does not contain harmful content. In addition, the output modification component (1043) can determine to what extent it matches the result desired by the user and, if additional processing is necessary, can proceed with the process. The output modification component (1043) can additionally configure and provide the user with hints to avoid unwanted output.

A generative AI model (1060) may generally refer to an artificial intelligence neural network that creates new types of data based on user input information. The generative AI model (1060) may include an image-generating model and/or a language-generating model. Representative models for generating images include a generative adversarial network (GAN) and a variational auto-encoder (VAE), and examples include a VAE and a Diffusion-based generative model that uses a Transformer structure. A language-generating model is a model trained to statistically output the most appropriate output based on input values, and representative examples include models such as CHAT-GPT 3 and CHAT-GPT 4. In addition, there are also LMMs (large multimodal models) that can recognize various types of data input, such as text, images, and voice, and generate new data corresponding to them.

FIGS. 11A and 11B are drawings showing the front and rear of a head mounting device (1100) according to one embodiment.

Referring to FIGS. 11A and 11B, in one embodiment, camera modules (1111, 1112, 1113, 1114, 1115, 1116) and/or depth sensors (1117) for obtaining information related to the surrounding environment of the head mounting device (1100) may be arranged on the first side (1110) of the housing.

In one embodiment, the camera modules (1111, 1112) can acquire images related to the environment surrounding the head-mounted device.

In one embodiment, the camera modules (1113, 1114, 1115, 1116) can acquire images while the head-mounted device (1100) is worn by the user. The camera modules (1113, 1114, 1115, 1116) can be used for hand detection and tracking, and user gesture (e.g., hand movement) recognition. The camera modules (1113, 1114, 1115, 1116) can be used for 11DoF, 6DoF head tracking, position (spatial, environmental) recognition, and/or movement recognition. In one embodiment, the camera modules (1111, 1112) can also be used for hand detection and tracking, and user gesture.

In one embodiment, a depth sensor (1117) may be configured to transmit a signal and receive a signal reflected from a subject, and may be used for purposes such as time of flight (TOF) to determine the distance to an object. Instead of or in addition to the depth sensor (1117), camera modules (1113, 1114, 1115, 1116) may determine the distance to an object.

According to one embodiment, a camera module (1125, 1126) for facial recognition and/or a display (1121) (and/or a lens) may be disposed on the second side (1120) of the housing.

In one embodiment, a face recognition camera module (1125, 1126) adjacent to the display (1121) may be used to recognize a user's face, or may recognize and/or track both eyes of the user.

In one embodiment, the display (1121) (and/or lens) may be disposed on the second side (1120) of the head mounted device (1100). In one embodiment, the head mounted device (1100) may not include camera modules (1115, 1116) among the plurality of camera modules (1113, 1114, 1115, 1116). Although not illustrated in FIGS. 11A and 11B , the head mounted device (1100) may further include at least one of the configurations illustrated in FIGS. 1 and 2 .

As described above, according to one embodiment, the head-mounted device (1100) may have a form factor for being worn on a user's head. The head-mounted device (1100) may further include a strap and/or a wearing member for being fixed on a body part of the user. The head-mounted device (1100) may provide a user experience based on augmented reality, virtual reality, and/or mixed reality while being worn on the user's head. However, the present disclosure is not limited thereto, and according to one embodiment, the head-mounted device of the present disclosure may include a configuration other than the configuration illustrated in FIGS. 11A and 11B , or may include a configuration in which the configuration illustrated in FIGS. 11A and 11B is deleted and/or modified, and the head-mounted device is not limited to the contents illustrated.

A head mounted device according to one embodiment of the present disclosure may include a first camera sensor corresponding to a first angle of view, a second camera sensor corresponding to the first angle of view, at least one third camera sensor corresponding to the second angle of view, an acceleration sensor, a memory storing instructions, and a processor. According to one embodiment, the instructions, when individually or collectively executed by the processor, may cause the head mounted device to acquire a first image corresponding to a first viewpoint through at least one of the first camera sensor and the second camera sensor.

In one embodiment, the instructions, when individually or collectively executed by the processor, may cause the head mounted device to acquire a second image corresponding to the first point in time via the at least one third camera sensor.

In one embodiment, the instructions, when individually or collectively executed by the processor, may cause the head mounted device to identify a first image corresponding to the first time point, a second image corresponding to the first time point, and a third image corresponding to a second time point after the first time point based on sensing data acquired through the acceleration sensor.

In one embodiment, the memory may further include a first artificial intelligence model.

In one embodiment, the instructions, when individually or collectively executed by the processor, may cause the head-mounted device to input a first image corresponding to the first time point, a second image corresponding to the first time point, and the acquired sensing data into the first artificial intelligence model to identify a third image corresponding to the second time point.

In one embodiment, the first image may include at least one of a left-eye image corresponding to the first camera sensor and a right-eye image corresponding to the second camera sensor.

In one embodiment, the instructions, when individually or collectively executed by the processor, may cause the head mounted device to acquire, via at least one of the first camera sensor and the second camera sensor, at least one of a left eye image corresponding to the first viewpoint and a right eye image corresponding to the first viewpoint.

In one embodiment, the instructions, when individually or collectively executed by the processor, may cause the head-mounted device to input the at least one acquired image into the first artificial intelligence model to identify a left-eye image corresponding to the second point in time and a right-eye image corresponding to the second point in time.

In one embodiment, the memory may further include a second artificial intelligence model.

In one embodiment, the instructions, when individually or collectively executed by the processor, may cause the head-mounted device to input a left-eye image corresponding to the identified second time point and a right-eye image corresponding to the identified second time point into the second artificial intelligence model, thereby identifying a post-processed left-eye image corresponding to the second time point and a post-processed right-eye image corresponding to the second time point.

According to one embodiment, the second artificial intelligence model may be supervised-learned by comparing a first image acquired through at least one of the first camera sensor and the second camera sensor with a third image identified through the first artificial intelligence model.

In one embodiment, the instructions, when individually or collectively executed by the processor, may cause the head mounted device to alternately acquire a left-eye image corresponding to the first camera sensor and a right-eye image corresponding to the second camera sensor.

According to one embodiment, the head-mounted device may further include a display.

In one embodiment, the instructions, when individually or collectively executed by the processor, may cause the head mounted device to display a third image corresponding to the identified second point in time through the display.

In one embodiment, the third image may include images of a plurality of frames.

In one embodiment, the first point in time and the third point in time after the second point in time may be points in time at which images are acquired from at least one of the first camera sensor, the second camera sensor, and the at least one third camera sensor.

In one embodiment, the instructions, when individually or collectively executed by the processor, may cause the head mounted device to determine whether a fourth image corresponding to the third point in time has been acquired after at least a portion of the third image has been displayed.

In one embodiment, the instructions, when individually or collectively executed by the processor, may cause the head mounted device to terminate display of the third image and display the fourth image through the display based on the acquisition of the fourth image.

In one embodiment, the third image may include images of a plurality of frames.

In one embodiment, the instructions, when individually or collectively executed by the processor, may cause the head mounted device to determine whether a fourth image corresponding to the third point in time has been acquired after at least a portion of the third image has been displayed.

In one embodiment, the instructions, when individually or collectively executed by the processor, may cause the head mounted device to acquire a fifth image corresponding to a fourth point in time using the remaining portion of the third image and the fourth image, based on the acquisition of the fourth image.

In one embodiment, the instructions, when individually or collectively executed by the processor, may cause the head mounted device to display the acquired fifth image through the display.

In one embodiment, the instructions, when individually or collectively executed by the processor, may cause the head mounted device to determine whether a fourth image corresponding to the third point in time has been acquired after the third image has been displayed.

In one embodiment, the instructions, when individually or collectively executed by the processor, may cause the head mounted device to acquire updated sensing data via the acceleration sensor based on the fourth image not being acquired.

In one embodiment, the instructions, when individually or collectively executed by the processor, may cause the head mounted device to input the first image, the second image corresponding to the first point in time, and the updated sensing data into a first artificial intelligence model to additionally generate a third image corresponding to the second point in time.

A method of operating a head mounted device according to one embodiment of the present disclosure may include an operation of acquiring a first image corresponding to a first viewpoint through at least one of a first camera sensor of the head mounted device corresponding to a first angle of view and a second camera sensor of the head mounted device corresponding to the first angle of view.

In one embodiment, the method may include acquiring a second image corresponding to the first viewpoint through at least one third camera sensor of the head-mounted device corresponding to the second angle of view.

According to one embodiment, the operating method may include an operation of confirming a third image corresponding to a second time point after the first time point based on a first image corresponding to the acquired first time point, a second image corresponding to the acquired first time point, and sensing data acquired through an acceleration sensor of the head-mounted device.

According to one embodiment, the operation of confirming the third image may input the first image corresponding to the first time point, the second image corresponding to the first time point, and the acquired sensing data into the first artificial intelligence model to confirm the third image corresponding to the second time point.

In one embodiment, the first image may include at least one of a left-eye image corresponding to the first camera sensor and a right-eye image corresponding to the second camera sensor.

According to one embodiment, the operation of acquiring the first image may include an operation of acquiring at least one of a left-eye image corresponding to the first viewpoint and a right-eye image corresponding to the first viewpoint, through at least one of the first camera sensor and the second camera sensor.

According to one embodiment, the operation of confirming the third image may include an operation of inputting at least one of the acquired images into the first artificial intelligence model to confirm a left eye image corresponding to the second point in time and a right eye image corresponding to the second point in time.

According to one embodiment, the operating method may further include an operation of inputting a left eye image corresponding to the confirmed second time point and a right eye image corresponding to the confirmed second time point into a second artificial intelligence model, thereby confirming a post-processed left eye image corresponding to the second time point and a post-processed right eye image corresponding to the second time point.

According to one embodiment, the second artificial intelligence model may be supervised-learned by comparing a first image acquired through at least one of the first camera sensor and the second camera sensor with a third image identified through the first artificial intelligence model.

According to one embodiment, the control method may include an operation of alternately acquiring a left-eye image corresponding to the first camera sensor and a right-eye image corresponding to the second camera sensor.

According to one embodiment, the operating method may further include an operation of displaying a third image corresponding to the identified second point in time through a display of the head-mounted device.

In one embodiment, the third image may include images of a plurality of frames.

According to one embodiment, the method may include an operation of determining whether a fourth image corresponding to the third point in time has been acquired after at least a portion of the third image has been displayed.

In one embodiment, the first point in time and the third point in time after the second point in time may be points in time at which images are acquired from at least one of the first camera sensor, the second camera sensor, and the at least one third camera sensor.

In one embodiment, the method may include an operation of terminating display of the third image and displaying the fourth image through the display based on acquisition of the fourth image.

In one embodiment, the third image may include images of a plurality of frames.

According to one embodiment, the method may include an operation of determining whether a fourth image corresponding to the third point in time has been acquired after at least a portion of the third image has been displayed.

In one embodiment, the method may include an operation of acquiring a fifth image corresponding to a fourth point in time using the remaining portion of the third image and the fourth image, based on the acquisition of the fourth image.

According to one embodiment, the method of operation may include an operation of displaying the acquired fifth image through the display.

In a non-transitory storage medium storing computer-readable instructions according to one embodiment of the present disclosure, the instructions, when executed by a processor of a head mounted device, can cause the head mounted device to acquire a first image corresponding to a first viewpoint through at least one of a first camera sensor of the head mounted device corresponding to a first angle of view and a second camera sensor of the head mounted device corresponding to the first angle of view.

In one embodiment, the instructions, when individually or collectively executed by the processor, may cause the head mounted device to acquire a second image corresponding to the first viewpoint via at least one third camera sensor of the head mounted device corresponding to the second angle of view.

In one embodiment, the instructions, when individually or collectively executed by the processor, may cause the head mounted device to identify a first image corresponding to the first time point, a second image corresponding to the first time point, and a third image corresponding to a second time point after the first time point based on sensing data acquired through an acceleration sensor of the head mounted device.

The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned will be clearly understood by a person having ordinary skill in the art to which the present disclosure pertains.

As used herein, the term “if” will be understood to mean “when, upon,” “in response to deciding,” or “in response to detecting,” depending on the context. Similarly, “if it is decided to do,” or “if [the stated condition or event] is detected,” will optionally be understood to mean “upon deciding,” or “in response to deciding,” “upon detecting [the stated condition or event],” or “in response to detecting [the stated condition or event].”

The devices described above may be implemented as hardware components, software components, and/or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. A processing device (or processing circuit) may execute an operating system (OS) and one or more software applications running on the operating system. In addition, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and/or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and/or data may be embodied in any type of machine, component, physical device, computer storage medium, or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. In this case, the medium may be one that continuously stores a computer-executable program or one that temporarily stores it for execution or download. In addition, the medium may be various recording or storage means in the form of a single or multiple hardware combinations, and is not limited to a medium directly connected to a computer system, but may also be distributed over a network. Examples of the medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and those configured to store program commands, including ROM, RAM, and flash memory. In addition, examples of other media may include an app store that distributes applications, a site that supplies or distributes various software, or a recording or storage medium managed by a server.

Although the embodiments described above have been described by way of limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above teachings. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and/or components such as the described systems, structures, devices, and circuits are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

According to one embodiment, the method according to various embodiments disclosed in this document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and arranged in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims (15)

There is a head mounted device, A first camera sensor corresponding to the first angle of view; A second camera sensor corresponding to the first angle of view; At least one third camera sensor corresponding to the second angle of view; acceleration sensor; Memory that stores instructions; and Contains a processor, The above instructions, when individually or collectively executed by the processor, cause the head mounting device to: Acquire a first image corresponding to a first point in time through at least one of the first camera sensor and the second camera sensor, Acquire a second image corresponding to the first point in time through at least one third camera sensor, A head-mounted device that causes a third image corresponding to a second time point after the first time point to be identified based on a first image corresponding to the first time point, a second image corresponding to the first time point, and sensing data acquired through the acceleration sensor. In the first paragraph, The above memory is, Including the first artificial intelligence model, The above instructions, when individually or collectively executed by the processor, cause the head mounting device to: A first image corresponding to the first point in time, a second image corresponding to the first point in time, and the acquired sensing data are input to the first artificial intelligence model to cause a third image corresponding to the second point in time to be confirmed, wherein the first image is At least one of a left eye image corresponding to the first camera sensor and a right eye image corresponding to the second camera sensor, The above instructions, when individually or collectively executed by the processor, cause the head mounting device to: Obtaining at least one of a left eye image corresponding to the first viewpoint and a right eye image corresponding to the first viewpoint through at least one of the first camera sensor and the second camera sensor, A head-mounted device that inputs at least one of the acquired images into the first artificial intelligence model, thereby causing the left-eye image corresponding to the second time point and the right-eye image corresponding to the second time point to be identified. In claim 1 or 2, The above memory is, Including a second artificial intelligence model, The above instructions, when individually or collectively executed by the processor, cause the head mounting device to: By inputting the left eye image corresponding to the second time point confirmed above and the right eye image corresponding to the second time point confirmed above into the second artificial intelligence model, a post-processed left eye image corresponding to the second time point and a post-processed right eye image corresponding to the second time point are caused to be confirmed, The above second artificial intelligence model is, A head-mounted device that performs supervised learning by comparing a first image acquired through at least one of the first camera sensor and the second camera sensor with a third image confirmed through the first artificial intelligence model. In any one of the first to third paragraphs, The above instructions, when individually or collectively executed by the processor, cause the head mounting device to: A head-mounted device that causes a left-eye image corresponding to the first camera sensor and a right-eye image corresponding to the second camera sensor to be alternately acquired. In any one of the first to fourth paragraphs, The above head mounting device, including display; The above instructions, when individually or collectively executed by the processor, cause the head mounting device to: A head-mounted device that causes a third image corresponding to the second point in time identified above to be displayed through the display. In any one of the first to fifth paragraphs, The third image above is, Contains images of multiple frames, The first point in time above, and the third point in time after the second point in time above, A point in time when an image is acquired from at least one of the first camera sensor, the second camera sensor, and the at least one third camera sensor, The above instructions, when individually or collectively executed by the processor, cause the head mounting device to: After at least a portion of the third image is displayed, it is determined whether a fourth image corresponding to the third time point is acquired, A head mounted device that causes display of the third image to be terminated and the fourth image to be displayed through the display based on acquisition of the fourth image. In any one of claims 1 to 6, The third image above is, Contains images of multiple frames, The first point in time above, and the third point in time after the second point in time above, A point in time when an image is acquired from at least one of the first camera sensor, the second camera sensor, and the at least one third camera sensor, The above instructions, when individually or collectively executed by the processor, cause the head mounting device to: After at least a portion of the third image is displayed, it is determined whether a fourth image corresponding to the third time point is acquired, Based on the acquisition of the fourth image, a fifth image corresponding to the fourth time point is acquired using the remaining part of the third image and the fourth image, A head-mounted device that causes the fifth image obtained above to be displayed through the display. In any one of the first to seventh paragraphs, The above memory is, Including the first artificial intelligence model, The first point in time above, and the third point in time after the second point in time above, A point in time when an image is acquired from at least one of the first camera sensor, the second camera sensor, and the at least one third camera sensor, The above instructions, when individually or collectively executed by the processor, cause the head mounting device to: After the third image is displayed, it is checked whether the fourth image corresponding to the third time point is acquired, Based on the above fourth image not being acquired, updated sensing data is acquired through the acceleration sensor, A head-mounted device that inputs the first image, the second image corresponding to the first point in time, and the updated sensing data into a first artificial intelligence model, thereby causing a third image corresponding to the second point in time to be additionally generated. In a method of operating a head mounted device, An operation of acquiring a first image corresponding to a first viewpoint through at least one of a first camera sensor of the head-mounted device corresponding to a first angle of view and a second camera sensor of the head-mounted device corresponding to the first angle of view; An operation of acquiring a second image corresponding to the first viewpoint through at least one third camera sensor of the head-mounted device corresponding to the second angle of view; and An operating method comprising: an operation of confirming a first image corresponding to the first point in time obtained above, a second image corresponding to the first point in time obtained above, and a third image corresponding to a second point in time after the first point in time based on sensing data acquired through an acceleration sensor of the head mounting device. In paragraph 9, The action of checking the third image above is: A first image corresponding to the first time point, a second image corresponding to the first time point, and the acquired sensing data are input into a first artificial intelligence model to confirm a third image corresponding to the second time point, The first image above is, At least one of a left eye image corresponding to the first camera sensor and a right eye image corresponding to the second camera sensor, The operation of obtaining the above first image is as follows: An operation of acquiring at least one of a left-eye image corresponding to the first point of view and a right-eye image corresponding to the first point of view through at least one of the first camera sensor and the second camera sensor, The action of checking the third image above is: An operating method comprising an operation of inputting at least one image obtained above into the first artificial intelligence model to confirm a left eye image corresponding to the second time point and a right eye image corresponding to the second time point. In claim 9 or 10, It further includes an operation of inputting the left eye image corresponding to the second time point confirmed above and the right eye image corresponding to the second time point confirmed above into a second artificial intelligence model, and confirming the post-processed left eye image corresponding to the second time point and the post-processed right eye image corresponding to the second time point; The above second artificial intelligence model is, Supervised learning is performed by comparing a first image acquired through at least one of the first camera sensor and the second camera sensor with a third image confirmed through the first artificial intelligence model. The operation of obtaining the above first image is as follows: An operating method comprising an operation of alternately acquiring a left-eye image corresponding to the first camera sensor and a right-eye image corresponding to the second camera sensor. In any one of the 9th to 12th clauses, An operating method further comprising: an operation of displaying a third image corresponding to the second point in time confirmed above through a display of the head mounting device. In any one of the 9th to 12th clauses, The third image above is, Contains images of multiple frames, The first point in time above, and the third point in time after the second point in time above, A point in time when an image is acquired from at least one of the first camera sensor, the second camera sensor, and the at least one third camera sensor, The above method of operation is, After at least a portion of the third image is displayed, an operation of checking whether a fourth image corresponding to the third time point has been acquired; and An operating method further comprising: an operation of terminating the display of the third image and displaying the fourth image through the display based on the acquisition of the fourth image. In any one of Articles 11 to 13, The third image above is, Contains images of multiple frames, The first point in time above, and the third point in time after the second point in time above, A point in time when an image is acquired from at least one of the first camera sensor, the second camera sensor, and the at least one third camera sensor, The above method of operation is, An operation of checking whether a fourth image corresponding to the third time point has been acquired after at least a part of the third image is displayed; An operation of acquiring a fifth image corresponding to a fourth time point using the remaining portion of the third image and the fourth image based on the acquisition of the fourth image; and An operating method, comprising: an operation of displaying the acquired fifth image through the display; In a non-transitory storage medium storing computer-readable instructions, the instructions, when executed by a processor of a head mounted device, cause the head mounted device to: Acquire a first image corresponding to a first viewpoint through at least one of a first camera sensor of the head-mounted device corresponding to a first angle of view and a second camera sensor of the head-mounted device corresponding to the first angle of view, Acquire a second image corresponding to the first viewpoint through at least one third camera sensor of the head-mounted device corresponding to the second viewpoint, A non-transitory storage medium that causes a third image corresponding to a second time point after the first time point to be identified based on sensing data acquired through a first image corresponding to the first time point, a second image corresponding to the first time point, and an acceleration sensor of the head mounting device.