WO2025154911A1 - Method and device for acquiring content related to target object - Google Patents

Abstract

An electronic device according to the present invention includes: a display; a processor; and a memory for storing instructions. The instructions, when executed by the processor, cause the electronic device to: acquire image information corresponding to a target object selected from among physical objects arranged in a physical space in the vicinity of the electronic device; acquire content related to the target object on the basis of the acquired image information and information about the vicinity of the electronic device; deactivate an interaction object, operable in response to a user input, on the basis of satisfying a condition related to output of the content; and output the acquired content in an area corresponding to the target object through the display.

Description

Method and device for obtaining content related to a target object

Below, a technique for obtaining content related to a target object is disclosed.

Recently, Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR) technologies that apply computer graphics technology are being developed. At this time, VR technology refers to a technology that uses a computer to create a virtual space that does not exist in the real world and then makes the virtual space feel like reality, and AR or MR technology refers to a technology that adds computer-generated information on top of the real world to express it, that is, a technology that allows real-time interaction with users by combining the real world and the virtual world.

Among these, augmented reality and mixed reality technologies are being used in conjunction with technologies in various fields (e.g., broadcasting technology, medical technology, and game technology). Representative examples of augmented reality technology being used in the broadcasting field include cases where the weather map in front of a weathercaster giving a weather forecast on TV changes naturally, or where advertising images that do not exist in the stadium are inserted into the screen and broadcast as if they were actually in the stadium during a sports broadcast.

As a representative service that provides augmented reality or mixed reality to users, there is the metaverse. This metaverse is a compound word of 'meta', meaning fiction and abstraction, and 'universe', meaning the real world, and refers to a three-dimensional virtual world. The metaverse is a more advanced concept than the existing term virtual reality environment, and provides an augmented reality environment in which virtual worlds such as the web and the internet are absorbed into the real world.

The above information may be provided as related art to aid in understanding this document. None of the above is claimed to be prior art related to this document or can be used to determine prior art.

An electronic device comprises: a display; at least one processor including a processing circuit; and a memory including one or more storage media storing instructions, wherein when the instructions are executed by the at least one processor, the instructions individually or collectively cause the electronic device to obtain image information corresponding to a target object determined from among physical objects arranged in a physical space around the electronic device, obtain content related to the target object based on the obtained image information and the surrounding information of the electronic device, deactivate an interactive object operable in response to a user's input based on satisfying a condition regarding output of the content, and output the obtained content in an area corresponding to the target object through the display.

A method performed by an electronic device may include: an operation of acquiring image information corresponding to a target object determined from among physical objects arranged in a physical space around the electronic device; an operation of acquiring content related to the target object based on the acquired image information and surrounding information of the electronic device; an operation of deactivating an interaction object operable in response to a user's input based on satisfying a condition regarding output of the content; and an operation of outputting the acquired content in an area corresponding to the target object through a display.

FIG. 1 is a block diagram illustrating an exemplary configuration of an electronic device according to various embodiments.

FIG. 2 illustrates an example of an optical see-through device according to various embodiments.

FIG. 3 illustrates examples of optical systems for an eye tracking camera, a transparent member, and a display according to various embodiments.

FIGS. 4A and 4B are diagrams showing examples of the front and back of an electronic device according to various embodiments.

FIG. 5 illustrates examples of construction of a virtual space, input from a user within the virtual space, and output to the user according to various embodiments.

FIG. 6 is a diagram illustrating an example of an operation of an electronic device providing space to a user according to various embodiments.

FIG. 7 is a flowchart illustrating an example of a method for an electronic device to output content related to a target object according to various embodiments.

FIG. 8 illustrates an example of an operation of an electronic device providing a three-dimensional image of a space according to various embodiments.

FIG. 9 is a drawing illustrating an example of an operation in which the operation mode of an electronic device is switched between a basic operation mode and a rest operation mode when a target object is a movable object according to various embodiments.

FIG. 10 is a drawing illustrating an example of an operation in which the operation mode of an electronic device is switched between a basic operation mode and a rest operation mode when displaying a scene in an internal area of a target object according to various embodiments.

FIG. 11 is a diagram illustrating an example of an operation of an electronic device obtaining content based on a content creation model according to various embodiments.

FIG. 12 is a drawing illustrating an example of an operation of an electronic device according to various embodiments to output content in an area larger than an area corresponding to a target object.

FIG. 13 is a diagram illustrating an example of a configuration for obtaining and/or outputting content when an electronic device selects multiple target objects according to various embodiments.

FIG. 14 is a block diagram illustrating an example configuration of an electronic device according to various embodiments.

Hereinafter, embodiments will be described in detail with reference to the attached drawings. In describing with reference to the attached drawings, identical components are given the same reference numerals regardless of the drawing numbers, and redundant descriptions thereof will be omitted.

FIG. 1 is a block diagram illustrating an exemplary configuration of an electronic device according to various embodiments.

FIG. 1 is a block diagram of an electronic device (101) in a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with the electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of the electronic device (104) or the server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

The processor (120) may control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) by executing, for example, software (e.g., a program (140)), and may perform various data processing or calculations. The processor (120) may include at least one processor including a processing circuit. According to one embodiment, as at least a part of the data processing or calculation, the processor (120) may store a command or data received from another component (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the command or data stored in the volatile memory (132), and store result data in a nonvolatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

The auxiliary processor (123) may control at least a portion of functions or states associated with at least one of the components of the electronic device (101) (e.g., the display module (160), the sensor module (176), or the communication module (190)), for example, while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing device) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. Such learning may be performed, for example, in the electronic device (101) itself on which the artificial intelligence model is executed, or may be performed through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers. The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), deep Q-networks, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may additionally or alternatively include a software structure.

The memory (130) can store various data used by at least one component (e.g., the processor (120) or the sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., the program (140)) and input data or output data for commands related thereto. The memory (130) can include a volatile memory (132) or a nonvolatile memory (134). According to one embodiment, the memory (130) can include one or more storage media that store instructions that cause the operation of the electronic device (101).

The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

The audio output module (155) can output an audio signal to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes such as multimedia playback or recording playback. The receiver can be used to receive an incoming call. According to one embodiment, the receiver can be implemented separately from the speaker or as a part thereof.

A display module (160) (e.g., a display) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) can include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) can include a touch sensor configured to detect a touch, or a pressure sensor configured to measure a strength of a force generated by the touch.

The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can obtain sound through an input module (150), or output sound through an audio output module (155), or an external electronic device (e.g., an electronic device (102)) (e.g., a speaker or a headphone) directly or wirelessly connected to the electronic device (101).

The sensor module (176) can detect an operating state (e.g., power or temperature) of the electronic device (101) or an external environmental state (e.g., user state) and generate an electric signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., the electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that a user can perceive through a tactile or kinesthetic sense. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module (180) can capture still images and moving images. According to one embodiment, the camera module (180) can include one or more lenses, image sensors, image signal processors, or flashes.

The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

The battery (189) can power at least one component of the electronic device (101). In one embodiment, the battery (189) can include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

The communication module (190) may support establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., the electronic device (102), the electronic device (104), or the server (108)), and performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., the application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, a corresponding communication module may communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data relation)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as a plurality of separate components (e.g., multiple chips). The wireless communication module (192) may use subscriber information (e.g., international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196) to identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199).

The wireless communication module (192) can support a 5G network and next-generation communication technology after a 4G network, for example, NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), terminal power minimization and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) may support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) may support various requirements specified in an electronic device (101), an external electronic device (e.g., electronic device (104)), or a network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL) each, or 1 ms or less for round trip) for URLLC realization.

The antenna module (197) can transmit or receive signals or power to or from the outside (e.g., an external electronic device). According to one embodiment, the antenna module (197) can include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) can include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), can be selected from the plurality of antennas by, for example, the communication module (190). A signal or power can be transmitted or received between the communication module (190) and the external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) can be additionally formed as a part of the antenna module (197).

According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC positioned on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) positioned on or adjacent a second side (e.g., a top side or a side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

At least some of the above components can be connected to each other and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

According to one embodiment, commands or data may be transmitted or received between an electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199).

Each of the external electronic devices (102, 103) and the server (108) may be the same type of device as or different from the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 103) or the server (108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of executing the function or service by itself or in addition, request one or more external electronic devices to perform at least a part of the function or service. The one or more external electronic devices that receive the request may execute at least a part of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a part of a response to the request. In this specification, an example is mainly described in which an electronic device (101) is an augmented reality device (e.g., an electronic device (201) of FIG. 2, an electronic device (301) of FIG. 3, or an electronic device (401) of FIG. 4), and an external electronic device (102, 103) or a server (108) among servers transmits the results of executing a virtual space and additional functions or services related to the virtual space to the electronic device (101).

The server (108) may include a processor (181), a communication module (182), and a memory (183). The processor (181), the communication module (182), and the memory (183) may be configured similarly to the processor (120), the communication module (190), and the memory (130) of the electronic device (101). For example, the processor (181) may provide a virtual space and interaction between users within the virtual space by executing a command stored in the memory (183). The processor (181) may generate at least one of visual information, auditory information, or tactile information of the virtual space and an object within the virtual space. For example, as visual information, the processor (181) may generate rendering data (e.g., visual rendering data) that renders the appearance (e.g., shape, size, color, or texture) of the virtual space and the appearance (e.g., shape, size, color, or texture) of an object positioned within the virtual space. In addition, the processor (181) may generate rendering data that renders a change (e.g., a change in the appearance of an object, a sound generation, or a tactile generation) based on at least one of an interaction between objects (e.g., a physical object, a virtual object, or an avatar object) in a virtual space, or a user's input to an object (e.g., a physical object, a virtual object, or an avatar object). The communication module (182) may establish communication with a first electronic device (e.g., an electronic device (101)) of a user and a second electronic device (e.g., an electronic device (102)) of another user. The communication module (182) may transmit at least one of the visual information, the tactile information, or the auditory information described above to the first electronic device and the second electronic device. For example, the communication module (182) may transmit rendering data.

For example, the server (108) renders content data executed in the application and transmits it to the electronic device (101), and the electronic device (101) that receives the data can output the content data to the display module (160). If the electronic device (101) detects user movement through an IMU sensor or the like, the processor (120) of the electronic device (101) can correct the rendering data received from the external electronic device (102) based on the movement information and output it to the display module (160). Alternatively, the movement information can be transmitted to the server (108) to request rendering so that the screen data is updated accordingly. However, the present invention is not limited thereto, and the rendering described above can be performed by various forms of external electronic devices (102, 103), such as a smartphone or a case device that can store and charge the electronic device (101). Rendering data corresponding to the virtual space described above created by the external electronic device (102, 103) can be provided to the electronic device (101). As another example, the electronic device (101) may receive virtual space information (e.g., vertex coordinates, texture, color defining the virtual space) and object information (e.g., vertex coordinates, texture, color defining the appearance of the object) from the server (108) and perform rendering on its own based on the received data.

FIG. 2 illustrates an example of an optical see-through device according to various embodiments.

The electronic device (201) may include at least one of a display (e.g., a display module (160) of FIG. 1), a vision sensor, a light source (230a, 230b), an optical element, or a substrate. An electronic device (201) that has a transparent display and provides an image through the transparent display may be referred to as an optical see-through device (OST device).

The display may include, for example, a liquid crystal display (LCD), a digital mirror device (DMD), a liquid crystal on silicon (LCoS), an organic light emitting diode (OLED), or a micro light emitting diode (micro LED).

In one embodiment, when the display is formed of one of a liquid crystal display, a digital mirror display, or a silicon liquid crystal display, the electronic device (201) may include a light source (230a, 230b) that irradiates light to a screen output area of the display (e.g., a screen display unit (215a, 215b)). In another embodiment, when the display can generate light on its own, for example, when it is formed of one of an organic light emitting diode or a micro LED, the electronic device (201) may provide a good quality virtual image to the user even without including a separate light source (230a, 230b). In one embodiment, if the display is implemented with an organic light emitting diode or a micro LED, the light source (230a, 230b) is unnecessary, and thus the electronic device (201) may be lightweight.

Referring to FIG. 2, the electronic device (201) may include a display, a first transparent member (225a) and/or a second transparent member (225b), and a user may use the electronic device (201) while wearing it on his or her face. The first transparent member (225a) and/or the second transparent member (225b) may be formed of a glass plate, a plastic plate, or a polymer, and may be manufactured to be transparent or translucent. According to one embodiment, the first transparent member (225a) may be arranged to face the user's right eye, and the second transparent member (225b) may be arranged to face the user's left eye. The display may include a first display (205) that outputs a first image (e.g., a right image) corresponding to the first transparent member (225a) and a second display (210) that outputs a second image (e.g., a left image) corresponding to the second transparent member (225b). In one embodiment, when each display is transparent, each display and transparent member can be positioned at a position facing the user's eyes to form a screen display unit (215a, 215b).

In one embodiment, light emitted from the display (205, 210) may be guided along an optical path through an input optical member (220a, 220b) into a waveguide. Light traveling inside the waveguide may be guided toward a user's eyes through an output optical member (e.g., an output optical member (340) of FIG. 3). The screen display units (215a, 215b) may be determined based on the light emitted toward the user's eyes.

For example, light emitted from the display (205, 210) may be reflected in the grating region of the waveguide formed in the input optical member (220a, 220b) and the screen display portions (215a, 215b) and transmitted to the user's eyes.

The optical element may include at least one of a lens or an optical waveguide.

The lens can adjust the focus so that the screen output to the display can be seen by the user's eyes. The lens can include, for example, at least one of a Fresnel lens, a pancake lens, or a multi-channel lens.

The optical waveguide can transmit image light generated from the display to the user's eyes. For example, the image light can refer to light emitted by a light source (230a, 230b) that passes through a screen output area of the display. The optical waveguide can be made of glass, plastic, or polymer. The optical waveguide can include a nano-pattern formed on a portion of an inner or outer surface, for example, a grating structure having a polygonal or curved shape. An exemplary structure of the optical waveguide is described later in FIG. 3.

The vision sensor may include at least one of a camera sensor or a depth sensor.

The first camera (265a, 265b) is a recognition camera and may be a camera used for 3DoF, 6DoF head tracking, hand detection, hand tracking, and space recognition. The first camera (265a, 265b) may mainly include a GS (Global shutter) camera. Since a stereo camera is required for head tracking and space recognition, the first camera (265a, 265b) may include two or more GS cameras. The GS camera may have superior performance compared to the RS (Rolling shutter) camera in terms of detecting and tracking fine movements such as rapid hand movements and fingers. For example, the GS camera may have low image blur. The first camera (265a, 265b) may capture image data used for space recognition for 6DoF and SLAM function through depth shooting. Additionally, a user gesture recognition function can be performed based on image data captured by the first camera (265a, 265b).

The second camera (270a, 270b) is an ET (Eye Tracking) camera and can be used to capture image data for detecting and tracking the user's eyes. The second camera (270a, 270b) is described later in FIG. 3.

The third camera (245) may be a camera for taking pictures. The third camera (245) may include a high-resolution camera for capturing images of HR (High Resolution) or PV (Photo Video). The third camera (245) may include a color camera equipped with functions for obtaining high-quality images, such as an AF function and optical image stabilization (OIS). The third camera (245) may be a GS camera or an RS camera.

The fourth camera unit (e.g., the face recognition camera (425, 426) of FIG. 4 below) is a face recognition camera, and the FT (Face Tracking) camera can be used to detect and track a user's facial expression.

A depth sensor (not shown) may refer to a sensor that senses information for confirming the distance to an object, such as TOF (Time of Flight). TOF is a technology that measures the distance to an object using signals (e.g., near-infrared, ultrasound, laser, etc.). A depth sensor based on TOF technology can measure the flight time of a signal by emitting a signal from a transmitter and measuring the signal from a receiver.

The light sources (230a, 230b) (e.g., illumination modules) may include elements (e.g., light emitting diodes (LEDs)) that irradiate light of various wavelengths. The illumination modules may be attached to various locations depending on the intended use. As one use example, a first illumination module (e.g., LED element) attached around the frame of the augmented reality glasses device may emit light to assist in gaze detection when tracking eye movements with an ET camera. The first illumination module may exemplarily include an IR LED of an infrared wavelength. As another use example, a second illumination module (e.g., LED element) may be attached around a hinge (240a, 240b) connecting the frame and the temple, or adjacent to a camera mounted around a bridge connecting the frames. The second illumination module may emit light to supplement the ambient brightness when the camera is shooting. In cases where subject detection is not easy in a dark environment, a second lighting module can illuminate.

A substrate (235a, 235b) (e.g., a printed circuit board (PCB)) can support the components described above.

A printed circuit board (PCB) may be arranged on the temple portion of the glasses. The FPCB may transmit electrical signals to each module (e.g., a camera, a display, an audio module, a sensor module) and other printed circuit boards. According to one embodiment, at least one printed circuit board may be in the form of a first substrate, a second substrate, and an interposer arranged between the first substrate and the second substrate. Electrical signals may be transmitted to each module and other printed circuit boards.

Other components may include, for example, at least one of a plurality of microphones (e.g., a first microphone (250a), a second microphone (250b), a third microphone (250c)), a plurality of speakers (e.g., a first speaker (255a), a second speaker (255b)), a battery (260), an antenna, or a sensor (e.g., an acceleration sensor, a gyro sensor, a touch sensor, etc.).

FIG. 3 illustrates examples of optical systems for an eye tracking camera, a transparent member, and a display according to various embodiments.

FIG. 3 is a diagram for explaining the operation of an eye-tracking camera included in an electronic device according to one embodiment. Referring to FIG. 3, a process is illustrated in which an eye-tracking camera (310) (e.g., the first eye-tracking camera (270a) and the second eye-tracking camera (270b) of FIG. 2) of an electronic device (301) according to one embodiment tracks a user's eye (309), or in other words, the user's gaze, by using light (e.g., infrared light) output from a display (320) (e.g., the first display (205) and the second display (210) of FIG. 2).

The second camera (e.g., the second camera (270a, 270b) of FIG. 2) may be an eye tracking camera (310) that collects information for positioning the center of a virtual image projected onto the electronic device (301) according to the direction in which the pupil of the wearer of the electronic device (301) is looking. The second camera may also include a GS camera to detect the pupil and track rapid eye movement. ET cameras may also be installed for the left and right eyes, respectively, and each camera having the same performance and specifications may be used. The eye tracking camera (310) may include a gaze tracking sensor (315). The gaze tracking sensor (315) may be included inside the eye tracking camera (310). Infrared light output from the display (320) may be transmitted to the user's eyes (309) as infrared reflected light (303) by the half mirror. The gaze tracking sensor (315) can detect infrared reflected light (303) and infrared transmitted light (305) reflected from the user's eyes (309). The eye tracking camera (310) can track the user's eyes (309), or in other words, the user's gaze, based on the detection result of the gaze tracking sensor (315).

The display (320) may include a plurality of visible light pixels and a plurality of infrared pixels. The visible light pixels may include R, G, and B pixels. The visible light pixels may output visible light corresponding to a virtual object image. The infrared pixels may output infrared light. The display (320) may include, for example, micro light emitting diodes (LEDs) or organic light emitting diodes (OLEDs).

The display optical waveguide (350) and the eye-tracking camera optical waveguide (360) may be included inside a transparent member (370) (e.g., the first transparent member (225a) and the second transparent member (225b) of FIG. 2). The transparent member (370) may be formed of a glass plate, a plastic plate, or a polymer, and may be manufactured to be transparent or translucent. The transparent member (370) may be placed so as to face the user's eyes. At this time, the distance between the transparent member (370) and the user's eyes (309) may be called 'eye relief' (380).

The transparent member (370) may include optical waveguides (350, 360). The transparent member (370) may include an input optical member (330) and an output optical member (340). Additionally, the transparent member (370) may include an eye-tracking splitter (375) that separates input light into multiple waveguides.

According to one embodiment, light incident on one end of the display light pipe (350) may be propagated inside the display light pipe (350) by the nano-pattern and provided to the user. In addition, the display light pipe (350) configured as a free-form prism may provide image light to the user through the reflected mirror from the incident light. The display light pipe (350) may include at least one diffractive element (e.g., a Diffractive Optical Element (DOE), a Holographic Optical Element (HOE)) or at least one reflective element (e.g., a reflective mirror). The display light pipe (350) may guide display light (e.g., image light) emitted from a light source to the user's eyes by using at least one diffractive element or reflective element included in the display light pipe (350). For reference, also, in FIG. 3, the output optical member (340) is expressed as being separate from the eye-tracking optical waveguide (360), but the output optical member (340) may be included inside the eye-tracking optical waveguide (360).

According to various embodiments, the diffractive element may include an input optical member (330) and an output optical member (340). For example, the input optical member (330) may mean an input grating region. The output optical member (340) may mean an output grating region. The input grating region may serve as an input terminal that diffracts (or reflects) light output from (e.g., a Micro LED) to transmit the light to a transparent member (e.g., a first transparent member, a second transparent member) of a screen display unit. The output grating region may serve as an outlet that diffracts (or reflects) light transmitted to a transparent member (e.g., a first transparent member, a second transparent member) of a waveguide to a user's eye.

According to various embodiments, the reflective element may include a total internal reflection (TIR) optical element or waveguide for total internal reflection. For example, total internal reflection may mean a way of guiding light such that light (e.g., a virtual image) input through an input grating region is 100% reflected from one side (e.g., a specific side) of the waveguide at an angle of incidence such that it is 100% transmitted to the output grating region.

In one embodiment, light emitted from the display (320) may be guided along an optical path through an input optical member (330) into a waveguide. Light traveling inside the waveguide may be guided toward a user's eyes through an output optical member (340). The screen display may be determined based on the light emitted toward the eyes.

FIGS. 4A and 4B are diagrams showing examples of the front and back of an electronic device according to various embodiments. FIG. 4A may be an external appearance of the electronic device (401) as viewed from a first direction (①), and FIG. 4B may be an external appearance of the electronic device (401) as viewed from a second direction (②). When a user wears the electronic device (401), the external appearance that the user's eyes see may be FIG. 4B.

Referring to FIG. 4A, according to various embodiments, an electronic device (401) (e.g., the electronic device (101) of FIG. 1, the electronic device (201) of FIG. 2, and the electronic device (301) of FIG. 3) may provide a service that provides an extended reality (XR) experience to a user. For example, an XR or XR service may be defined as a service that collectively refers to virtual reality (VR), augmented reality (AR), and/or mixed reality (MR).

According to one embodiment, the electronic device (401) may mean a head-mounted device or a head-mounted display worn on a user's head, but may also be configured in the form of at least one of glasses, goggles, a helmet, or a hat. The electronic device (401) may include an OST (optical see-through) type configured to allow external light to reach the user's eyes through glasses when worn, or a VST (video see-through) type configured to block external light so that, when worn, light emitted from a display reaches the user's eyes but external light does not reach the user's eyes.

According to one embodiment, the electronic device (401) may be worn on the head of a user and may provide the user with an image related to an extended reality (XR) service. For example, the electronic device (401) may provide XR content (hereinafter, referred to as an XR content image) that outputs at least one virtual object to be overlapped in an area determined as a display area or a field of view (FoV) of the user. According to one embodiment, the XR content may mean an image or image related to a real space acquired through a camera (e.g., a camera for taking pictures) or an image or image in which at least one virtual object is added to a virtual space. According to one embodiment, the electronic device (401) may provide XR content based on a function being performed by the electronic device (401) and/or a function being performed by one or more external electronic devices (e.g., the electronic devices (102, 104) of FIG. 1, the server (108) of FIG. 1).

According to one embodiment, the electronic device (401) is at least partially controlled by an external electronic device (e.g., electronic devices (102 or 104) of FIG. 1), and may perform at least one function under the control of the external electronic device, but may also perform at least one function independently.

Referring to FIG. 4A, a vision sensor may be placed on a first surface of a housing of a main body (410) of an electronic device (401). The vision sensor may include cameras (e.g., cameras for second functions (411, 412), cameras for first functions (415)) and/or a depth sensor (417) for obtaining information related to the surrounding environment of the electronic device (401).

In one embodiment, the second function cameras (411, 412) can obtain images related to the surrounding environment of the electronic device (401). The first function cameras (415) can obtain images when the wearable electronic device is worn by the user. The first function cameras (415) can be used for hand detection and tracking, and recognition of user gestures (e.g., hand movements). The first function cameras (415) can be used for 3DoF, 6DoF head tracking, location (spatial, environmental) recognition, and/or movement recognition. In one embodiment, the second function cameras (411, 412) can also be used for hand detection and tracking, and user gestures.

In one embodiment, the depth sensor (417) may be configured to transmit a signal and receive a signal reflected from a subject, and may be used for purposes such as time of flight (TOF) to determine the distance to an object. Instead of or in addition to the depth sensor (417), cameras (411, 412, 415, 416) may determine the distance to an object.

Referring to FIG. 4b, a camera (425, 426) for facial recognition and/or a display (421) (and/or a lens) may be placed on the second surface (420) of the housing of the main body (410).

In one embodiment, a facial recognition camera (425, 426) adjacent to the display may be used to recognize the user's face, or may recognize and/or track the user's two eyes.

In one embodiment, the display (421) (and/or lens) may be disposed on the second side (420) of the electronic device (401). In one embodiment, the electronic device (401) may not include some of the plurality of cameras (415). Although not shown in FIGS. 4A and 4B , the electronic device (401) may further include at least one of the configurations illustrated in FIG. 2 .

According to one embodiment, the electronic device (401) may include a main body (410) that mounts at least some of the components of FIG. 1, a display (421) (e.g., a display module (160) of FIG. 1) disposed in a first direction (①) of the main body (410), a first function camera (e.g., a recognition camera) (415) disposed in a second direction (②) of the main body (410), a second function camera (e.g., a shooting camera) (411, 412) disposed in a second direction (②), a third function camera (e.g., a gaze tracking camera) (428) disposed in the first direction (①), a fourth function camera (e.g., a face recognition camera) (425, 426) disposed in the first direction (①), a depth sensor (417) disposed in the second direction (②), and a touch sensor (413) disposed in the second direction (②). Although not shown in the drawing, the main body (410) includes a memory (e.g., memory (130) of FIG. 1) and a processor (e.g., processor (120) of FIG. 1), and may further include other configurations shown in FIG. 1.

According to one embodiment, the display (421) may include a liquid crystal display (LCD), a digital mirror device (DMD), a liquid crystal on silicon (LCoS), an organic light emitting diode (OLED), or a micro light emitting diode (micro LED).

In one embodiment, when the display (421) is formed of one of a liquid crystal display (LCD), a digital mirror display (DMI), or a silicon liquid crystal display (SiLCD), the electronic device (401) may include a light source that irradiates light to a screen output area of the display (421). In another embodiment, when the display (421) can generate light on its own, for example, when the electronic device (401) is formed of one of an organic light emitting diode (OLED) or a micro LED, the electronic device (401) may provide a user with good quality XR content images even without including a separate light source. In one embodiment, if the display (421) is implemented with an organic light emitting diode (OLED) or a micro LED, a light source is unnecessary, and thus the electronic device (401) may be lightweight.

According to one embodiment, the display (421) may include a first transparent member (421a) and/or a second transparent member (421b). The user may use the electronic device (401) while wearing it on his or her face. The first transparent member (421a) and/or the second transparent member (421b) may be formed of a glass plate, a plastic plate, or a polymer, and may be manufactured to be transparent or translucent. According to one embodiment, the first transparent member (421a) may be arranged to face the user's left eye in the fourth direction (④), and the second transparent member (421b) may be arranged to face the user's right eye in the third direction (③). According to various embodiments, if the display (421) is transparent, it may be arranged at a position facing the user's eyes to configure a display area.

According to one embodiment, the display (421) may include a lens including a transparent waveguide. The lens may serve to adjust a focus so that a screen (e.g., an XR content image) output to the display (421) can be shown to the user's eyes. For example, light emitted from the display panel may pass through the lens and be transmitted to the user through a waveguide formed within the lens. The lens may be composed of a Fresnel lens, a Pancake lens, or a multi-channel lens.

An optical waveguide (e.g., a waveguide) may serve to transmit light generated from the display (421) to the user's eyes. The optical waveguide may be made of glass, plastic, or polymer, and may include a nano-pattern formed on a portion of an inner or outer surface, for example, a grating structure having a polygonal or curved shape. According to one embodiment, light incident on one end of the optical waveguide, that is, an output image of the display (421), may be propagated inside the optical waveguide and provided to the user. In addition, the optical waveguide composed of a free-form prism may provide the incident light to the user through a reflective mirror. The optical waveguide may include at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) or at least one reflective element (e.g., a reflective mirror). The optical waveguide may guide an image output from the display (421) to the user's eyes by using at least one diffractive element or reflective element included in the optical waveguide.

According to one embodiment, the diffractive element may include an input optical member/output optical member (not shown). For example, the input optical member may mean an input grating region, and the output optical member (not shown) may mean an output grating region. The input grating region may serve as an input terminal that diffracts (or reflects) light output from a light source (e.g., a Micro LED) to transmit the light to a transparent member (e.g., a first transparent member (421a), a second transparent member (421b)) of the display area. The output grating region may serve as an outlet that diffracts (or reflects) light transmitted to a transparent member (e.g., a first transparent member, a second transparent member) of the optical waveguide to a user's eye.

In some embodiments, the reflective element may include a total internal reflection (TIR) optical element or waveguide for total internal reflection. For example, total internal reflection may mean a way of directing light such that light (e.g., a virtual image) entering through an input grating region is substantially 100% reflected from one side (e.g., a specific side) of the optical waveguide, thereby causing substantially 100% transmission to the output grating region.

In one embodiment, light emitted from the display (421) can be guided along an optical path through an input optical member to a waveguide. Light traveling inside the optical waveguide can be guided toward a user's eye through an output optical member. The display area can be determined based on the light emitted toward the eye.

According to one embodiment, the electronic device (401) may include a plurality of cameras. For example, the cameras may include a first function camera (e.g., a recognition camera) (415) disposed in the second direction (②) of the main body (410), a second function camera (e.g., a shooting camera) (411, 412) disposed in the second direction (②), a third function camera (e.g., a gaze tracking camera) (428) disposed in the first direction (①), and/or a fourth function camera (e.g., a face recognition camera) (425, 426) disposed in the first direction (①), but may further include cameras having other functions not shown.

The first function camera (e.g., recognition camera) (415) can be used for the purpose of detecting user movement or recognizing user gestures. The first function camera (415) can support at least one of head tracking, hand detection and hand tracking, and space recognition. For example, the first function camera (415) mainly uses a GS (global shutter) camera with superior performance compared to an RS (rolling shutter) camera to detect hand movements and fine movements of fingers and track movements, and can be configured as a stereo camera including two or more GS cameras for head tracking and space recognition. The first function camera (415) can perform a SLAM (simultaneous localization and mapping) function to recognize information (e.g., location and/or direction) related to the surrounding space through space recognition for 6DoF and depth shooting.

The second function camera (e.g., a shooting camera) (411, 412) can be used to capture the outside and generate an image or video corresponding to the outside and transmit it to a processor (e.g., the processor (120) of FIG. 1). The processor can display the image provided from the second function camera (411, 412) on the display (421). The second function camera (411, 412) may be referred to as HR (high resolution) or PV (photo video) and may include a high-resolution camera. For example, the second function camera (411, 412) may include a color camera equipped with functions for obtaining high-quality images, such as an AF (auto focus) function and an optical image stabilizer (OIS), but is not limited thereto, and the second function camera (411, 412) may also include a GS camera or an RS camera.

A third function camera (e.g., a camera for eye tracking) (428) may be placed on the display (421) (or inside the main body) so that the camera lens faces the user's eyes when the user wears the electronic device (401). The third function camera (428) may be used for the purpose of detecting and tracking (ET: eye tracking) the pupil. The processor may track the movements of the user's left and right eyes in the images received from the third function camera (428) to determine the gaze direction. By tracking the position of the pupil in the images, the processor may ensure that the center of the XR content image displayed in the display area is positioned according to the direction in which the pupil is looking. As an example, a GS camera may be used as the third function camera (428) to detect the pupil and track the movement of the pupil. The third function cameras (428) may be installed respectively for the left and right eyes, and each camera having the same performance and specifications may be used.

A fourth functional camera (e.g., a camera for facial recognition) (425, 426) may be used to detect and track (FT: face tracking) a user's facial expression when the user wears an electronic device (401).

According to one embodiment, the electronic device (401) may include a lighting unit (e.g., LED) (not shown) as an auxiliary means for the cameras. As an example, the third function camera (425) may use lighting included in the display so that the emitted light (e.g., IR LED of infrared wavelength) is directed toward the user's both eyes as an auxiliary means for facilitating gaze detection when tracking eye movements. As another example, the second function camera (411, 412) may further include a lighting unit (e.g., flash) as an auxiliary means for supplementing the surrounding brightness when shooting outside.

According to one embodiment, the depth sensor (or depth camera) (417) can be used for the purpose of checking the distance to an object (e.g., an object), such as time of flight (TOF). TOF (time of flight) is a technology that measures the distance to an object using a signal (e.g., near-infrared, ultrasound, or laser). After a signal is transmitted from a transmitter, a receiver measures the signal, and the distance to an object can be measured based on the flight time of the signal.

According to one embodiment, the touch sensor (413) may be arranged in the second direction (②) of the main body (410). For example, when a user wears the electronic device (401), the user's eyes may look at the first direction (①) of the main body. The touch sensor (413) may be implemented as a single type or a left/right separated type depending on the shape of the main body (410), but is not limited thereto. For example, when the touch sensor (413) is implemented as a left/right separated type as illustrated in FIG. 4A, when a user wears the electronic device (401), the first touch sensor (413a) may be arranged at the user's left eye position, such as in the fourth direction (④), and the second touch sensor (413b) may be arranged at the user's right eye position, such as in the third direction (③).

The touch sensor (413) can recognize a touch input in at least one of, for example, a capacitive, pressure-sensitive, infrared, or ultrasonic manner. For example, the capacitive touch sensor (413) can recognize a physical touch (or contact) input or a hovering input (or proximity) of an external object. According to some embodiments, the electronic device (401) may utilize a proximity sensor (not shown) to recognize proximity of an external object.

According to one embodiment, the touch sensor (413) has a two-dimensional surface and can transmit touch data (e.g., touch coordinates) of an external object (e.g., a user's finger) that comes into contact with the touch sensor (413) to a processor (e.g., the processor (120) of FIG. 1). The touch sensor (413) can detect a hovering input for an external object (e.g., a user's finger) that approaches within a first distance from the touch sensor (413), or detect a touch input that touches the touch sensor (413).

According to one embodiment, the touch sensor (413) may provide two-dimensional information about the point of contact as "touch data" to the processor (120) when an external object touches the touch sensor (413). The touch data may be described as a "touch mode." The touch sensor (413) may provide hovering data about the point of time or location when an external object hovers around the touch sensor (413) when the external object is located within a first distance (or in proximity, hovering above the touch sensor) from the touch sensor (413), to the processor (120). The hovering data may be described as a "hovering mode/proximity mode."

According to one embodiment, the electronic device (401) may obtain hovering data using at least one of a touch sensor (413), a proximity sensor (not shown), and/or a depth sensor (417) to generate information about a distance, location, or time point between the touch sensor (413) and an external object.

According to one embodiment, the interior of the main body (410) may include a processor (e.g., processor (120) of FIG. 1) and a memory (e.g., memory (130) of FIG. 1).

The memory can store various instructions that can be performed by the processor. The instructions can include arithmetic and logical operations, data movement, or control commands such as input/output that can be recognized by the processor. The memory can temporarily or permanently store various data, including volatile memory (e.g., volatile memory (132) of FIG. 1) and nonvolatile memory (e.g., nonvolatile memory (134) of FIG. 1).

The processor may be a configuration that is operatively, functionally, and/or electrically connected to each component of the electronic device (401) and can perform calculations or data processing related to control and/or communication of each component. Operations performed by the processor may be stored in a memory and, when executed, executed by instructions that cause the processor to operate.

Hereinafter, there will be no limitation to the computational and data processing functions that the processor can implement on the electronic device (401), but a series of operations related to the XR content service function will be described. The operations of the processor described below can be performed by executing commands stored in the memory.

According to one embodiment, the processor may generate a virtual object based on virtual information based on image information. The processor may output a virtual object related to an XR service together with background space information through the display (421). For example, the processor may capture an image related to a real space corresponding to a field of view of a user wearing the electronic device (401) through a second function camera (411, 412) to obtain image information, or may generate a virtual space for a virtual environment. For example, the processor may control to display XR content (hereinafter, referred to as an XR content screen) on the display (421) such that at least one virtual object is overlapped in an area determined as a field of view (FoV) of the user.

According to one embodiment, the electronic device (401) may have a form factor for being worn on a user's head. The electronic device (401) may further include a strap for being secured on a body part of the user, and/or a wearing member. The electronic device (401) may provide a user experience based on augmented reality, virtual reality, and/or mixed reality while being worn on the user's head.

FIG. 5 illustrates examples of construction of a virtual space, input from a user within the virtual space, and output to the user according to various embodiments.

An electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (201) of FIG. 2, the electronic device (301) of FIG. 3, and the electronic device (401) of FIG. 4) can obtain spatial information about a physical space in which the sensor is located by using a sensor. The spatial information can include a geographical location of the physical space in which the sensor is located, a size of the space, an appearance of the space, a location of a physical object (551) arranged in the space, a size of the physical object (551), an appearance of the physical object (551), and illuminant information. The appearance of the space and the physical object (551) can include at least one of a shape, a texture, or a color of the space and the physical object (551). The illuminant information is information about a light source that emits light acting within the physical space, and can include at least one of the intensity, direction, or color of the illumination. The aforementioned sensor can collect information for providing augmented reality. For example, referring to the augmented reality device illustrated in FIGS. 2 to 4, the sensor may include a camera and a depth sensor. However, the present invention is not limited thereto, and the sensor may further include at least one of an infrared sensor, a depth sensor (e.g., a lidar sensor, a radar sensor, or a stereo camera), a gyro sensor, an acceleration sensor, or a geomagnetic sensor.

The electronic device (501) can collect spatial information over multiple time frames. For example, in each time frame, the electronic device (501) can collect spatial information about a portion of a scene within a sensing range (e.g., a field of view (FOV)) of a sensor at a location of the electronic device (501) in physical space. By analyzing the spatial information of multiple time frames, the electronic device (501) can track changes in an object (e.g., movement of a position or change of a state) over time. The electronic device (501) can also obtain integrated spatial information (e.g., an image that spatially stitches scenes around the electronic device (501) in physical space) for the integrated sensing range of the multiple sensors by comprehensively analyzing the spatial information collected through the multiple sensors.

An electronic device (501) according to one embodiment can analyze a physical space into three-dimensional information by utilizing various input signals of a sensor (e.g., sensing data of an RGB camera, an infrared sensor, a depth sensor, or a stereo camera). For example, the electronic device (501) can analyze at least one of a shape, a size, and a position of a physical space, and a shape, a size, or a position of a physical object (551).

For example, the electronic device (501) can detect an object captured in a scene corresponding to the field of view of the camera by using sensing data of the camera (e.g., a captured image). The electronic device (501) can determine a label of a physical object (551) (e.g., information indicating a classification of an object, including a value indicating a chair, a monitor, or a plant) and an area (e.g., a bounding box) occupied by the physical object (551) in the two-dimensional scene from the two-dimensional scene image of the camera. Accordingly, the electronic device (501) can obtain two-dimensional scene information at a position viewed by the user (590). In addition, the electronic device (501) can also calculate a location of the electronic device (501) in a physical space based on the sensing data of the camera.

The electronic device (501) can obtain the location information of the user (590) and the depth information of the actual space in the direction of viewing by using the sensing data (e.g., depth data) of the depth sensor. The depth information is information indicating the distance from the depth sensor to each point, and can be expressed in the shape of a depth map. The electronic device (501) can analyze the distance of each pixel in the three-dimensional position viewed by the user (590).

The electronic device (501) can obtain information including a 3D point cloud and mesh by using various sensing data. The electronic device (501) can analyze a physical space to obtain a surface, mesh, or 3D coordinate point cluster that constitutes the space. The electronic device (501) can obtain a 3D point cloud representing physical objects based on the information obtained as described above.

The electronic device (501) can analyze a physical space to obtain information including at least one of a three-dimensional position coordinate, a three-dimensional shape, or a three-dimensional size (e.g., a three-dimensional bounding box) of physical objects placed within the physical space.

Accordingly, the electronic device (501) can obtain information on a physical object detected in a three-dimensional space and semantic segmentation information on the three-dimensional space. The physical object information can include at least one of a position, an appearance (e.g., shape, texture, and color), or a size of a physical object (551) in a three-dimensional space. The semantic segmentation information is information that semantically divides a three-dimensional space into subspaces, and can include, for example, information indicating that the three-dimensional space is divided into an object and a background, and information indicating that the background is divided into a wall, a floor, and a ceiling. The electronic device (501) can obtain and store three-dimensional information (e.g., spatial information) on the physical object (551) and the physical space as described above. The electronic device (501) can store three-dimensional position information of the user (590) in the space together with the spatial information.

An electronic device (501) according to one embodiment can construct a virtual space (500) based on a physical location of the electronic device (501) and/or a user (590). The electronic device (501) can generate the virtual space (500) by referring to the spatial information described above. The electronic device (501) can generate a virtual space (500) of the same scale as a physical space based on the spatial information, and can place objects in the generated virtual space (500). The electronic device (501) can provide a complete virtual reality to the user (590) by outputting an image that replaces the entire physical space. The electronic device (501) can provide mixed reality (MR) or augmented reality (AR) by outputting an image that replaces a part of the physical space. However, although the construction of a virtual space (500) based on spatial information acquired through analysis of the aforementioned physical space is described, the electronic device (501) may also construct a virtual space (500) regardless of the physical location of the user (590). In this specification, the virtual space (500) may represent a space corresponding to augmented reality or virtual reality.

For example, the electronic device (501) can provide a virtual graphic representation that replaces at least a portion of a physical space. The electronic device (501) based on optical see-through can output a virtual graphic representation by overlaying the virtual graphic representation on a screen area corresponding to at least a portion of a space on a screen display unit. The electronic device (501) based on video see-through can output an image generated by replacing an image area corresponding to at least a portion of a space among a spatial image corresponding to a physical space rendered based on spatial information with a virtual graphic representation. The electronic device (501) can replace at least a portion of a background in a physical space with a virtual graphic representation, but is not limited thereto. The electronic device (501) can also perform only additional placement of a virtual object (552) within a virtual space (500) based on spatial information without changing the background.

The electronic device (501) can place and output a virtual object (552) in a virtual space (500). The electronic device (501) can set a manipulation area of the virtual object (552) in a space occupied by the virtual object (552) (e.g., a volume corresponding to the appearance of the virtual object (552)). The manipulation area can represent an area where manipulation of the virtual object (552) occurs. In addition, the electronic device (501) can output a physical object (551) by replacing it with the virtual object (552). The virtual object (552) corresponding to the physical object (551) can have a shape identical to or similar to that of the physical object (551). However, the present invention is not limited thereto, and the electronic device (501) can also set only a manipulation area in a space occupied by the physical object (551) or in a location corresponding to the physical object (551) without outputting a virtual object (552) that replaces the physical object (551). In other words, the electronic device (501) can transmit visual information representing the physical object (551) (e.g., light reflected from the physical object (551) or an image captured of the physical object (551)) to the user (590) without change, and set a manipulation area for the physical object (551). The manipulation area can be set to a shape and volume similar to the space occupied by the virtual object (552) or the physical object (551), but is not limited thereto. The electronic device (501) can also set a manipulation area smaller than the space occupied by the virtual object (552) or the space occupied by the physical object (551).

According to one embodiment, the electronic device (501) may place a virtual object (e.g., an avatar object) representing a user (590) within the virtual space (500). When the avatar object is provided in a first-person view, the electronic device (501) may visualize a graphical representation corresponding to a part (e.g., a hand, a torso, or a leg) of the avatar object to the user (590) through the aforementioned display (e.g., an optical see-through display or a video see-through display). However, the present invention is not limited thereto, and when the avatar object is provided in a third-person view, the electronic device (501) may also visualize a graphical representation corresponding to the entire shape (e.g., a back view) of the avatar object to the user (590) through the aforementioned display. The electronic device (501) may provide the user (590) with an experience integrated with the avatar object.

In addition, the electronic device (501) can provide an avatar object of another user who has entered the same virtual space (500). The electronic device (501) can receive feedback information that is the same as or similar to feedback information (e.g., information based on at least one of visual, auditory, or tactile senses) provided to another electronic device (501) who has entered the same virtual space (500). For example, when an object is placed in a certain virtual space (500) and multiple users access the virtual space (500), the electronic devices (501) of the multiple users can receive feedback information (e.g., graphical representation, sound signal, or haptic feedback) of the same object placed in the virtual space (500) and provide the feedback information to each user (590).

The electronic device (501) can detect an input to an avatar object of another electronic device (501), and can also receive feedback information from an avatar object of another electronic device (501). The exchange of input and feedback for each virtual space (500) can be performed by a server (e.g., a server (108) of FIG. 1). For example, a server (e.g., a server providing a metaverse space) can transmit input and feedback between an avatar object of a user (590) and an avatar object of another user between the users (590). However, the present invention is not limited thereto, and the electronic device (501) can establish direct communication with another electronic device (501) without going through a server to provide an input based on an avatar object or receive feedback.

For example, the electronic device (501) may determine that a physical object (551) corresponding to the selected manipulation area has been selected by the user (590) based on detecting a user's input for selecting an manipulation area. The user's (590) input may include at least one of a gesture input using a part of the body (e.g., a hand, an eye), an input using a separate virtual reality accessory device, or a user's voice input.

The gesture input is an input corresponding to a gesture identified based on tracking of a body part (510) of a user (590), and for example, the gesture input may include an input for pointing or selecting an object. The gesture input may include at least one of a gesture in which a body part (e.g., a hand) faces an object for a predetermined period of time or longer, a gesture in which a body part (e.g., a finger, an eye, a head) points to an object, or a gesture in which a body part and an object make spatial contact. A gesture in which an eye points to an object can be identified based on eye tracking. A gesture in which the head points to an object can be identified based on head tracking.

Tracking of a body part (510) of a user (590) may be performed primarily based on a camera of the electronic device (501), but is not limited thereto. The electronic device (501) may also track the body part (510) based on the cooperation of sensing data of a vision sensor (e.g., image data of a camera and depth data of a depth sensor) and information collected by an accessory device described below (e.g., controller tracking, finger tracking within the controller). Finger tracking may be performed by sensing the distance or contact between an individual finger and the controller based on a sensor built into the controller (e.g., an infrared sensor).

The virtual reality accessory device may include a ride-on device, a wearable device, a controller device (520), or other sensor-based devices. The ride-on device is a device that a user (590) rides on and operates, and may include, for example, at least one of a treadmill-type device or a chair-type device. The wearable device is a manipulation device that is worn on at least a part of the body of the user (590), and may include, for example, at least one of a full-body and half-body suit-type controller, a vest-type controller, a shoe-type controller, a bag-type controller, a glove-type controller (e.g., a haptic glove), or a face mask-type controller. The controller device (520) may include, for example, an input device (e.g., a stick-type controller or a gun) that is operated by a hand, a foot, a toe, or other body part (510).

The electronic device (501) may establish direct communication with the accessory device to track at least one of the location or motion of the accessory device, but is not limited thereto. The electronic device (501) may also communicate with the accessory device via a base station for virtual reality.

For example, the electronic device (501) may determine that the virtual object (552) has been selected based on detecting an action of gazing at the virtual object (552) for a predetermined period of time or longer through the aforementioned Eye Gaze Tracking technology. As another example, the electronic device (501) may recognize a gesture indicating the virtual object (552) through the hand tracking technology. The electronic device (501) may determine that the virtual object (552) has been selected based on whether the direction in which the tracked hand points indicates the virtual object (552) for a predetermined period of time or longer, or whether the hand of the user (590) contacts or enters an area occupied by the virtual object (552) within the virtual space (500).

The user's voice input is an input corresponding to the user's voice acquired by the electronic device (501), and may include, for example, voice data sensed by an input module (e.g., a microphone) of the electronic device (501) or received from an external electronic device of the electronic device (501). The electronic device (501) may determine that a physical object (551) or a virtual object (552) has been selected by analyzing the user's voice input. For example, the electronic device (501) may determine that at least one of the physical object (551) or the virtual object (552) corresponding to the detected keyword has been selected based on detecting a keyword indicating at least one of the physical object (551) or the virtual object (552) from the user's voice input.

The electronic device (501) may provide feedback, as described below, in response to the user (590) input described above.

The feedback may include visual feedback, auditory feedback, tactile feedback, olfactory feedback, or gustatory feedback. The feedback may be rendered by the server (108), the electronic device (101), or the external electronic device (102), as described above in FIG. 1.

Visual feedback may include outputting an image through a display (e.g., a transparent display or an opaque display) of the electronic device (501).

Auditory feedback may include outputting sound through a speaker of the electronic device (501).

The haptic feedback may include force feedback that simulates weight, shape, texture, dimension, and dynamics. For example, the haptic glove may include haptic elements (e.g., electrical muscles) that can simulate touch by tensing and relaxing the body of the user (590). The haptic elements within the haptic glove may act as tendons. The haptic glove may provide haptic feedback to the entire hand of the user (590). The electronic device (501) may provide feedback indicating the shape, size, and stiffness of an object through the haptic glove. For example, the haptic glove may generate forces that mimic the shape, size, and stiffness of the object. The exoskeleton of the haptic glove (or suit-like device) may include sensors and finger motion measurement devices, and may transmit tactile information to the body by transmitting forces (e.g., forces based on electromagnetic, DC motors, or pneumatics) that pull cables on the fingers of the user (590). Hardware providing tactile feedback may include sensors, actuators, power sources, and wireless transmission circuitry. The haptic glove may operate by inflating and deflating inflatable air bladders on the surface of the glove.

The electronic device (501) may provide feedback to the user (590) based on the selection of an object within the virtual space (500). For example, the electronic device (501) may output a graphical representation indicating the selected object (e.g., a representation highlighting the selected object) through the display. For another example, the electronic device (501) may output a sound (e.g., a voice) guiding the selected object through the speaker. For another example, the electronic device (501) may provide the user (590) with a haptic movement simulating a sense of touch for the corresponding object by transmitting an electrical signal to a haptic-enabled accessory device (e.g., a haptic glove).

FIG. 6 is a diagram illustrating an example of an operation of an electronic device providing space to a user according to various embodiments.

An electronic device according to one embodiment (e.g., an electronic device (101) of FIG. 1, an electronic device (201) of FIG. 2, an electronic device (301) of FIG. 3, an electronic device (401) of FIG. 4, an electronic device (501) of FIG. 5) may display an image that renders objects (621, 622, 623, 624, 625, 626) arranged in a space. The space according to one embodiment may include at least one of a physical space and a virtual space.

For example, the space may include a virtual space constructed based on a physical space. For example, an electronic device may obtain spatial information about a physical space around the electronic device, and construct and provide a virtual space based on properties (e.g., scale) of the obtained physical space. In various embodiments of the present disclosure, a space including a virtual space constructed based on a physical space around the electronic device may also be expressed as a VST space (video see-through space).

For example, the space may include a virtual space constructed independently of the physical space around the electronic device. For example, the space may be a virtual space constructed based on a physical space different from the physical space around the electronic device. For example, the space may be a virtual space constructed by a server (e.g., server (108) of FIG. 1). In various embodiments of the present disclosure, a space including a virtual space constructed independently of the physical space around the electronic device may also be expressed as a VR space (virtual reality space).

For example, the space may include both a physical space around the electronic device and a virtual space in which virtual objects are arranged. The virtual space in which virtual objects are arranged may be constructed based on the physical space around the electronic device. In other words, the space may include a space in which the physical space around the electronic device and the virtual space in which virtual objects are arranged are combined. For example, when the electronic device is an optical see-through (OST)-based electronic device (e.g., the electronic device (201) of FIG. 2), at least a part of the physical space around the electronic device may be directly recognized by the user (610) through a transparent member, and at least a part of the virtual space may be provided to the user (610) by displaying elements of the virtual space (e.g., visual effects, virtual objects) on the display. In other words, light from the outside (e.g., a background of a physical space around the electronic device or a physical object) reaches the user's (610) eyes through the transparent member, so that the user (610) can recognize the physical space around the user (610), and elements of the virtual space are displayed through the display, so that the user (610) can recognize the virtual space overlaid on the physical space around the user (610). For example, the electronic device can provide the user (610) with a space in which elements of the virtual space are additionally arranged with the physical space around the user (610) as the background. In various embodiments of the present disclosure, a space including the physical space around the electronic device and the virtual space can also be expressed as an optical see-through space (OST space) or an augmented reality space (AR space).

Objects placed in space may include physical objects and virtual objects.

For example, a virtual object may include a virtual object for executing an application (e.g., an icon of the application), a virtual object providing information (e.g., a virtual object displaying weather information, a virtual object displaying a memo), a virtual object including an execution screen of the application (e.g., a window displaying an execution screen of the application). A virtual object may include, for example, a widget, an icon, a visualization anchor, or an execution screen of the application (or a window displaying the execution screen).

For example, a virtual object may be related to a physical object. A virtual object based on a physical object may include a virtual object that replaces the physical object and/or a virtual object that is related to the physical object (e.g., a virtual controller for controlling the physical object).

Referring to FIG. 6, when a user (610) wearing the electronic device enters a space, the electronic device may display a screen (620) for the space. The electronic device may determine a subspace to be displayed to the user (610) from the space based on the line of sight of the user (610), and display the screen (620) based on rendering data that renders objects (621, 622, 623, 624, 625, 626) arranged in the subspace. In FIG. 6, the objects (621, 622, 623, 624, 625, 626) on the screen (620) are depicted as being arranged in a curved area, but are not limited thereto, and may be arranged on a surface other than the curved area.

According to one embodiment, the electronic device can obtain information about the viewpoint (e.g., the perspective from which the user is looking) of a user (610) wearing the electronic device. The electronic device can display a screen (620) of a space corresponding to the viewpoint based on the information about the viewpoint of the user (610).

In the screen (620) of FIG. 6, at least a part of the background of the space may be displayed in the remaining portion excluding the objects (621, 622, 623, 624, 625, 626). For example, if the space is a VST space, the remaining portion excluding the objects (621, 622, 623, 624, 625, 626) may display a virtual space in which the physical space around the electronic device is reconstructed. For example, if the space is an AR space, the remaining portion excluding the objects (621, 622, 623, 624, 625, 626) may show the physical space around the electronic device. For example, if the space is a VR space, the remaining portion excluding the objects (621, 622, 623, 624, 625, 626) may display a virtual space (e.g., a virtual space constructed independently from the surroundings of the electronic device).

FIG. 7 is a flowchart illustrating an example of a method for an electronic device to output content related to a target object according to various embodiments.

An electronic device according to one embodiment (e.g., an electronic device (101) of FIG. 1, an electronic device (201) of FIG. 2, an electronic device (301) of FIG. 3, an electronic device (401) of FIG. 4, an electronic device (501) of FIG. 5) may provide a user with a space (e.g., a VST space, an AR space) based on a physical space around the electronic device.

An electronic device may operate in one of a plurality of operation modes. For example, the plurality of operation modes of the electronic device may include a first operation mode (e.g., a basic operation mode) and a second operation mode (e.g., an idle operation mode). The electronic device may display and activate an interaction object while the operation mode of the electronic device is the first operation mode. The electronic device may deactivate the interaction object and output content for relieving user fatigue while the operation mode of the electronic device is the second operation mode (e.g., an idle operation mode). As described below, switching between the first operation mode and the second operation mode may be performed based on a condition.

In operation (710), the electronic device can obtain image information corresponding to a target object determined (e.g., selected, identified) from among physical objects arranged in a physical space around the electronic device. The electronic device can select the target object from among objects arranged in a space provided to the user.

According to one embodiment, the electronic device may obtain background information about an external environment (e.g., a physical space around the electronic device) sensed by a sensor (e.g., a camera sensor) of the electronic device. The electronic device may analyze the background information to select at least one object requiring movement as a target object. According to one embodiment, if the object requiring at least one movement is determined to be an object capable of being given life in a real environment, an object capable of movement, or an object capable of applying at least one interactive animation (or effect), the object may be selected as the target object.

According to one embodiment, the electronic device can obtain image information about a physical space around the electronic device. The electronic device can detect physical objects placed in the physical space. For example, the electronic device can calculate a score for generating content based on each physical object. The electronic device can determine a physical object with the highest score (or physical objects with the highest scores) among the physical objects as a target object.

In one embodiment, the space provided to the user may be a space based on the physical space around the electronic device. For example, the space may include a virtual space (e.g., VST space) constructed based on the physical space around the electronic device. For example, the space may include a space (e.g., AR space) that combines the physical space around the electronic device and the virtual space in which virtual objects are placed.

According to one embodiment, the electronic device can select a target object from among physical objects placed in a physical space around the electronic device (or the user). The target object is an object selected from among the physical objects, and may mean an object that is a target of content to be acquired (e.g., a video, an image, etc.). As described below, the electronic device can acquire a video of a target object with motion.

For example, an electronic device may select, as a target object, an object that can move depending on the surrounding environment among physical objects. For example, an electronic device may select, as a target object, an object that can shake when wind blows in the physical space around the electronic device (e.g., a plant pot, a flag, a pinwheel).

However, the electronic device according to various embodiments of the present disclosure is not limited to selecting a movable physical object as a target object. As described later in FIG. 10, the target object may be selected as a physical object that displays a scene in an internal area of the target object.

According to one embodiment, the image information corresponding to the target object may include an image capturing the target object. The electronic device may obtain an image capturing the selected target object. For example, the electronic device may obtain an image captured for the target object based on a sensor (e.g., a camera sensor, a depth sensor) mounted on the electronic device. For example, the electronic device may obtain an image capturing the target object by segmenting an area corresponding to the target object from an image of space (e.g., a stereoscopic image).

According to one embodiment, after the electronic device determines (e.g., selects) a target object, the electronic device can obtain image information corresponding to the target object. According to one embodiment, the electronic device can obtain image information about a physical space around the electronic device. The electronic device can select the target object by analyzing the image information about the physical space. The electronic device can divide (e.g., crop) a portion of an area corresponding to the target object from an image of the physical space.

In operation (720), the electronic device can obtain content related to the target object (e.g., a video of the target object moving) based on the obtained image information and the surrounding information of the electronic device.

According to one embodiment, the surrounding information of the electronic device may include environmental information about the physical space surrounding the electronic device. For example, the surrounding information of the electronic device may include weather information and/or time information about the physical space surrounding the electronic device.

Weather information may refer to information about the weather of a geographical area around the electronic device. For example, the weather information may include at least one of temperature information, rain information, wind information, fine dust information, or humidity information.

The time information may refer to information about the time of a geographical area around the electronic device. For example, the time information may include at least one of season information of a geographical area around the electronic device, date information, or standard time information of a geographical area around the electronic device.

According to one embodiment, the electronic device may obtain content related to the target object based on image information and surrounding information captured by the target object. For example, the content related to the target object may include a moving video of the target object and/or other images of the target object. For example, the content may obtain content (e.g., an image, a video) that can be displayed in an internal area of the target object.

In one embodiment, if the content includes a video, the video may include a plurality of frame images of the target object. The video may include a result of applying a movement based on an environment corresponding to the surrounding information to the target object shown in the image information. For example, if the target object is a plant pot and the surrounding information includes wind blowing in a geographical area around the electronic device, the video may include the plant pot shaking in the wind.

In one embodiment, if the content includes an image, the image may include a modification of at least a portion of a visual property of the target object. The visual property may include color, shape, size, and/or texture.

In various embodiments of the present disclosure, content acquired based on a target object is not limited to a single content. According to one embodiment, an electronic device can acquire a plurality of contents (or a plurality of contents related to the target object) based on a target object.

In one embodiment, the electronic device can obtain additional objects using a content creation model. The content creation model is described in more detail below in FIG. 11.

In operation (730), the electronic device may deactivate an operable interaction object in response to a user input based on satisfying a condition regarding the output of the content.

According to one embodiment, the condition regarding the output of the content may include a condition regarding a user input indicating that the user's intention is expected not to perform any operation on the electronic device for a certain period of time. According to one embodiment, the condition regarding the output of the content may mean a condition corresponding to a transition of the operation mode of the electronic device from a first operation mode (e.g., a basic operation mode) to a second operation mode (e.g., an idle operation mode).

For example, a condition regarding the output of the content may include at least one of: the user's gaze not pointing at the interaction object for a threshold period of time, the user's biometric information meeting a predetermined condition, or the user's input triggering an action of the electronic device not being detected from the user for a threshold period of time.

The fact that the user's gaze does not point to the interaction object for a threshold period of time may include that the interaction object selected based on the user's gaze does not exist for the threshold period of time. For example, an electronic device according to one embodiment may detect that the interaction object is pointed at by the user's gaze if the overlap between the pointer area based on the user's gaze and the area corresponding to the interaction object is maintained for a specific period of time or longer.

The user's biometric information may be determined based on sensing data acquired through a biometric sensor. For example, the biometric information may include heart rate information, blood pressure information, electrocardiogram information (ECG information), and/or stress information. For example, a condition (e.g., a condition corresponding to a transition from a basic operation mode to a rest operation mode) for each of the examples of the user's biometric information or a combination of two or more of the examples of the user's biometric information may be determined in advance. The electronic device may monitor at least one of the biometric information. The electronic device may change the operation mode of the electronic device from the first operation mode to the second operation mode based on at least one of the biometric information satisfying the predetermined condition.

User input that triggers an action of the electronic device may include, for example, user input to an interactive object that is operable in response to the user input. Interaction objects that are operable in response to the user input are described in more detail below.

However, in various embodiments of the present disclosure, the conditions for outputting content (or conditions corresponding to switching from the first operation mode to the second operation mode) are not limited to the examples described above. For example, if the electronic device is expected to not perform any operation on the electronic device for a certain period of time based on the result of tracking the user's gaze (e.g., if the user is expected not to focus on the provided space), the electronic device may change the operation mode of the electronic device from the first operation mode to the second operation mode.

For example, the electronic device may obtain a saliency map based on the result of tracking the user's gaze. The saliency map may indicate a point on which the user focuses among images corresponding to a space provided to the user. For example, the electronic device may change the operation mode of the electronic device from a first operation mode to a second operation mode based on the degree of matching between the saliency map obtained based on tracking the user's gaze and the interaction object (e.g., when the score indicating the degree of matching is less than a threshold score).

In one embodiment, a condition regarding the output of content (e.g., display of an image or playback of a video) may include obtaining an input corresponding to the output of the content (e.g., content output input, trigger input of an idle motion mode). For example, the user's input (e.g., trigger input of an idle motion mode) may include at least one of a button input, a motion input, a gesture input, or a voice input.

A button input may mean an input detected through a physical button included in an electronic device or an external device (e.g., an input module (150) of FIG. 1, or a controller device (520) of FIG. 5) that is connected to the electronic device through communication. For example, the electronic device or the external device may include a button assigned to a function of changing an operation mode of the electronic device to an idle operation mode, and may detect a trigger input of the idle operation mode when detecting that the button is pressed.

Motion input may refer to a motion applied to an electronic device or an external device (e.g., an input module (150) of FIG. 1, or a controller device (520) of FIG. 5) that is connected to the electronic device by communication. The motion input may include an input detected when a user rotates, tilts, and/or moves the electronic device or the external device. For example, a specific motion may be assigned to a function of changing an operation mode of the electronic device to an idle operation mode, and a trigger input of the idle operation mode may be detected when the electronic device or the external device detects that a specific motion has occurred. For example, the specific motion may be determined as a motion that moves along at least a portion of a determined movement trajectory (e.g., a circle, a closed curve, or a check mark (e.g., a V mark)).

A gesture input may refer to an input corresponding to a gesture identified based on tracking of a user's body part (e.g., a finger, an eye, or a head). The gesture input may include a gesture input corresponding to a gesture of pointing and/or dragging an object with a user's body part. For example, a specific gesture may be assigned to a function of changing an operation mode of an electronic device to an idle operation mode, and a trigger input of the idle operation mode may be detected when the electronic device or an external device detects that a specific gesture has occurred. For example, a specific gesture may be determined as a gesture of pointing an area corresponding to an interaction object with the user's eye and/or grabbing an area corresponding to an interaction object with the user's hand.

A voice input may refer to an input corresponding to a user's voice obtained through an electronic device or an external device that is connected to the electronic device by communication. The electronic device may determine that the user's voice input requests changing the operation mode of the electronic device to the idle operation mode by analyzing the voice input. If the user's intention determined based on the user's voice input is to change the operation mode of the electronic device to the idle operation mode, the electronic device may obtain the voice input as a trigger input of the idle operation mode.

According to one embodiment, the electronic device can obtain voice data corresponding to the user's voice. The electronic device can analyze the voice data using a natural language platform. For example, the natural language platform can include an automatic speech recognition module (ASR module) and/or a natural language understanding module (NLU module).

For example, an automatic speech recognition module can convert a user's speech input into text data. A natural language understanding module can use text data of a speech input to determine a user's intent. For example, a natural language understanding module can perform syntactic analysis or semantic analysis to determine a user's intent. In one embodiment, a natural language understanding module can use linguistic features (e.g., grammatical elements) of morphemes or phrases to determine the meaning of words extracted from a speech input, and can match the meaning of the determined words to the intent to determine the user's intent.

The user input (e.g., trigger input in the idle motion mode) is not limited to consisting of one of button input, motion input, gesture input, or voice input, and may be a user input that combines two or more of button input, motion input, gesture input, voice input, or bio-signal input. The user input that combines two or more of button input, motion input, gesture input, voice input, or bio-signal input may also be expressed as 'multimodal input'.

The electronic device may deactivate an interaction object operable in response to a user's input if the condition regarding the output of the content (e.g., display of an image, playback of a video) is satisfied. The interaction object operable in response to a user's input may mean an interaction object to which a specific operation of the electronic device is assigned. In various embodiments of the present disclosure, the interaction object operable in response to a user's input may also be expressed as an interaction object.

For example, an interaction object may include an icon object corresponding to an application. An execution operation of a corresponding application and/or a display operation of an execution screen of the application may be assigned to the icon object. For example, an interaction object may include a widget object. An update operation of corresponding information and/or a display operation of an execution screen of the corresponding application may be assigned to the widget object.

For example, an interaction object may include a button object and/or a controller object including two or more button objects. A button object (or controller object) may be assigned a specific action to the button object (or controller object). The electronic device may perform the action assigned to the button object (or controller object) based on a user input obtained through the button object (or controller object). Examples of interaction objects are described in more detail below in FIG. 8.

Disabling an interaction object may include at least partially limiting display of the interaction object. For example, the electronic device may include not displaying the interaction object. However, disabling an interaction object is not limited thereto.

For example, disabling an interaction object may include displaying the interaction object with predetermined visual properties (e.g., transparency greater than a threshold transparency, brightness less than a threshold brightness). Disabling an interaction object may include ignoring user input detected via the interaction object. For example, an electronic device may restrict the electronic device from performing an operation assigned to the interaction object even if it detects user input selecting the interaction object.

In various embodiments of the present disclosure, deactivation of an interaction object may mean deactivation of a visual property of the interaction object, and may be independent of cessation of an operation related to the interaction object (e.g., execution of an application corresponding to the interaction object). For example, the interaction object may correspond to a controller for playing music. Based on the electronic device satisfying a condition regarding output of the content, the interaction object may be deactivated (e.g., display may be restricted) and the music may be continued to play.

In various embodiments of the present disclosure, the electronic device is not limited to deactivating all interaction objects when the condition regarding the output of the content is satisfied. According to one embodiment, the electronic device can determine the deactivation of the interaction object based on the position where the interaction object is aligned. The interaction object can be positioned within a space including a background and a virtual object based on a physical space around the electronic device. For example, the interaction object can be positioned at a position aligned according to a background based on a physical space around the electronic device, or at a position aligned according to the electronic device within the space.

The interaction object may be provided in a form attached to a specific location in the physical space around the electronic device, when the interaction object is positioned in a position aligned with the background. The electronic device may maintain the display of the interaction object based on the interaction object being positioned in a position aligned with the background.

When an interaction object is arranged at a position aligned with an electronic device within a space (e.g., a pose of the electronic device, a position of the electronic device, a heading direction of the electronic device), the interaction object can be provided while maintaining a relative arrangement with respect to the electronic device, independent of the movement of the electronic device within the space. For example, when the interaction object is arranged at a position aligned with the electronic device within the space, the interaction object can be provided in the form of a floating object whose display is maintained at a specific position within a user's field of vision. The electronic device can restrict the display of the interaction object based on the fact that the interaction object is arranged at a position aligned with the electronic device within the space.

According to one embodiment, an interaction object arranged at a position aligned with a background may be perceived by the user as a virtual object coupled (e.g., attached) to a physical space, and thus may cause relatively less fatigue to the user than an interaction object arranged at a position aligned with an electronic device. The electronic device may effectively alleviate fatigue that may be caused to the user by minimizing differences between screens provided in operation modes by maintaining the display of the interaction object arranged at a position aligned with a background and limiting the display of the interaction object arranged at a position aligned with the electronic device.

In operation (740), the electronic device can output content related to the target object (e.g., playing a video, displaying an image) in an area corresponding to the target object through a display.

In one embodiment, the electronic device can display content instead of the target object. For example, when the electronic device provides an AR space to the user, the electronic device can overlay content corresponding to the target object in an area corresponding to the target object. For example, when the electronic device provides a VST space to the user, the electronic device can replace an area corresponding to the target object in an image of the VST space with content of the target object.

According to one embodiment, the area corresponding to the target object can be determined based on the location of the target object. For example, the electronic device can determine a reference point of the target object (e.g., a center point of the target object, a feature point of the target object) and determine an area corresponding to the target object based on the reference point of the target object. For example, the electronic device can obtain an area corresponding to the target object using a content generation model (e.g., a content generation model (1120) of FIG. 11). For example, output data of the content generation model can include information about an area corresponding to the target object. The area corresponding to the target object can be determined based on information about the target object (e.g., image information corresponding to the target object).

According to one embodiment, the area corresponding to the target object may be determined as an area that the target object can influence. For example, the area corresponding to the target object may be determined based on a range of movement of the target object if the target object is a movable object. For example, the area corresponding to the target object may be determined based on an area where a change in the visual property is induced by the target object if the target object can cause a change in the visual property of an area other than the target object. For example, if the target object is a light bulb, an area where light emitted by the light bulb reaches may be determined as the area corresponding to the target object. If the target object is a fan, an area where airflow due to wind generated by the fan exists may be determined as the area corresponding to the target object.

For example, the area corresponding to the target object can be determined as the area where the target object is visible. The area where the target object is visible can mean an area where the target object is visible among the display areas of the display of the electronic device.

For example, the area corresponding to the target object can be determined as an area that includes the area in which the target object is visible and is larger than the area in which the target object is visible (e.g., an area covering the area in which the target object is visible).

For example, the area corresponding to the target object can be determined as a part of the area where the target object is visible.

For example, the area corresponding to the target object may be determined as an area including a portion of the area where the target object is visible and a portion of the area where the target object is not visible (e.g., an area partially overlapping the area where the target object is visible).

However, in various embodiments of the present disclosure, the electronic device is not limited to displaying content only in an area corresponding to the target object. According to one embodiment, the electronic device may display content in an area larger than the area corresponding to the target object. The electronic device displaying content in an area larger than the area corresponding to the target object will be described in more detail later in FIG. 12.

In one embodiment, the electronic device can stop outputting the content based on satisfying a condition relating to stopping outputting the content. The electronic device can activate an operable interaction object in response to a user's input.

The condition for stopping the output of the content may include a condition regarding a user input indicating that the user's intention is expected to perform an operation on the electronic device. In one embodiment, the condition for stopping the output of the content may mean a condition corresponding to a transition of the operation mode of the electronic device from a second operation mode (e.g., an idle operation mode) to a first operation mode (e.g., a basic operation mode).

A condition for stopping output of content (e.g., a condition for stopping display of an image, a condition for stopping playback of a video) may be determined based on detecting a user input corresponding to stopping output of content (e.g., a change from a second operation mode to a first operation mode). This may include acquiring a user input corresponding to stopping output of content (e.g., an input for stopping content output, an input for stopping an idle operation mode).

For example, a user's input (e.g., a pause input in idle motion mode) may include at least one of a button input, a motion input, a gesture input, or a voice input.

For example, the electronic device may include a button assigned to a function of changing the operating mode from an idle operating mode to a default operating mode, and when the button is detected to be pressed, an input of cessation of the idle operating mode may be detected.

For example, a particular motion may be assigned to a function that changes the operating mode of the electronic device from an idle operating mode to a default operating mode, and an interruption input of the idle operating mode may be detected when the particular motion is detected to have occurred in the electronic device or an external device.

For example, a particular gesture may be assigned to a function that changes the operating mode of the electronic device from an idle operating mode to a default operating mode, and an interruption input from the idle operating mode may be detected when the particular gesture is detected to have occurred on the electronic device or an external device.

For example, the electronic device may acquire the voice input as an interrupt input of the idle operation mode if the user's intention, as determined based on the user's voice input, is to change the operation mode of the electronic device from the idle operation mode to the default operation mode.

For example, a condition for stopping output of content may include a condition based on an object or sound detected in a physical space around the electronic device. By way of example, a condition for stopping output of content may include at least one of a physical object approaching a physical space within a threshold distance from the electronic device, or detecting a sound (e.g., a preset alarm) in the physical space around the electronic device.

For example, a condition for stopping output of content may include occurrence of a predetermined event. The predetermined event may include an event having a priority higher than a threshold priority. The priority of the event may be preset according to an attribute of the event and may be set by the user. For example, the predetermined event may include an incoming call event and/or an alarm event.

According to one embodiment, an electronic device can perform control of content and interaction objects based on switching between operational modes of the electronic device.

An electronic device can change an operation mode of the electronic device from a first operation mode to a second operation mode based on satisfying a condition corresponding to a transition from a first operation mode to a second operation mode when the operation mode of the electronic device is a first operation mode. The electronic device can perform output of content and deactivation of an interaction object while the operation mode of the electronic device is a second operation mode. The electronic device can change an operation mode of the electronic device from a second operation mode to a first operation mode based on satisfying a condition corresponding to a transition from a second operation mode to the first operation mode when the operation mode of the electronic device is a second operation mode. The electronic device can restrict outputting of content in an area corresponding to a target object and activate an interaction object while the operation mode of the electronic device is a first operation mode.

In various embodiments of the present disclosure, the electronic device mainly describes, but is not limited to, performing an operation of determining whether a condition regarding output of content is satisfied, an operation of deactivating an interaction object, and/or an operation of outputting content (e.g., operation (730), operation (740)) after an operation of determining a target object (e.g., operation (710)) and/or an operation of obtaining content (e.g., operation (720)). According to one embodiment, the electronic device may determine whether a condition regarding output of content is satisfied, and if the condition regarding output of content is satisfied, perform an operation of determining a target object (e.g., operation (710)) and/or an operation of obtaining content (e.g., operation (720)), and then perform a deactivation operation of the interaction object and/or an operation of outputting content (e.g., operation (730), operation (740)).

FIG. 8 illustrates an example of an operation of an electronic device providing a three-dimensional image of a space according to various embodiments.

An electronic device according to one embodiment (e.g., an electronic device (101) of FIG. 1, an electronic device (201) of FIG. 2, an electronic device (301) of FIG. 3, an electronic device (401) of FIG. 4, an electronic device (501) of FIG. 5) may provide a user with a space (801) based on a physical space around the electronic device (e.g., a VST space, an AR space).

In one embodiment, the electronic device may display a stereoscopic image of a space (801) including a background (810) and virtual objects based on a physical space around the electronic device.

The space (801) provided to the user may include a background (810) based on a physical space and virtual objects placed within the space (801). The background (810) based on a physical space may be a virtual space constructed based on a physical space or may mean at least a part of a physical space.

The interaction object may include at least one of a window object (811) including a web page screen, an icon object (812) (or widget object) corresponding to an application, or a window object (813) including an execution screen of an application that processes a document of a specific format (e.g., Word, PowerPoint, Excel).

The electronic device can display a stereoscopic image of a space (801) through at least one display. The stereoscopic image can include a pair of images, a left image corresponding to a left eye of the user and a right image corresponding to a right eye of the user. The pair of images included in the stereoscopic image can include a pair of images reflecting binocular disparity corresponding to depth information of each point (or object) of the image. In various embodiments of the present disclosure, the stereoscopic image can also be expressed as a three-dimensional image and/or a space image.

FIG. 9 is a drawing illustrating an example of an operation in which the operation mode of an electronic device is switched between a basic operation mode and a rest operation mode when a target object is a movable object according to various embodiments.

An electronic device according to one embodiment (e.g., an electronic device (101) of FIG. 1, an electronic device (201) of FIG. 2, an electronic device (301) of FIG. 3, an electronic device (401) of FIG. 4, an electronic device (501) of FIG. 5) may perform deactivation of an interaction object (912) and output of a content (913) of a target object (911) when an operation mode of the electronic device is switched from a basic operation mode to a rest operation mode. According to one embodiment, when the content (913) is a video, the electronic device may play the video.

In the basic operation mode (901), the electronic device can provide the user with a target object (911) placed in a physical space as is, and activate (e.g., display) an interaction object (912) placed in the space.

Providing a target object (911) placed in a physical space as it is to a user may mean limiting the electronic device from generating movement of the target object (911). If the target object (911) placed in a physical space is static, the user can recognize the target object (911) as static, and if the target object (911) placed in a physical space has a specific motion, the user can recognize the target object (911) having the specific motion as it is.

For example, if the electronic device is an OST-based electronic device (or if the space is an AR space), providing the user with a target object (911) placed in the physical space may mean that the target object (911) placed in the physical space is directly recognized by the user through a transparent member.

For example, if the electronic device is a VST-based electronic device (or if the space is a VST space), providing a user with a target object (911) placed in the physical space may mean displaying a real-time image of the target object (911) placed in the physical space.

An electronic device may select a movable object as a target object (911) among physical objects placed in a physical space around the electronic device. For example, in FIG. 9, a plant pot placed in a physical space may be selected as the target object (911).

The electronic device can change the operating mode of the electronic device from the basic operating mode (901) to the idle operating mode (902) based on satisfying a condition for outputting content (e.g., a condition corresponding to switching from the basic operating mode (901) to the idle operating mode (902).

In the idle operation mode (902), the electronic device can deactivate the interaction object (912) and output the content (913) of the target object (911) in the area corresponding to the target object (911). In FIG. 9, for example, the content (913) can include a video showing a plant in a pot swaying in the wind.

FIG. 10 is a drawing illustrating an example of an operation in which the operation mode of an electronic device is switched between a basic operation mode and a rest operation mode when displaying a scene in an internal area of a target object according to various embodiments.

An electronic device according to one embodiment (e.g., an electronic device (101) of FIG. 1, an electronic device (201) of FIG. 2, an electronic device (301) of FIG. 3, an electronic device (401) of FIG. 4, an electronic device (501) of FIG. 5) may perform deactivation of an interaction object and output of contents of a target object when an operation mode of the electronic device is switched from a basic operation mode to an idle operation mode.

According to one embodiment, the electronic device can select a target object as a physical object capable of displaying a scene in an interior area of the target object. By way of example, the target object can include at least one of a door, a picture frame, a poster, a physical window, a photograph, a picture, or another electronic device including a display area (e.g., a television, a tablet, a smartphone).

In FIG. 10, in the basic operation mode (1001), the electronic device may provide a user with a space including an interaction object (1012) and a target object, a poster (1011). For example, the poster (1011) may include a picture of a waterfall.

According to one embodiment, an electronic device may obtain image information corresponding to a target object, and obtain content including a video of at least a portion of the target object moving based on the image information and surrounding information.

An electronic device can obtain content for a scene included in a target object when the target object displays a scene in an internal area of the target object. The content for the scene can include a video and/or image in which motions of objects included in the scene and/or visual properties (e.g., color, texture, style) of objects included in the scene are changed while maintaining semantic information of the objects included in the scene.

For example, the electronic device can obtain image information including a scene based on displaying the scene in an internal area of a target object. The image information including the scene can include an image corresponding to a poster (1011). The electronic device can obtain surrounding information including at least one of weather information or time information about the surroundings of the electronic device. Based on the image information and the surrounding information, the electronic device can obtain content related to the scene with visual properties corresponding to the weather or time indicated by the surrounding information.

For example, if the surrounding information of the electronic device indicates that it is raining, the electronic device can obtain content about a waterfall in rainy weather. For example, if the surrounding information of the electronic device indicates that it is sunny, the electronic device can obtain content about a waterfall in sunny weather. For example, if the surrounding information of the electronic device indicates a time corresponding to sunset, the electronic device can obtain content about a waterfall at a time of sunset.

An electronic device according to one embodiment can obtain content for a scene of a target object using a content generation model. The content generation model is described in more detail later in FIG. 11.

The electronic device can change the operating mode of the electronic device from the basic operating mode (1001) to the idle operating mode (1002) based on satisfying a condition for outputting content (e.g., a condition corresponding to switching from the basic operating mode (1001) to the idle operating mode (1002)).

The electronic device can output content in an internal area of a target object. In the idle operation mode (1002), the electronic device can deactivate the interaction object (1012) and output content (1013) of the target object (1011) in an area corresponding to the target object (1011). In FIG. 10, for example, the content (1013) can include a video showing a waterfall falling.

According to one embodiment, when the electronic device determines that the target object is an object capable of displaying a scene in an internal area of the target object, the electronic device may output preceding content regarding variation of the target object, and then output content related to the target object. The preceding content may include a moving picture showing a change in the shape of the target object.

For example, if the target object is a physical window, the preceding content may include a video showing the state of the physical window changing from a closed state to an open state. The electronic device may output the preceding content showing the state of the physical window determined as the target object changing from a closed state to an open state, and then output content about the scene in an area corresponding to an interior area of the physical window in the open state.

For example, if the target object is a door, the preceding content may include a video showing the state of the door changing from a closed state to an open state. The electronic device may output the preceding content showing the state of the door determined as the target object changing from a closed state to an open state, and then output content about the scene in an area corresponding to the interior area of the door in the open state.

According to one embodiment, content may be generated based on context information of the user. The context information of the user may include information about a place (e.g., a travel destination), a landscape, an entity, and/or a person (e.g., an artist) registered by the user. For example, the context information of the user may be determined based on information appearing in images included in a user terminal (e.g., an electronic device). For example, the content may include a result of reconstructing a surrounding landscape (e.g., a mountain, a lake, or a golf course seen from the user's position) based on location information of the electronic device.

FIG. 11 is a diagram illustrating an example of an operation of an electronic device obtaining content based on a content creation model according to various embodiments.

An electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1, the electronic device (201) of FIG. 2, the electronic device (301) of FIG. 3, the electronic device (401) of FIG. 4, and the electronic device (501) of FIG. 5) may obtain content (1131) of a target object using a content generation model (1120). For example, the content (1131) of the target object may include a moving image of the target object moving and/or a moving image related to a scene included in an internal area of the target object. For example, the content (1131) of the target object may include an image in which at least a portion of the target object is deformed and/or an image in which at least a portion of a scene included in an internal area of the target object is deformed.

The content generation model (1120) may refer to a model that is generated and/or trained to output output data including information about the content (1131) of the target object by being applied to input data including information about the target object (e.g., image information about the target object (1111)) and information about the surrounding environment (e.g., surrounding information about the electronic device (1112)). The content generation model (1120) may include a model based on a machine learning model (e.g., a neural network, a large language model (LLM), a generative artificial intelligence).

The input data of the content creation model (1120) according to one embodiment is not limited to including image information (1111) about the target object and surrounding information (1112) about the electronic device. For example, the input data of the content creation model (1120) may further include additional information about the target object. The additional information about the target object may mean information different from an image about the target object. For example, the additional information about the target object may include at least one of a type, a class, or a state (or a mode) of the target object. The electronic device may obtain additional information about the target object by analyzing an image about the target object. When the target object includes another electronic device, the electronic device may receive information about the target object through communication from the other electronic device, which is the target object.

The content creation model (1120) may be stored within the electronic device, or may be stored in an external device (e.g., another electronic device, a server, a cloud) accessible to the electronic device. Content of a target object may be created using the content creation model (1120). For example, the electronic device may directly create content of a target object using the content creation model (1120) (e.g., on-device), or the electronic device may transmit information about the target object (e.g., image information about the target object (1111)) and information about the surrounding environment (e.g., surrounding information of the electronic device (1112)) to another electronic device storing the content creation model (1120) and receive information about the content (1131) of the generated target object (e.g., on-cloud).

According to various embodiments of the present disclosure, the content (1131) of a target object is mainly described as being generated by a generation model (e.g., a content generation model (1120)), but is not limited thereto. The generation model according to one embodiment may be used to output information about a movement (or motion) of a target object based on information about the target object (e.g., image information (1111) about the target object) and information about a surrounding environment (e.g., surrounding information of an electronic device). In various embodiments of the present disclosure, the generation model used to output information about the movement (or motion) of the target object may also be expressed as a motion generation model. The electronic device may obtain content of the target object (e.g., content (1131) of the target object) by applying the movement (or motion) of the target object to information about the target object (e.g., image information (1111) about the target object).

According to one embodiment, the electronic device may further acquire a sound (1132) corresponding to the content (1131) of the target object together with the content (1131). For example, the electronic device may acquire a sound related to the movement of the target object appearing in the content (e.g., a video) based on the acquired image information and the surrounding information of the electronic device. For example, the electronic device may acquire a sound including background music of the content (e.g., an image) based on the acquired image information and the surrounding information of the electronic device. The electronic device may play the acquired sound together with the output of the content. For example, the content generation model (1120) may be generated and/or trained to output output data including information about the content (1131) and the sound (1132) of the target object.

For example, if the target object is a plant pot and the content (1131) of the target object represents a plant in the pot swaying in the wind, the sound (1132) may include the sound of wind and/or the sound of plant leaves rustling in the wind.

For example, if the target object is a waterfall and the content (1131) of the target object represents a waterfall falling on a rainy day, the sound (1132) may include the sound of rain and/or the sound of water falling from a waterfall.

The electronic device can output the content (1131) of the target object and play the sound (1132) when a condition for outputting the content is satisfied (e.g., when the operation mode of the electronic device is changed from a first operation mode to a second operation mode).

FIG. 12 is a drawing illustrating an example of an operation of an electronic device according to various embodiments to output content in an area larger than an area corresponding to a target object.

An electronic device according to one embodiment (e.g., an electronic device (101) of FIG. 1, an electronic device (201) of FIG. 2, an electronic device (301) of FIG. 3, an electronic device (401) of FIG. 4, an electronic device (501) of FIG. 5) can output (e.g., display) content of a target object in an area larger than an area corresponding to the target object.

In FIG. 12, in the basic operation mode (1201), the electronic device may provide a user with a space including an interaction object (1212) and a target object, a poster (1211). For example, the poster (1211) may include a picture of a waterfall.

The electronic device can obtain image information including a scene included in an internal area of the poster (1211). The electronic device can obtain surrounding information of the electronic device. Based on the image information and the surrounding information, the electronic device can obtain content related to the scene with visual properties corresponding to the weather or time indicated by the surrounding information.

In various embodiments of the present disclosure, content is mainly described as being generated based on image information and surrounding information, but is not limited thereto. According to one embodiment, the electronic device can generate content based on image information. For example, the electronic device can generate content independently (e.g., regardless) of surrounding information.

The electronic device can change the operating mode of the electronic device from the basic operating mode (1201) to the idle operating mode (1202) based on satisfying a condition for outputting content (e.g., a condition corresponding to switching from the basic operating mode (1201) to the idle operating mode (1202).

The electronic device can output content in an area larger than an area corresponding to a target object among the display areas of the display. In the idle operation mode (1202), the electronic device can deactivate the interaction object (1212) and output content (1213) of the target object (1211) in an area larger than the area corresponding to the target object (1211). In FIG. 12, for example, the content (1213) can include a moving picture showing a waterfall falling.

An electronic device according to one embodiment can provide a user experience in an idle operation mode (1202) that is distinct from a basic operation mode (1201) by outputting content of a target object in an area larger than an area corresponding to the target object. As illustrated in FIG. 12, the electronic device can provide immersive content to the user by outputting content in most of an area corresponding to a space provided to the user (e.g., a display area of a display). As a result, the user can be immersed in the content, and thus the burden on the user for interaction with the electronic device or an interaction object provided by the electronic device can be alleviated.

FIG. 13 is a diagram illustrating an example of a configuration for obtaining and/or outputting content when an electronic device selects multiple target objects according to various embodiments.

An electronic device according to one embodiment (e.g., an electronic device (101) of FIG. 1, an electronic device (201) of FIG. 2, an electronic device (301) of FIG. 3, an electronic device (401) of FIG. 4, an electronic device (501) of FIG. 5) can acquire and output contents of a plurality of target objects.

An electronic device can select multiple target objects from among physical objects.

In the basic operation mode (1301), the electronic device can provide the user with a space in which a light bulb (1312) and a poster (1311) are placed. The electronic device can select the light bulb (1312) and the poster (1311) as target objects among the physical objects. The poster (1311) can include a scene of a beach photographed in an interior area of the poster (1311).

The electronic device can obtain content for each of the selected plurality of target objects. According to one embodiment, the electronic device can obtain content for each of the plurality of target objects based on image information for the corresponding target object and surrounding information of the electronic device. The electronic device can obtain content for each target object by applying input data for each target object to a content generation model (e.g., content generation model (1120) of FIG. 11).

In Fig. 13, the surrounding information of the electronic device can indicate the time of sunset. The electronic device can obtain content about a beach at sunset as the content of the poster (1311). The electronic device can obtain content about the light bulb (1312) emitting light of the color of the sun's light source at sunset as the content of the light bulb (1312).

According to one embodiment, when a plurality of target objects are selected, the electronic device can sequentially output contents of the target objects according to sequentially satisfying conditions. The priorities of the plurality of target objects can be determined, and each target object can be replaced with the contents of the corresponding target object based on the priorities.

The priorities for multiple target objects can be determined based on the sizes of regions corresponding to the multiple target objects. For example, the larger the size of the region corresponding to the target object, the higher the priority of the target object can be determined.

The priority for multiple target objects may be determined based on the degree to which each target object is related to the natural environment. For example, a first object including a plant may be more related to the natural environment than a second object including a car. For example, a first object including a scene of a mountain or a beach may be more related to the natural environment than a second object including a scene of a city.

For example, the electronic device may determine that the object or a scene included in the object belongs to at least one object set from among a first object set related to a natural environment or a second object set not related to a natural environment. The first object set may include at least one of a plant, an animal, a tree, a forest, a mountain, an ocean, a sky, a sun, a moon, a star, or a cloud. The second object set may include at least one of a car, a building, steel, a factory, a machine, plastic, or an industrial product. The electronic device may determine a priority of the target object based on whether the target object belongs to the first object set or the second object set.

In FIG. 13, for example, the electronic device may determine the priority of the poster (1311) to be higher than the priority of the light bulb (1312).

The electronic device may output the first content in an area corresponding to the first target object determined according to priorities of the plurality of target objects based on satisfying a first condition regarding output of the content. The first condition may mean a condition for outputting the content of the target object with the highest priority among the plurality of target objects.

In FIG. 13, the electronic device can change the operation mode of the electronic device from the basic operation mode (1301) to the first idle operation mode (1302) based on satisfying the first condition. In the first idle operation mode (1302), the electronic device can output the content (1321) of the poster (1311) in an area corresponding to the poster (1311). In the first idle operation mode (1302), the electronic device can restrict outputting the content (1331) of the light bulb (1312).

The electronic device may output the second content in an area corresponding to a second target object determined according to priorities of the plurality of target objects based on further satisfying a second condition regarding output of the content. The second condition may mean a condition for outputting the content of a target object with a second highest priority among the plurality of target objects. According to one embodiment, the second condition may include further satisfying an additional condition corresponding to the second condition while satisfying the first condition.

In FIG. 13, the electronic device can change the operation mode of the electronic device from the first rest operation mode (1302) to the second rest operation mode (1303) based on satisfying the second condition. In the second rest operation mode (1303), the electronic device can output the contents (1331) of the light bulb (1312) in the area corresponding to the light bulb (1312).

According to one embodiment, the operation mode of the electronic device may be a basic operation mode or one of a plurality of idle operation modes of a plurality of levels. As the level of the idle operation mode increases, the difference from the space provided in the basic operation mode may increase. For example, as the level of the idle operation mode increases, the number of interaction objects that are deactivated may increase, or the degree of deactivation of the interaction objects may be strengthened (e.g., increased transparency of the interaction object, decreased brightness of the interaction object). As the level of the idle operation mode increases, the size of the area where the content of the target object is output may increase.

The electronic device may change the operation mode of the electronic device from the basic operation mode (1301) to the first idle operation mode (1302) when a first condition is satisfied. The first condition may mean a condition corresponding to a transition from the basic operation mode (1301) to the first idle operation mode (1302). For example, the first condition may include that the user's gaze is maintained at a specific area (e.g., an area different from an area corresponding to an interaction object) for a first threshold time. The first idle operation mode (1302) may be a lowest level idle operation mode among idle operation modes of a plurality of levels.

The electronic device, in the first idle operation mode (1302), can output content for a target object with the highest priority in an area corresponding to the target object with the highest priority among a plurality of target objects. For example, the electronic device, in the first idle operation mode (1302), can replace a target object (e.g., a poster (1311)) assigned to the first idle operation mode (1302) with content (1321).

The electronic device may change the operation mode of the electronic device from the first rest operation mode (1302) to the second rest operation mode (1303) if the second condition is further satisfied. The second condition may mean a condition corresponding to increasing a level of the rest operation mode (e.g., switching from the first rest operation mode (1302) to the second rest operation mode (1303)). For example, the second condition may include not detecting a user input that triggers an operation of the electronic device for a second threshold time period. The second rest operation mode (1303) may be a rest operation mode of the second lowest level among rest operation modes of a plurality of levels.

The electronic device, in the second idle operation mode (1303), can output content for a target object with the second highest priority in an area corresponding to the target object with the second highest priority among a plurality of target objects. For example, the electronic device, in the second idle operation mode (1303), can replace a target object (e.g., a light bulb (1312)) assigned to the second idle operation mode (1303) with content (1331). The electronic device, in the second idle operation mode (1303), can maintain replacing a target object (e.g., a poster (1311)) assigned to a lower level idle operation mode (e.g., the first idle operation mode (1302)) than the second idle operation mode (1303) with content (e.g., content (1321)).

FIG. 14 is a block diagram illustrating an example configuration of an electronic device according to various embodiments.

An electronic device according to one embodiment (e.g., an electronic device (101) of FIG. 1, an electronic device (201) of FIG. 2, an electronic device (301) of FIG. 3, an electronic device (401) of FIG. 4, an electronic device (501) of FIG. 5) may include an image sensing module (1410), a user input sensing module (1420), a processing module (1430), an object extraction module (1440), a content generation module (1450), a display module (1460), a storage module (1470), and an object analysis module (1480).

The image sensing module (1410) can sense image data regarding a physical space and/or a physical object around the electronic device. For example, the image sensing module (1410) can include a camera sensor and/or a depth sensor. The image sensing module (1410) can transmit the image data to the object analysis module (1480).

The object analysis module (1480) can analyze image data of physical space and/or physical objects around the electronic device. The object analysis module (1480) can select a target object to be used for generating content among the physical objects.

The user input sensing module (1420) can sense (e.g., detect) a user's input. The user input sensing module (1420) can transmit information about the detected user's input to the processing module (1430). The user input sensing module (1420) can detect the satisfaction of a condition by monitoring a user's input (e.g., gaze input, gesture input) when a condition corresponding to a transition between operation modes of an electronic device is based on a user's input.

The processing module (1430) can generate a stereoscopic image of a space to be provided to the user. The processing module (1430) can receive information about an interaction object from an external device or obtain it from a storage unit. The processing module (1430) can generate and/or display a stereoscopic image including an interaction object.

The object extraction module (1440) can extract a target object from image data and/or a stereoscopic image. The object extraction module (1440) can divide (e.g., crop) an area corresponding to the target object from the image data and transmit image information corresponding to the target object (e.g., an image capturing the target object) to the content generation module (1450).

The content generation module (1450) can generate content for a target object based on information about the target object (e.g., image information about the target object) and/or information about the surrounding environment (e.g., surrounding information of the electronic device). The content generation module (1450) can transfer information about the generated content to the storage module (1470). In FIG. 14, the content generation module (1450) is illustrated as being included as a component of the electronic device (e.g., on-device), but is not limited thereto, and the content generation module (1450) can be included in another electronic device external to the electronic device (e.g., on-cloud).

The display module (1460) can display the generated content together with the target object, or display the generated content object instead of the target object.

The storage module (1470) can store information about generated content.

The electronic devices according to various embodiments disclosed in this document may be devices of various forms. The electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliance devices. The electronic devices according to embodiments of this document are not limited to the above-described devices.

It should be understood that the various embodiments of this document and the terminology used herein are not intended to limit the technical features described in this document to specific embodiments, but include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly dictates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first", "second", or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

The term "module" used in various embodiments of this document may include a unit implemented in hardware, software or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be an integrally configured component or a minimum unit of the component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the called at least one instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, ‘non-transitory’ simply means that the storage medium is a tangible device and does not contain signals (e.g. electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded between a seller and a buyer as a commodity. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play StoreTM) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a part of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as a memory of a manufacturer's server, a server of an application store, or an intermediary server.

According to various embodiments, each component (e.g., a module or a program) of the above-described components may include a single or multiple entities, and some of the multiple entities may be separately arranged in other components. According to various embodiments, one or more components or operations of the above-described corresponding components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, the multiple components (e.g., a module or a program) may be integrated into one component. In such a case, the integrated component may perform one or more functions of each of the multiple components identically or similarly to those performed by the corresponding component of the multiple components before the integration. According to various embodiments, the operations performed by the module, program, or other component may be executed sequentially, in parallel, repeatedly, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

The embodiments described above may be implemented as hardware components, software components, and/or a combination of hardware components and software components. For example, the devices, methods, and components described in the embodiments may be implemented using a general-purpose computer or a special-purpose computer, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and software applications running on the OS. In addition, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone, but those skilled in the art will appreciate that the processing device may include multiple processing elements and/or multiple types of processing elements. For example, a processing device may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

The software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and/or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, or computer storage medium or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over network-connected computer systems and stored or executed in a distributed manner. The software and data may be stored on a computer-readable recording medium.

The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination, and the program commands recorded on the medium may be those specially designed and configured for the embodiment or may be those known to and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands such as ROMs, RAMs, flash memories, etc. Examples of program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc.

The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Claims (15)

In electronic devices (101; 201; 301; 401; 501), display(160; 205; 210; 320; 421); At least one processor (120) comprising a processing circuit; and A memory (130) comprising one or more storage media storing instructions, When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) Obtain image information (1111) corresponding to a target object (911) determined among physical objects placed in the physical space around the electronic device (101; 201; 301; 401; 501), Based on the acquired image information (1111) and the surrounding information (1112) of the electronic device (101; 201; 301; 401; 501), content (913; 1131) related to the target object (911) is acquired, Based on satisfying the conditions regarding the output of the above content (913; 1131), the operable interaction object (912) is disabled in response to the user's input, Through the above display (160; 205; 210; 320; 421), the acquired content (913; 1131) is output in the area corresponding to the target object (911). To do, Electronic devices (101; 201; 301; 401; 501). In the first paragraph, The conditions for outputting the above content (913; 1131) are: At least one of the following: the user's gaze does not point to the interaction object (912) for a threshold time period, the user's biometric information satisfies a predetermined condition, or the user's input that triggers an operation of the electronic device (101; 201; 301; 401; 501) is not detected from the user for a threshold time period. Electronic devices (101; 201; 301; 401; 501). In any one of paragraphs 1 and 2, When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) Displaying a stereoscopic image of a space including a background (810) and an interaction object (912) based on a physical space around the electronic device (101; 201; 301; 401; 501) through the display (160; 205; 210; 320; 421), The image information (1111) is obtained by segmenting the part corresponding to the target object (911) from the above stereoscopic image. To do, Electronic devices (101; 201; 301; 401; 501). In any one of claims 1 to 3, When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) Select multiple target objects from the above physical objects, Obtain content for each of the above-mentioned multiple target objects, Based on satisfying the first condition regarding the output of the above content, the first content is output in an area corresponding to the first target object determined according to the priority of the plurality of target objects, Based on further satisfying the second condition regarding the output of the above content, the second content is output in an area corresponding to the second target object determined according to the priority of the plurality of target objects. To do, Electronic devices (101; 201; 301; 401; 501). In any one of claims 1 to 4, When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) Based on displaying a scene in the internal area of the target object (911), the image information (1111) including the scene is obtained, Obtaining the surrounding information (1112) including at least one of weather information or time information about the surroundings of the electronic device (101; 201; 301; 401; 501), Based on the image information (1111) and the surrounding information (1112), content (913; 1131) related to the scene is obtained with visual properties corresponding to the weather or time indicated by the surrounding information (1112). Output the content (913; 1131) in the internal area of the target object (911) To do, Electronic devices (101; 201; 301; 401; 501). In any one of claims 1 to 5, When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) Based on satisfying the condition for stopping the output of the above content (913; 1131), the output of the above content (913; 1131) is stopped, Activate the above interaction object (912) To do, Electronic devices (101; 201; 301; 401; 501). In any one of claims 1 to 6, When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) In a space including a background (810) based on the physical space and the interaction object (912), the display of the interaction object (912) is maintained based on the fact that the interaction object (912) is positioned in a position aligned with the background (810). The display of the interaction object (912) is limited based on the fact that the interaction object (912) is arranged in a position aligned with the electronic device (101; 201; 301; 401; 501) within the space. To do, Electronic devices (101; 201; 301; 401; 501). In any one of claims 1 to 7, When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) Based on the acquired image information (1111) and the surrounding information (1112) of the electronic device (101; 201; 301; 401; 501), a sound (1132) regarding the movement of the target object (911) appearing in the content (913; 1131) is acquired, Playing the acquired sound (1132) together with the output of the above content (913; 1131) To do, Electronic devices (101; 201; 301; 401; 501). In any one of claims 1 to 8, When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) The content (913; 1131) is output in an area larger than the area corresponding to the target object (911) among the display areas of the above display (160; 205; 210; 320; 421). To do, Electronic devices (101; 201; 301; 401; 501). In any one of claims 1 to 9, When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) When the operation mode of the electronic device (101; 201; 301; 401; 501) is the first operation mode, based on satisfying a condition corresponding to a transition from the first operation mode to the second operation mode, the operation mode of the electronic device (101; 201; 301; 401; 501) is changed from the first operation mode to the second operation mode, While the operation mode of the electronic device (101; 201; 301; 401; 501) is the second operation mode, outputting the content (913; 1131) and deactivating the interaction object (912) are performed. When the operation mode of the electronic device (101; 201; 301; 401; 501) is the second operation mode, based on satisfying a condition corresponding to a transition from the second operation mode to the first operation mode, the operation mode of the electronic device (101; 201; 301; 401; 501) is changed from the second operation mode to the first operation mode, While the operation mode of the electronic device (101; 201; 301; 401; 501) is the first operation mode, outputting the content (913; 1131) in the area corresponding to the target object (911) is restricted and the interaction object (912) is activated. To do, Electronic devices (101; 201; 301; 401; 501). In a method performed by an electronic device (101; 201; 301; 401; 501), An operation of acquiring image information (1111) corresponding to a target object (911) determined among physical objects arranged in a physical space around the electronic device (101; 201; 301; 401; 501); An operation of acquiring content (913; 1131) related to the target object (911) based on the acquired image information (1111) and surrounding information (1112) of the electronic device (101; 201; 301; 401; 501); An action of disabling an operable interaction object (912) in response to a user's input based on satisfying a condition regarding the output of the above content (913; 1131); and An operation of outputting the acquired content (913; 1131) in an area corresponding to the target object (911) through a display (160; 205; 210; 320; 421). A method including: In Article 11, The conditions for outputting the above content (913; 1131) are: At least one of the following: the user's gaze does not point to the interaction object (912) for a threshold time period, the user's biometric information satisfies a predetermined condition, or the user's input that triggers an operation of the electronic device (101; 201; 301; 401; 501) is not detected from the user for a threshold time period. method. In any one of Articles 11 to 12, The operation of obtaining the above image information (1111) is as follows: An operation of displaying a stereoscopic image of a space including a background (810) and an interaction object (912) based on a physical space around the electronic device (101; 201; 301; 401; 501) through the display (160; 205; 210; 320; 421); and Including an operation of obtaining the image information (1111) by segmenting a portion corresponding to the target object (911) from the three-dimensional image. method. In any one of Articles 11 to 13, The action of outputting the above content (913; 1131) is: An operation of selecting multiple target objects from among the above physical objects; and Including an operation of obtaining content for each of the plurality of selected target objects, The operation of outputting the above acquired content (913; 1131) is as follows: An operation of outputting a first content in an area corresponding to a first target object determined according to the priority of the plurality of target objects based on satisfying a first condition regarding the output of the content; and An operation of outputting a second content in an area corresponding to a second target object determined according to the priority of the plurality of target objects, based on further satisfying a second condition regarding the output of the content, method. A computer-readable recording medium storing one or more computer programs including commands for performing the method of any one of claims 11 to 14.