KR20250023244A - Method for supporting object selection in virtual environment and electronic device supporting the same - Google Patents

Abstract

Various embodiments of the present disclosure provide a method for supporting object selection in a virtual environment and an electronic device supporting the same. An electronic device according to an embodiment of the present disclosure may include a display and a processor. The processor according to an embodiment may operate to recognize an input device in a virtual environment. The processor may be configured to determine a virtual space for determining a target virtual object based on an attribute of the input device. The processor according to an embodiment may operate to determine the target virtual object based on the virtual space. The processor according to an embodiment may operate to generate affordances respectively corresponding to the target virtual object. The processor according to an embodiment may operate to display the affordance according to a specified arrangement method. Various embodiments are possible.

Description

METHOD FOR SUPPORTING OBJECT SELECTION IN VIRTUAL ENVIRONMENT AND ELECTRONIC DEVICE SUPPORTING THE SAME

Embodiments of the present disclosure provide a method capable of supporting object selection in a virtual environment and an electronic device supporting the same.

Recently, research and development are being conducted on expanded reality (XR) technologies such as virtual reality (VR), augmented reality (AR), and/or mixed reality (MR). Recently, VR, AR, and/or MR technologies have been utilized in various fields (e.g., entertainment, infotainment, smart home, and/or smart factory), and hardware and/or software parts of electronic devices for this purpose are continuously being researched and developed.

For example, a wearable electronic device can operate independently (e.g., in a standalone manner) or in conjunction with at least two or more devices (e.g., in a tethered manner), and can provide a single image through the display of the wearable electronic device by overlaying various digital contents (e.g., virtual images) on the real world through an application related to the AR service. For example, recently, AR environments such as the tethered AR method, in which an electronic device (e.g., a smartphone) and a wearable electronic device are connected and virtual contents generated by the electronic device are provided through the display of the wearable electronic device, and the standalone AR method, in which the wearable electronic device generates virtual contents on its own without being connected to the electronic device and provides them through the display, have been implemented.

In this way, with the recent technological advancement of AR services, the number of users using AR services is increasing, and the user needs are also increasing accordingly. For example, users using AR services are increasingly demanding to select and display content in the AR environment more accurately, intuitively, and conveniently.

The above information may be provided as related art for the purpose of assisting in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art related to the present disclosure.

In one embodiment of the present disclosure, a method for supporting object selection in a virtual environment and an electronic device supporting the same are provided.

In one embodiment of the present disclosure, a method for providing an affordance (or guide handle) for easily selecting an object in a virtual environment based on an input device recognized in the virtual environment and an electronic device supporting the same are provided.

In one embodiment of the present disclosure, a method for determining a display position and type of affordance for supporting object selection in a virtual environment and an electronic device supporting the same are provided.

The technical problems to be achieved in this document are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art to which the present invention belongs from the description below.

An electronic device according to one embodiment of the present disclosure may include a display and a processor operatively connected with the display. The processor according to one embodiment may be operable to recognize an input device in a virtual environment. The processor according to one embodiment may be operable to determine a virtual space for determining a target virtual object based on a property of the input device. The processor according to one embodiment may be operable to determine the target virtual object based on the virtual space. The processor according to one embodiment may be operable to generate an affordance corresponding to each of the target virtual objects. The processor according to one embodiment may be operable to display the affordance according to a specified arrangement method.

A method of operating an electronic device according to one embodiment of the present disclosure may include an operation of recognizing an input device in a virtual environment. The method of operating may include an operation of determining a virtual space for determining a target virtual object based on a property of the input device. The method of operating may include an operation of determining the target virtual object based on the virtual space. The method of operating may include an operation of generating an affordance corresponding to each of the target virtual objects. The method of operating may include an operation of displaying the affordance according to a specified arrangement method.

In order to solve the above-mentioned problem, various embodiments of the present disclosure may include a computer-readable recording medium having recorded thereon a program for executing the method on a processor.

According to one embodiment, a non-transitory computer-readable storage medium (or a computer program product) storing one or more programs is described. According to one embodiment, the one or more programs may include instructions that, when executed by a processor of an electronic device, perform operations of recognizing an input device in a virtual environment, determining a virtual space for determining a target virtual object based on a property of the input device, determining the target virtual object based on the virtual space, generating an affordance corresponding to each of the target virtual objects, and displaying the affordance according to a specified arrangement method.

Further scope of the applicability of the present disclosure will become apparent from the detailed description below. However, since various changes and modifications within the spirit and scope of the present disclosure will become apparent to those skilled in the art, it should be understood that the detailed description and specific embodiments, such as the preferred embodiments of the present disclosure, are given by way of example only.

According to an electronic device, an operation method thereof, and a recording medium according to one embodiment of the present disclosure, an affordance (or a guide handle) for easily selecting an object in a virtual environment can be provided based on an input device recognized in the virtual environment. According to one embodiment, in a virtual environment including dense objects, it is possible to support a user to accurately and easily select a desired object among dense objects. According to one embodiment, when selecting an object in a virtual environment, the shape of the object is maintained, and an affordance is provided based on an object included in a selectable range by a user input, so as to maintain a user's sense of immersion in using the virtual environment. According to one embodiment, user needs according to the use of a virtual service (e.g., AR service and/or VR service) can be satisfied, and a new user experience (UX) can be provided to the user.

In addition, various effects that can be directly or indirectly understood through this document can be provided. The effects that can be obtained from this disclosure are not limited to the effects mentioned above, and other effects that are not mentioned can be clearly understood by those skilled in the art to which this disclosure belongs from the description below.

In connection with the drawing description, the same or similar reference numerals may be used for identical or similar components.
FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.
FIG. 2 is a diagram illustrating an example of an electronic device according to one embodiment.
FIG. 3 is a diagram illustrating an example of the internal structure of an electronic device according to one embodiment.
FIG. 4A is a diagram illustrating a network environment between an electronic device and another electronic device according to one embodiment.
FIG. 4b is a diagram illustrating an example of displaying a virtual environment in an electronic device according to one embodiment.
FIG. 5 is a diagram schematically illustrating the configuration of an electronic device according to one embodiment.
FIG. 6 is a flowchart illustrating a method of operating an electronic device according to one embodiment of the present disclosure.
FIGS. 7A, 7B, 7C, and 7D are diagrams illustrating examples of operations for providing affordance in association with a virtual object in an electronic device according to one embodiment of the present disclosure.
FIG. 8 is a flowchart illustrating a method of operating an electronic device according to one embodiment of the present disclosure.
FIG. 9 is a diagram illustrating examples of indicators for various input devices in an electronic device according to one embodiment of the present disclosure.
FIG. 10 is a diagram illustrating an example of a virtual space generated by an electronic device and an indicator corresponding to an input device according to one embodiment of the present disclosure.
FIG. 11 is a flowchart illustrating a method of operating an electronic device according to one embodiment of the present disclosure.
FIG. 12 is a diagram illustrating an example of providing a virtual space corresponding to an input device in an electronic device according to one embodiment of the present disclosure.
FIG. 13 is a diagram illustrating an example of resolution for each input device provided by an electronic device according to one embodiment of the present disclosure.
FIG. 14 is a flowchart illustrating a method of operating an electronic device according to one embodiment of the present disclosure.
FIG. 15 is a flowchart illustrating a method of operating an electronic device according to one embodiment of the present disclosure.
FIGS. 16A and 16B are diagrams illustrating examples of determining a target virtual object in an electronic device according to one embodiment of the present disclosure.
FIG. 17 is a flowchart illustrating a method of operating an electronic device according to one embodiment of the present disclosure.
FIG. 18 is a flowchart illustrating a method of operating an electronic device according to one embodiment of the present disclosure.
FIGS. 19A, 19B, and 19C are diagrams illustrating examples of providing affordance in an electronic device according to one embodiment of the present disclosure.
FIG. 20 is a flowchart illustrating a method of operating an electronic device according to one embodiment of the present disclosure.
FIGS. 21A, 21B, and 21C are diagrams illustrating examples of determining the size of an affordance in an electronic device according to one embodiment of the present disclosure.
FIGS. 22A and 22B are diagrams illustrating examples of operations for providing affordance in association with a virtual object in an electronic device according to one embodiment of the present disclosure.
FIGS. 23A, 23B, and 23C are diagrams illustrating examples of operations for providing affordance in association with a virtual object in an electronic device according to one embodiment of the present disclosure.
FIGS. 24A, 24B, and 24C are diagrams illustrating examples of operations for providing affordance in association with a virtual object in an electronic device according to one embodiment of the present disclosure.
FIGS. 25A and 25B are diagrams illustrating examples of operations for providing affordance in association with a virtual object in an electronic device according to one embodiment of the disclosure.
FIG. 26 is a diagram illustrating an example of an operation of providing affordance in an electronic device according to one embodiment of the present disclosure.
FIGS. 27A, 27B, and 27C are diagrams illustrating examples of operations for providing affordance in association with a virtual object in an electronic device according to one embodiment of the present disclosure.
FIGS. 28A and 28B are diagrams illustrating examples of operations for providing affordance in association with a virtual object in an electronic device according to one embodiment of the present disclosure.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In connection with the description of the drawings, the same or similar reference numerals may be used for the same or similar components. In addition, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and conciseness.

FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments.

Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

The processor (120) may control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) by executing, for example, software (e.g., a program (140)), and may perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculations, the processor (120) may store a command or data received from another component (e.g., a sensor module (176) or a communication module (190)) in the volatile memory (132), process the command or data stored in the volatile memory (132), and store result data in the nonvolatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit (CPU) or an application processor (AP)) or an auxiliary processor (123) (e.g., a graphic processing unit (GPU), a neural processing unit (NPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use lower power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

The auxiliary processor (123) may control at least a part of functions or states related to at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing device) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, in the electronic device (101) itself on which the artificial intelligence model is executed, or may be performed through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers. The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), deep Q-networks, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may additionally or alternatively include a software structure.

The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or nonvolatile memory (134).

The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (OS) (142), middleware (144), or an application (146).

The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

The audio output module (155) can output an audio signal to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes such as multimedia playback or recording playback. The receiver can be used to receive an incoming call. According to one embodiment, the receiver can be implemented separately from the speaker or as a part thereof.

The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) can include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) can include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can obtain sound through an input module (150), or output sound through an audio output module (155), or an external electronic device (e.g., an electronic device (102)) (e.g., a speaker or a headphone) directly or wirelessly connected to the electronic device (101).

The sensor module (176) can detect an operating state (e.g., power or temperature) of the electronic device (101) or an external environmental state (e.g., user state) and generate an electric signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.

The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., the electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that a user can perceive through a tactile or kinesthetic sense. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module (180) can capture still images and moving images. According to one embodiment, the camera module (180) can include one or more lenses, image sensors, image signal processors, or flashes.

The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

The battery (189) can power at least one component of the electronic device (101). In one embodiment, the battery (189) can include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

The communication module (190) may support establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., the electronic device (102), the electronic device (104), or the server (108)), and performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., the application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module or a power line communication module). Among these communication modules, a corresponding communication module may communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or a wide area network (WAN))). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as a plurality of separate components (e.g., multiple chips). The wireless communication module (192) may use subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196) to identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199).

The wireless communication module (192) can support a 5G network and next-generation communication technology after a 4G network, for example, NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB, enhanced mobile broadband), minimization of terminal power and connection of multiple terminals (mMTC, massive machine type communications), or high reliability and low latency communications (URLLC, ultra-reliable and low-latency communications). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in an electronic device (101), an external electronic device (e.g., electronic device (104)), or a network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) may support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL) each, or 1 ms or less for round trip) for URLLC realization.

The antenna module (197) can transmit or receive signals or power to or from the outside (e.g., an external electronic device). According to one embodiment, the antenna module (197) can include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) can include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), can be selected from the plurality of antennas by, for example, the communication module (190). A signal or power can be transmitted or received between the communication module (190) and the external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) can be additionally formed as a part of the antenna module (197).

According to various embodiments, the antenna module (197) can form a mmWave antenna module. According to one embodiment, the mmWave antenna module can include a printed circuit board, an RFIC positioned on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) positioned on or adjacent a second side (e.g., a top side or a side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

At least some of the above components may be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, a general purpose input and output (GPIO), a serial peripheral interface (SPI), or a mobile industry processor interface (MIPI)).

In one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). In one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform at least a part of the function or service. One or more external electronic devices that have received the request may execute at least a part of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result, as is or additionally processed, as at least a part of a response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide an ultra-low latency service by using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (104) may include an IoT (Internet of Things) device. The server (108) may be an intelligent server using machine learning and/or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199). The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

The electronic devices according to various embodiments disclosed in this document may be devices of various forms. The electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliance devices. The electronic devices according to embodiments of this document are not limited to the above-described devices.

It should be understood that the various embodiments of this document and the terminology used herein are not intended to limit the technical features described in this document to specific embodiments, but include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly dictates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first", "second", or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

The term "module" used in various embodiments of this document may include a unit implemented in hardware, software or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be an integrally configured component or a minimum unit of the component or a portion thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the called at least one instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, ‘non-transitory’ simply means that the storage medium is a tangible device and does not contain signals (e.g. electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded between a seller and a buyer as a commodity. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store ^TM ) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a part of the computer program product may be at least temporarily stored or temporarily generated in a machine-readable storage medium, such as a memory of a manufacturer's server, a server of an application store, or an intermediary server.

According to various embodiments, each component (e.g., a module or a program) of the above-described components may include a single or multiple entities, and some of the multiple entities may be separately arranged in other components. According to various embodiments, one or more components or operations of the above-described corresponding components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, the multiple components (e.g., a module or a program) may be integrated into one component. In such a case, the integrated component may perform one or more functions of each of the multiple components identically or similarly to those performed by the corresponding component of the multiple components before the integration. According to various embodiments, the operations performed by the module, program, or other component may be executed sequentially, in parallel, repeatedly, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Before describing various embodiments of the present disclosure, an electronic device (101) to which an embodiment of the present disclosure can be applied is described.

FIG. 2 is a diagram illustrating an example of an electronic device according to one embodiment.

According to one embodiment, FIG. 2 illustrates an example in which the electronic device (101) is a wearable electronic device (201) in the form of glasses (e.g., a glasses-type display device or AR (augmented reality) glasses), but is not limited thereto. For example, the wearable electronic device (201) may include various forms of devices that include a display and are worn (or installed) on a part of a user's body (e.g., a face or a head) to provide augmented reality (AR), mixed reality (MR), and/or virtual reality (VR) services. For example, the wearable electronic device (201) may be implemented in at least one form of glasses, goggles, a helmet, or a hat, but is not limited thereto.

According to one embodiment, the wearable electronic device (201) may include a VST (video see-through) device. For example, the VST device may selectively provide augmented reality and virtual reality services. The VST device may include a plurality of cameras and may track the gaze of a user based on the plurality of cameras (e.g., the gaze tracking cameras (312-1, 312-2) of FIG. 3). In one embodiment, eye tracking may refer to a technology for tracking the position and/or movement of a user's gaze. According to one embodiment, the VST device may capture or detect the real world in front of the user based on the plurality of cameras and display the real world through a display (a display corresponding to each of the user's two eyes). The VST device may provide an augmented reality service by displaying a virtual object while displaying the real world.

The wearable electronic device (201) described below may be a device including at least some of the components included in the electronic device (101) described above with reference to FIG. 1. Even if not mentioned in the description below, it may be interpreted that the wearable electronic device (201) according to the present disclosure may include various components as described with reference to FIG. 1.

According to one embodiment, the wearable electronic device (201) may be worn on a user's face and may provide an image (e.g., an augmented reality image, a mixed reality image, and/or a virtual reality image) to the user. According to one embodiment, the wearable electronic device (201) may provide an AR service that superimposes virtual information (or virtual objects) on at least a part of an actual real space (or environment). For example, the wearable electronic device (201) may provide the user with virtual information by superimposing it on an actual real space corresponding to the wearer's field of view (FOV).

Referring to FIG. 2, a wearable electronic device (201) may include a glass member (or window member) (210) positioned at positions corresponding to the user's two eyes (e.g., left eye and right eye), a main frame (or main body) (240) that fixes the glass member (210), a support frame (or support member) (250) that is connected to both ends of the main frame (240) and is placed on the user's ear, and a camera module (280) (e.g., a camera for taking pictures).

According to one embodiment, the glass member (210) may include a first glass (220) corresponding to the user's left eye and a second glass (230) corresponding to the user's right eye. According to one embodiment, the glass member (210) may be supported by the main frame (240). For example, the glass member (210) may be fitted into an opening formed in the main frame (240). According to one embodiment, an AR image emitted from a display module (e.g., the display module (160) of FIG. 1) may be projected onto the glass member (210).

According to one embodiment, the glass member (210) may have a waveguide (e.g., a waveguide or a transparent waveguide) formed in at least a portion of the region. In one embodiment, the waveguide may serve to guide an AR image emitted from the display module to the user's eyes. A detailed description of the waveguide is described with reference to the description related to the first glass (220) and the second glass (230) of FIG. 3.

According to one embodiment, the glass member (210) is implemented as a form that is separated into a first glass (220) and a second glass (230) to correspond to the left and right eyes of the user, respectively, as illustrated in FIG. 2. However, according to some embodiments, the glass member (210) may be implemented as a single glass form without distinction between the first glass (220) and the second glass (230).

According to one embodiment, the main frame (240) and the support frame (250) may be implemented in the form of glasses.

In one embodiment, the main frame (240) may be a structure that is at least partially placed on the user's nose. According to one embodiment, the main frame (240) may support the glass member (210). According to one embodiment, the main frame (240) may be formed of a synthetic resin material. According to one embodiment, the glass member (210) may be fitted into an opening formed in the main frame (240), thereby allowing the main frame (240) to support the glass member (210).

In one embodiment, the support frame (250) may include a first support frame (260) mounted on a first direction ear (e.g., a left ear) and a second support frame (270) mounted on a second direction ear (e.g., a right ear). For example, the main frame (240) and the support frame (250) (e.g., the first support frame (260) and the second support frame (270)) may be connected through a hinge portion (not shown) and may be combined so as to enable folding.

In one embodiment, the support frame (250) may be rotatably connected to the main frame (240). According to one embodiment, the support frame (250) may include a first support frame (260) and a second support frame (270). The first support frame (260) may be connected to the main frame (240) from the left side (e.g., the first direction) with respect to the main frame (240) when viewed from the ‘A’ direction. The second support frame (270) may be connected to the main frame (240) from the right side (e.g., the second direction) with respect to the main frame (240) when viewed from the ‘A’ direction.

In one embodiment, the support frame (250) may be fixedly installed on the main frame (240). For example, a first support frame (260) connected to the left side of the main frame (240) and a second support frame (270) connected to the right side of the main frame (240) may be formed to be connected to each other. According to one embodiment, the support frames (250) connected to both sides of the main frame (240) may be worn in a ring shape and fitted onto the user's head. In addition, the support frame (250) may be transformed into various shapes so that the wearable electronic device (201) may be worn on the user's face.

In one embodiment, the support frame (250) may be formed to be hung over the user's ear. For example, the wearable electronic device (201) may be worn on the user's face in such a way that the support frame (250) connected to the main frame (240) is hung over the user's ear. In one embodiment, the support frame (250) may be rotated relative to the main frame (240). In one embodiment, the support frame (250) may be rotated in a direction closer to the main frame (240) to reduce the volume of the wearable electronic device (201).

According to one embodiment, a display module (e.g., the display module (160) of FIG. 1) can output an AR image generated by a processor (120) of an electronic device (201) (e.g., the processor (120) of FIG. 1) or a processor of another electronic device (e.g., another electronic device (401) of FIG. 4A) connected to the electronic device (201) (e.g., a tethered electronic device (e.g., a smart phone)). When an AR image is generated by the display module and projected onto a glass member (210), an object included in the AR image can be combined with visible light (L) incident from the front (e.g., in a direction viewed by a user) through the glass member (210), thereby implementing AR. The display module may be a very small projector (e.g., a micro projector, a pico projector). For example, the display module may be a laser scanning display (LSD), a digital micro-mirror display (DMD), and/or a liquid crystal on silicon (LCoS). According to one embodiment, the display module may be a transparent display. In this case, the light-emitting element included in the display module may be directly placed on the glass member (210). In addition, the display module may be a variety of display devices for implementing AR.

In one embodiment, the glass member (210), the support frame (250), and/or the display module may be provided as a pair to correspond to the left and right eyes of the user. For example, the glass member (210) may include a first glass (220) and a second glass (230), and the support frame (250) may include a first support frame (260) and a second support frame (270). According to one embodiment, at least some of the components described above may have different configurations corresponding to the left eye and the right eye.

In one embodiment, the camera module (280) may represent, for example, a camera for shooting (e.g., a front shooting camera). For example, the camera module (280) may be implemented by being placed in the main frame (240) to shoot a subject in the front (or the front that the user is looking at) of the wearable electronic device (201). For example, the camera module (280) may be placed in a central portion (or a central point) between the first glass (220) and the second glass (230) in the main frame (240) to shoot a front of the main frame (240). In one embodiment, the front of the main frame (240) may mean a direction in which the user is looking when wearing the wearable electronic device (201).

According to one embodiment, the camera module (280) may include at least one camera, and the wearable electronic device (201) may perform operations such as photographing the real world located in front of the user, recognizing objects, and/or recognizing space including three-dimensional depth information through the camera module (280). According to one embodiment, the wearable electronic device (201) may also obtain relative position information of objects recognized based on the wearable electronic device (201) or position information of objects in the recognized space through the camera module (280).

In one embodiment, the wearable electronic device (201) may include a camera module (280) and a plurality of other cameras, and a detailed description of the camera module (280) and the plurality of other cameras is described with reference to FIG. 3.

FIG. 3 is a drawing illustrating an example of the internal structure of an electronic device according to one embodiment.

According to one embodiment, the wearable electronic device (201) (e.g., the electronic device (101) of FIG. 1) may be a device implemented in a form that is worn on a user's face or head. According to one embodiment, the wearable electronic device (201) may include a plurality of glasses (e.g., the first glass (220) and the second glass (230)) corresponding to each of the user's two eyes (e.g., the left eye and the right eye) and may be implemented in the form of glasses. According to one embodiment, the wearable electronic device (201) may provide the user with an image related to an AR service. According to one embodiment, the wearable electronic device (201) may project or display a virtual object on the first glass (220) and/or the second glass (230), thereby allowing at least one virtual object to be superimposed on the real world perceived by the user through the first glass (220) and/or the second glass (230) of the wearable electronic device (201).

Referring to FIG. 3, a wearable electronic device (201) according to one embodiment may include a main frame (or main body) (240), a support frame (e.g., a first support frame (260), a second support frame (270)), and a hinge part (e.g., a first hinge part (340-1), a second hinge part (340-2)).

According to one embodiment, the main frame (240) and the support frames (e.g., the first support frame (260) and/or the second support frame (270)) can mount various components of the wearable electronic device (201). According to one embodiment, the main frame (240) and the support frames (260, 270) can be operatively connected via hinge portions (340-1, 340-2).

In one embodiment, the main frame (240) may include a portion formed to be at least partially resting on a user's nose.

In one embodiment, the support frame (260, 270) may include a support member that is shaped to be worn over the user's ear. In one embodiment, the support frame (260, 270) may include a first support frame (260) that is worn over the user's left ear and a second support frame (270) that is worn over the user's right ear.

According to one embodiment, the first hinge portion (340-1) may connect the first support frame (260) and the main frame (240) so that the first support frame (260) may be rotatable relative to the main frame (240). According to one embodiment, the second hinge portion (340-2) may connect the second support frame (270) and the main frame (240) so that the second support frame (270) may be rotatable relative to the main frame (240). According to another embodiment, the hinge portions (340-1, 340-2) of the wearable electronic device (201) may be omitted. For example, the support frames (260, 270) may be directly connected to the main frame (240) and fixedly installed.

According to one embodiment, the main frame (240) includes glasses corresponding to the user's two eyes (e.g., the left eye and the right eye) (e.g., the first glass (220) and the second glass (230)), display modules (e.g., the first display module (314-1) and the second display module (314-2)), optical waveguides (e.g., the first optical waveguide (320) and the second optical waveguide (330)), a camera module (280) (e.g., a camera for shooting or a front shooting camera), a recognition camera (e.g., the first recognition camera (311-1), the second recognition camera (311-2)), and an eye tracking camera (e.g., the first eye tracking camera (312-1), the second eye tracking camera (312-2)), at least one microphone (e.g., the first microphone (341-1), the second microphone (341-2) and/or the third microphone (341-3)), and at least one lighting member (e.g., the first lighting member). It may include an absence (342-1) and/or a second lighting absence (342-2).

According to one embodiment, the wearable electronic device (201) can display various information by projecting light generated from the display modules (314-1, 314-2) onto the glasses (220, 230). For example, light generated from the first display module (314-1) can be projected onto the first glass (220), and light generated from the second display module (314-2) can be projected onto the second glass (230). According to one embodiment, the first glass (220) and the second glass (230) can be formed of at least a portion of a transparent material (e.g., a transparent member). According to one embodiment, the first glass (220) (or the first transparent member) corresponding to the user's left eye can be connected to the first display module (314-1), and the second glass (230) (or the second transparent member) corresponding to the user's right eye can be connected to the second display module (314-2). According to one embodiment, the first glass (220) and the second glass (230) may be formed of a glass plate, a plastic plate, and/or a polymer, and may be made transparent or translucent.

According to one embodiment, the display modules (e.g., the first display module (314-1) and the second display module (314-2)) may include a liquid crystal display (LCD), a digital micro-mirror device (DMD), a liquid crystal on silicon (LCoS), an organic light emitting diode (OLED), or a micro light emitting diode (micro LED).

According to one embodiment, the first glass (220) and the second glass (230) may include a condensing lens and/or an optical waveguide (or a transparent waveguide) (e.g., the first optical waveguide (320), the second optical waveguide (330)). According to one embodiment, the optical waveguides (320, 330) may be at least partially positioned on a portion of the glasses (220, 230). For example, the first optical waveguide (320) may be partially positioned on the first glass (220), and the second optical waveguide (330) may be partially positioned on the second glass (230). According to one embodiment, the optical waveguides (320, 330) may serve to transmit a light source generated by the display modules (314-1, 314-2) to the user's eyes. According to one embodiment, light emitted from the display module (314-1, 314-2) may be incident on one side (or one end) of the glass (220, 230). The light incident on one side of the glass (220, 230) may be transmitted to the user through the optical waveguide (320, 330) formed (or positioned) within the glass (220, 230).

In one embodiment, the optical waveguide (320, 330) can be made of glass, plastic, or polymer, and can include a nano-pattern formed on one surface of the inner or outer surface. The nano-pattern can include a grating structure having a polygonal or curved shape. According to one embodiment, light incident on one surface of the glass (220, 230) can be propagated or reflected by the nano-pattern inside the optical waveguide (320, 330) and transmitted to the user. According to one embodiment, the optical waveguide (320, 330) can include at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) or at least one reflective element (e.g., a reflective mirror). According to one embodiment, the optical waveguide (320, 330) can guide light emitted from a display module (314-1, 314-2) (e.g., a light source unit) to a user's eye using at least one diffractive element or reflective element.

In one embodiment, the diffractive element may include an input optical member/output optical member (not shown). The input optical member may mean an input grating area, and the output optical member (not shown) may mean an output grating area. The input grating area may serve as an input terminal that diffracts (or reflects) light output from a display module (314-1, 314-2) (e.g., micro LED) to transmit it to a glass (e.g., a first glass (220), a second glass (230)). The output grating area may serve as an outlet that diffracts (or reflects) light transmitted to the glass (e.g., a first glass (220), a second glass (230)) of the optical waveguide (320, 330) to a user's eye.

In one embodiment, the reflective element may include a total internal reflection (TIR) optical element or waveguide for total internal reflection. Total internal reflection may mean a method of guiding light such that light (e.g., a virtual image) input through the input grating region is reflected about 100% from one side (e.g., a designated side) of the optical waveguide (320, 330) at an angle of incidence such that about 100% of the light is transmitted to the output grating region.

According to one embodiment, light emitted from a display module (e.g., 314-1, 314-2) may be guided along an optical path to an optical waveguide (320, 330) through an input optical member. Light traveling inside the optical waveguide (320, 330) may be guided toward a user's eyes through an output optical member.

According to one embodiment, the display module (314-1, 314-2) may include a plurality of panels (or display areas), and the plurality of panels may be positioned on the glass (220, 230). According to one embodiment, at least a portion of the display module (314-1, 314-2) may be formed of a transparent element, and a user may perceive an actual space behind the display module (314-1, 314-2) by seeing through the display module (314-1, 314-2). According to one embodiment, the display module (314-1, 314-2) may display a virtual object on at least a portion of the transparent element so that the user may see that the virtual object (or virtual information) is added to at least a portion of the actual real space. According to one embodiment, when the display module (314-1, 314-2) is a transparent micro LED, the optical waveguide (320, 330) configuration within the glass (220, 230) may be omitted.

According to one embodiment, the wearable electronic device (201) may include a plurality of cameras (e.g., a first camera, a second camera, and a third camera). For example, the first camera (e.g., the camera module (280) of FIG. 2) may be a camera (280) (e.g., an RGB camera) for capturing an image corresponding to a field of view (FoV) of a user and/or for measuring a distance to an object. The second camera may be an eye tracking camera module (312-1, 312-2) for confirming a direction of a user's gaze. The third camera may be a recognition camera (gesture camera module) (311-1, 311-2) for recognizing a certain space.

According to one embodiment, the capturing camera (280) can capture a front direction of the wearable electronic device (201), and the gaze tracking cameras (312-1, 312-2) can capture a direction opposite to the capturing direction of the capturing camera (280). For example, the first gaze tracking camera (312-1) can partially capture the user's left eye, and the second gaze tracking camera (312-2) can partially capture the user's right eye. According to one embodiment, the wearable electronic device (201) can display a virtual object (or virtual information) related to an AR service together with image information related to an actual real space acquired through the capturing camera (280). According to one embodiment, the wearable electronic device (201) can display a virtual object based on display modules arranged to correspond to both eyes of the user (e.g., the first display module (314-1) corresponding to the left eye and/or the second display module (314-2) corresponding to the right eye). According to one embodiment, the wearable electronic device (201) may display a virtual object based on preset configuration information (e.g., resolution, frame rate, brightness, and/or display area).

According to one embodiment, the camera for shooting (280) may include a high-resolution camera, such as a high-resolution (HR) camera and/or a photo video (PV) camera. For example, the camera for shooting (280) may be utilized to obtain high-quality images, such as with an auto-focus function and optical image stabilization (OIS). In addition to a color camera, the camera for shooting (280) may also be implemented as a global shutter (GS) camera and a rolling shutter (RS) camera.

According to one embodiment, the gaze tracking camera (312-1, 312-2) can detect the gaze direction (e.g., eye movement) of the user. For example, the gaze tracking camera (312-1, 312-2) can detect the user's pupil and track the gaze direction. In one embodiment, the tracked gaze direction can be utilized to move the center of a virtual image including a virtual object in response to the gaze direction. For example, the gaze tracking camera (312-1, 312-2) can utilize a GS (global shutter) camera to detect the pupil and track rapid eye movement without screen tearing, and the performance and specifications of each gaze tracking camera (312-1, 312-2) can be substantially the same.

According to one embodiment, the recognition camera (311-1, 311-2) can detect a user gesture and/or a certain space within a preset distance (e.g., a certain space). According to one embodiment, the recognition camera (311-1, 311-2) can be used for head tracking, hand detection and/or hand tracking of 3DoF, 6DoF. For example, the recognition camera (311-1, 311-2) can be utilized to perform a space recognition for 6DoF, a simultaneous localization and mapping (SLAM) function through depth shooting. According to some embodiments, the recognition camera (311-1, 311-2) can be utilized for a gesture recognition function for recognizing a user gesture. The recognition camera (311-1, 311-2) can include a camera including a GS. For example, the recognition camera (311-1, 311-2) may include a camera including a GS with less screen drag (or in which the RS phenomenon can be reduced), such as a RS (rolling shutter) camera, to detect and track rapid hand movements and/or fine movements of fingers.

According to one embodiment, the wearable electronic device (201) can detect an eye corresponding to the dominant eye and/or the auxiliary eye among the left eye and/or the right eye of the user using at least one camera (311-1, 311-2, 312-1, 312-2, 280). For example, the wearable electronic device (201) can detect an eye corresponding to the dominant eye and/or the auxiliary eye based on a direction of the user's gaze with respect to an external object or a virtual object.

The number and position of at least one camera (e.g., a camera for shooting (280), a gaze tracking camera (312-1, 312-2), and/or a camera for recognition (311-1, 311-2)) included in the wearable electronic device (201) illustrated in FIG. 3 may not be limited. For example, the number and position of at least one camera (e.g., a camera for shooting (280), a gaze tracking camera (312-1, 312-2), and/or a camera for recognition (311-1, 311-2)) may be variously changed based on the form (e.g., shape or size) of the wearable electronic device (201).

According to one embodiment, the wearable electronic device (201) may include at least one lighting member (or illumination LED) (e.g., a first lighting member (342-1), a second lighting member (342-2)) to increase the accuracy of at least one camera (e.g., a shooting camera (280), an eye-tracking camera (312-1, 312-2), and/or a recognition camera (311-1, 311-2)). For example, the first lighting member (342-1) may be positioned at a portion corresponding to the user's left eye, and the second lighting member (342-2) may be positioned at a portion corresponding to the user's right eye.

According to one embodiment, the lighting elements (342-1, 342-2) may be utilized for different purposes depending on the location where they are attached to the wearable electronic device (201). In one embodiment, the lighting elements (342-1, 342-2) may be used as an auxiliary means to increase accuracy (e.g., ease of eye gaze detection) when capturing a user's pupil with an eye tracking camera (312-1, 312-2), and may include an IR LED that generates light of an infrared wavelength. In one embodiment, the lighting elements (342-1, 342-2) may be used as an auxiliary means to supplement ambient brightness when it is difficult to detect a subject to be captured due to a dark environment or mixing and reflection of multiple light sources when capturing a user's gesture with a recognition camera (311-1, 311-2).

According to one embodiment, the wearable electronic device (201) may include a microphone (e.g., a first microphone (341-1), a second microphone (341-2), and a third microphone (341-3)) for receiving a user's voice and ambient sounds.

According to one embodiment, the first support frame (260) (or the first housing) and/or the second support frame (270) (the second housing) may include a printed circuit board (PCB) (e.g., a first printed circuit board (331-1), a second printed circuit board (331-2)), a speaker for outputting an audio signal (e.g., a first speaker (332-1), a second speaker (332-2)), a battery (e.g., a first battery (333-1), a second battery (333-2)), and/or a hinge part (e.g., a first hinge part (340-1), a second hinge part (340-2)).

According to one embodiment, the printed circuit board (331-1, 331-2) may include a flexible board (e.g., FPCB, flexible PCB). The printed circuit board (331-1, 331-2) may transmit electrical signals to each component (e.g., a camera, a display module, a microphone, and/or a speaker) of the wearable electronic device (201).

According to one embodiment, the speakers (332-1, 332-2) may include a first speaker (332-1) for transmitting an audio signal to the user's left ear and a second speaker (332-2) for transmitting an audio signal to the user's right ear.

According to one embodiment, the batteries (333-1, 333-2) may supply power to the printed circuit board (331-1, 331-2) through a power management module (e.g., the power management module (188) of FIG. 1). According to one embodiment, the wearable electronic device (201) may have a plurality of batteries (333-1, 333-2) through the first support frame (260) and the second support frame (270), and may supply power to the printed circuit board (331-1, 331-2) through the power management module. For example, the plurality of batteries (333-1, 333-2) may be electrically connected to the power management module.

Although not illustrated, a wearable electronic device (201) such as the examples of FIGS. 2 and 3 may be stored by a designated external device (e.g., a case) (not illustrated). According to one embodiment, the case may include a function of simply storing and charging the wearable electronic device (201). According to some embodiments, in addition to the purpose of charging and storing the wearable electronic device (201), the case may include components such as a camera and/or a processor, and may be used as an auxiliary computing device of the wearable electronic device (201) by utilizing these components. For example, when the wearable electronic device (201) is stored in the case, the case may communicate with the case (e.g., wired communication and/or wireless communication), and the case may perform some functions of a host device (e.g., a smart phone).

According to one embodiment, the wearable electronic device (201) may provide an AR service by itself or in conjunction with at least one other electronic device (e.g., a host device (e.g., the electronic device (101)). For example, the wearable electronic device (201) may be connected to a host device (e.g., another electronic device (401) of FIG. 4A) and may operate in a tethered AR manner to provide an AR service by connecting to a network (e.g., a cloud) through the host device. According to some embodiments, the wearable electronic device (201) may also operate in a standalone manner to provide an AR service by connecting to a network (e.g., a cloud) without being connected to a host device (e.g., the electronic device (101)).

FIG. 4A is a diagram illustrating a network environment between an electronic device and another electronic device according to one embodiment.

Referring to FIG. 4A, an electronic device (101) according to one embodiment (e.g., the wearable electronic device (201) of FIG. 2 or FIG. 3) may be connected to another electronic device (401). The electronic device (101) and the other electronic device (401) may be connected by wire or connected wirelessly (e.g., paired). For example, the electronic device (101) may be connected to the other electronic device (401) via short-range wireless communication (403) such as Bluetooth, low-power Bluetooth, Wi-Fi, Wi-Fi direct, or ultra-wide band (UWB). According to one embodiment, the other electronic device (401) may include a smart phone, a tablet PC (personal computer), a notebook, and/or a display device (e.g., a monitor or a TV). According to one embodiment, the electronic device (101) may include AR glasses, smart glasses, or a head mounted display (HMD).

According to one embodiment, the electronic device (101) may directly generate related data (e.g., AR images) for AR service (e.g., generate based on stored or processed data) or obtain the data from an external source (e.g., another electronic device (401) or a server (e.g., server (108) of FIG. 1)) and display the data through a display (e.g., display module (160) of FIG. 1). For example, the electronic device (101) may be worn on a user's body (e.g., face) and may display various digital contents (e.g., AR images) overlapping the real world as a single image (e.g., AR screen) through the display module (160). According to one embodiment, the electronic device (101) may receive image data (e.g., AR images) from another electronic device (401) and display the received image data together with various objects of the real world through the display module (160).

According to one embodiment, when the electronic device (101) is in communication connection with another electronic device (401), the electronic device (101) may periodically transmit image information captured by a camera module (e.g., the camera module (180) of FIG. 1) of the electronic device (101), and user's line of sight information (e.g., FOV and/or AOV) to the other electronic device (401), and/or when a change in the state of the electronic device (101) (e.g., a change in position or direction) occurs, the electronic device (101) may transmit the information to the other electronic device (101). According to one embodiment, when the electronic device (101) is connected to another electronic device (401), the electronic device (101) may provide (e.g., transmit) at least one piece of information, such as image information, line of sight information, device information, sensing information, function information, and/or location information, to the other electronic device (401).

FIG. 4b is a diagram illustrating an example of displaying a virtual environment in an electronic device according to one embodiment.

In one embodiment, FIG. 4B may represent a state in which a user wears an electronic device (101) and views an AR screen provided through a display module (160) of the electronic device (101). For example, FIG. 4B may represent a state in which a user views content (410) (or a virtual object or an augmented reality image) superimposed on at least a portion of a real space (400) (or a virtual environment) through the AR screen of the electronic device (101). According to one embodiment, the content (410) may include a virtual object shown on a field of view (FOV) (430) of the user. In one embodiment, the content (410) (or the augmented reality image) may be graphic data corresponding to at least one application provided as a virtual object in a real real space (400) in which a user wearing the electronic device (101) is located.

Referring to FIG. 4b, the electronic device (101) may be worn by a user and may display one or more contents (410) (or virtual objects (or information) related to the contents) on at least a part of a real space (400) through a display module (160) of the electronic device (101). For example, the electronic device (101) may provide at least one virtual information to the user by overlapping the real real space (400) corresponding to the user's field of view (430). In one embodiment, a situation in which content (410) is displayed may include, for example, a situation in which a user performs a gesture input and/or an eye gaze on a real object in a real space (400), a situation in which an execution screen of an application (e.g., a browser, a shopping app, and/or an SNS) capable of performing a search function is displayed, a situation in which an execution screen of an application (e.g., a player) for playing media content is displayed (e.g., a video playback screen or an image playback screen), and/or a situation in which a user executes a specified simulation in a virtual space (e.g., a collaboration tool usage situation or a game play situation). For example, the content (410) may include information that can be provided by each corresponding application, such as virtual information (or a virtual object or a graphic image) corresponding to a first application, virtual information corresponding to a second application, and/or virtual information corresponding to a third application.

According to one embodiment, the electronic device (101) may track the user's gaze (430) while the content (410) is provided, and provide virtual information related to at least one real object in the real space (400) corresponding to the user's gaze (430) or an execution screen of an application executed by the user as an AR screen (or virtual environment or execution screen) through the display module (160). For example, the user may use the electronic device (101) to use an AR service by displaying content (410) corresponding to various real objects in the real space (400) according to the movement of the user's gaze (430).

FIG. 5 is a diagram schematically illustrating the configuration of an electronic device according to one embodiment.

According to one embodiment, FIG. 5 illustrates an example in which the electronic device (101) is a wearable electronic device in the form of glasses (e.g., the wearable electronic device (201) of FIG. 2 or FIG. 3), but is not limited thereto. According to one embodiment, the electronic device (101) may include components of the electronic device (101) and the wearable electronic device (201) as described in the description with reference to FIGS. 1 to 3. Hereinafter, the term wearable electronic device (101) may be used as a term including the electronic device (101) of FIG. 1 and the wearable electronic device (201) of FIG. 2 or FIG. 3.

Referring to FIG. 5, the wearable electronic device (101) may include a processor (120), a memory (130), a display module (160), an audio module (170), a sensor module (176), a camera module (180), a power management module (188), a battery (189), a communication module (190), an antenna module (197), and/or a charging antenna module (540). According to one embodiment, the wearable electronic device (101) may be connected to an external electronic device (not shown) through a connection terminal (530) (e.g., USB TYPE-C). For example, the power management module (188) of the wearable electronic device (101) may receive power from an external electronic device through the connection terminal (530) and charge the battery (189). For example, the processor (120) of the wearable electronic device (101) may perform power line communication with an external electronic device through the connection terminal (530). For example, the power management module (188) of the wearable electronic device (101) may wirelessly receive power from an external electronic device through the charging antenna module (540) to charge the battery (189). According to one embodiment, the wearable electronic device (101) may be composed of a main frame (or body) (e.g., the main frame (240) of FIG. 2 or FIG. 3) and a support frame (or leg member) (e.g., the first support frame (260) and/or the second support frame (270) of FIG. 2 or FIG. 3). According to one embodiment, components of the wearable electronic device (101) may be arranged in the main frame (240) and/or the support frames (260, 270).

According to one embodiment, the processor (120) may execute a program stored in the memory (130) to control at least one other component (e.g., hardware or software component) and perform various data processing or calculations. According to one embodiment, the processor (120) may provide an augmented reality service to a user. The processor (120) may output at least one virtual object so that the at least one virtual object is superimposed and displayed in a real space corresponding to a field of view of a user wearing the wearable electronic device (101) through the display module (160).

According to one embodiment, the memory (130) may store various data used by at least one component (e.g., the processor (120) or the sensor module (176)) of the wearable electronic device (101). The data may include, for example, software (e.g., the program (140) of FIG. 1) and input data or output data for commands related thereto. The memory (130) may include volatile memory (not shown) or non-volatile memory (not shown). The memory (130) may store instructions that cause the processor (120) to operate when executed. The instructions may be stored as software (e.g., the program (140) of FIG. 1) on the memory (130) and may be executable by the processor (120).

The display module (160) can visually provide information to an external (e.g., a user) portion of the wearable electronic device (101). The display module (160) can include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) of the wearable electronic device (101) can include at least one glass (e.g., the first glass (220) of FIG. 2 or FIG. 3) and/or the second glass (230)). According to one embodiment, the first glass (220) can include at least a portion of the first display module (551), and the second glass (230) can include at least a portion of the second display module (553). For example, the first display module (551) and/or the second display module (553) can each include a display panel. The display panel may be composed of a transparent element so that a user can perceive an actual space through the display module (160). The display module (160) may display at least one virtual object on at least a portion of the display panel so that a user wearing the wearable electronic device (101) sees the virtual object as being added to an actual space.

For example, the user's field of view may include an angle and/or range at which the user can recognize an object. According to one embodiment, the display module (160) may include a first display module (551) corresponding to the left eye of the user and/or a second display module (553) corresponding to the right eye. According to one embodiment, the processor (120) may load setting information (e.g., resolution, frame rate, size of the display area, and/or sharpness) related to the performance of the display module (160) from the memory (130), and adjust the performance of the display module (160) based on the setting information.

According to one embodiment, each display panel included in the display module (160) may have individually determined setting information. For example, a first display panel corresponding to the left eye may be set based on the first setting information, and a second display panel corresponding to the right eye may be set based on the second setting information. According to one embodiment, the setting information may set at least a portion of one display panel included in the display module (160) differently. For example, the wearable electronic device (101) may set at least one of the resolution, the frame rate, and/or the sharpness of the display module (160) differently. According to one embodiment, the wearable electronic device (101) may reduce power consumption by at least partially changing the setting of the display module (160).

According to one embodiment, the audio module (170) may convert sound into an electric signal or, conversely, convert an electric signal into sound based on the control of the processor (120). For example, the audio module (170) may include a speaker (332-1, 332-2) of FIG. 3 and/or a microphone (e.g., a first microphone (341-1), a second microphone (341-2), and a third microphone (341-3)) of FIG. 3.

The sensor module (176) can detect an operating state (e.g., power or temperature) of the wearable electronic device (101) or an external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) of the wearable electronic device (101) can include a proximity sensor (521), an illuminance sensor (522), and/or a gyro sensor (523).

According to one embodiment, the proximity sensor (521) can detect an object adjacent to the wearable electronic device (101).

According to one embodiment, the light sensor (522) can measure the brightness level around the wearable electronic device (101). According to one embodiment, the processor (120) can check the brightness level around the wearable electronic device (101) using the light sensor (522) and change brightness-related setting information of the display module (160) based on the brightness level. For example, if the brightness of the surroundings is brighter than a preset brightness, the processor (120) can set the brightness level of the display module (160) higher so as to increase the user's visibility.

In one embodiment, the gyro sensor (523) can detect the posture and position of the wearable electronic device (101). For example, the gyro sensor (523) can detect whether the wearable electronic device (101) is properly worn on the user's head. For example, the gyro sensor (523) can detect movement of the wearable electronic device (101) or a user wearing the wearable electronic device (101). For example, the gyro sensor (523) can detect movement of a support frame (e.g., the first support frame (260) of FIG. 2 or FIG. 3, and/or the second support frame (270)) of the wearable electronic device (101).

According to one embodiment, the sensor module (176) of the wearable electronic device (101) may include an acceleration sensor and/or a Hall sensor. According to one embodiment, the acceleration sensor may detect a movement of a support frame (e.g., the first support frame (260) of FIG. 2 or FIG. 3 and/or the second support frame (270)) of the wearable electronic device (101). According to one embodiment, the Hall sensor may detect a movement of a support frame (e.g., the first support frame (260) of FIG. 2 or FIG. 3 and/or the second support frame (270)) of the wearable electronic device (101) by detecting a change in a magnetic field.

The communication module (190) can support establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the wearable electronic device (101) and an external electronic device (e.g., another electronic device, a server, and/or a charging device), and performance of communication through the established communication channel. The communication module (190) can operate independently from the processor (120) (e.g., an application processor) and can include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the wearable electronic device (101) can perform wireless communication with another electronic device (e.g., a smart phone or a wearable electronic device) through the communication module (190) (e.g., a wireless communication circuit). For example, the wearable electronic device (101) can perform wireless communication with another electronic device and exchange commands and/or data with each other. According to one embodiment, the wearable electronic device (101) can be controlled, at least in part, by another external electronic device through the communication module (190). For example, the wearable electronic device (101) can perform at least one function under the control of another external electronic device. According to one embodiment, the wearable electronic device (101) can transmit object information (e.g., an image) located in an actual real space, distance information to an object, user's gaze information, and/or user's gesture information to an external device (e.g., another electronic device and/or a server) through the communication module (190) through the camera module (180).

The antenna module (197) can transmit or receive signals or power to or from the outside (e.g., an external electronic device). According to one embodiment, the antenna module (197) can include an antenna including a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) can include a plurality of antennas (e.g., an array antenna). A signal or power can be transmitted or received between the communication module (190) and the external electronic device through at least one selected antenna. According to one embodiment, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) can be additionally formed as a part of the antenna module (197).

According to one embodiment, the wearable electronic device (101) can change at least a part of the settings of the display panel based on the control of another electronic device connected wirelessly and/or wiredly. According to one embodiment, the wearable electronic device (101) can transmit dominant eye/auxiliary eye related information (e.g., distance information to an object located in a real space, gaze tracking information of a user, gesture information of a user) acquired through a camera (e.g., camera module (180)) of the wearable electronic device (101) to another electronic device.

According to one embodiment, the other electronic device may transmit setting information of a display panel included in a glass (e.g., the first glass (220) and/or the second glass (230)) corresponding to the detected dominant or auxiliary eye to the wearable electronic device (101) based on the dominant eye/auxiliary eye related information received from the wearable electronic device (101). According to one embodiment, the wearable electronic device (101) may change at least a part of the setting of the display panel based on the setting information of the display panel received from the other electronic device. For example, the setting of the display panel may be changed to lower the quality of the display panel, and at least a part of the setting may be changed to an extent that the user cannot perceive. According to one embodiment, the wearable electronic device (101) may reduce the resolution of the display panel, reduce the frame rate, or adjust the size and position of the display area of the display panel.

The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) can include one or more lenses, image sensors, image signal processors, or flashes. According to one embodiment, the camera module (180) of the wearable electronic device (101) can include a gesture camera (511), an eye tracking camera (513), a depth camera (515), and/or an RGB camera (517).

According to one embodiment, the gesture camera (511) can detect the user's movement. The recognition cameras (311-1, 311-2) of FIG. 3 may include the gesture camera (511). For example, at least one gesture camera (511) may be placed on the wearable electronic device (101) and may detect the user's hand movement within a preset distance. The gesture camera (511) may include a SLAM camera (simultaneous localization and mapping camera) for recognizing information (e.g., location and/or direction) related to the surrounding space of the wearable electronic device (101). The gesture recognition area of the gesture camera (511) may be set based on the shooting range of the gesture camera (511).

According to one embodiment, the gaze tracking camera (513) (e.g., the first gaze tracking camera (312-1) and the second gaze tracking camera (312-2) of FIG. 3) can track movements of the user's left and right eyes. According to one embodiment, the processor (120) can use the gaze tracking camera (513) to identify the gaze direction of the left eye and the gaze direction of the right eye. For example, the gaze tracking camera (513) can include a first gaze tracking camera (312-1) for identifying the gaze direction of the left eye and a second gaze tracking camera (312-2) for identifying the gaze direction of the right eye. According to one embodiment, the processor (120) can determine the dominant eye and the auxiliary eye based on the gaze direction of the left eye and the gaze direction of the right eye.

According to one embodiment, the distance measuring camera (515) can measure the distance to an object located in front of the wearable electronic device (101). The photographing camera (280) of FIG. 2 or FIG. 3 can include the distance measuring camera (515). The distance measuring camera (515) can include a TOF (time of flight) camera and/or a depth camera. According to one embodiment, the distance measuring camera (515) can capture a front direction of the wearable electronic device (101), and the gaze tracking camera (513) can capture a direction opposite to the shooting direction of the distance measuring camera (515).

According to one embodiment, the wearable electronic device (101) can measure a distance to an object using a distance measuring camera (515), and change a setting of the display panel when the distance is equal to or greater than a threshold value. For example, the wearable electronic device (101) can maintain the display performance of the display panel when the distance to the object is less than or equal to a threshold value. According to one embodiment, the wearable electronic device (101) can recognize one of the objects located in the direction of the user's gaze (e.g., FOV) using the gaze tracking camera (513), and calculate the depth of the distance to the object using a depth camera, or measure the distance to the object using a TOF camera.

According to one embodiment, the RGB (red green blue) camera (517) can detect color-related information of an object and distance information to the object. According to one embodiment, the wearable electronic device (101) can include one type of camera by integrating the distance measuring camera (515) and the RGB camera (517). For example, the shooting camera (280) of FIG. 2 or FIG. 3 can include the distance measuring camera (515) and/or the RGB camera (517).

According to one embodiment, the gesture camera (511), the gaze tracking camera (513), the distance measuring camera (515), and/or the RGB camera (517) included in the camera module (180) may each be included in the wearable electronic device (101) or some of them may be implemented as integrated cameras. For example, the distance measuring camera (515) and the RGB camera (517) may be implemented as a single integrated camera.

According to one embodiment, the power management module (188) can manage power supplied to the wearable electronic device (101). The power management module (188) can include a plurality of power management modules (e.g., a first power management module (531), a second power management module (532)). At least some of the first power management module (531) or the second power management module (532) can be directly connected to the processor (120) to supply power. At least some of the first power management module (531) or the second power management module (532) can receive power from an external electronic device through a connection terminal (530) (e.g., TYPE-C) to charge the battery (189) or supply power to other components of the wearable electronic device (101). According to one embodiment, the wearable electronic device (101) can receive power from an external electronic device through a wireless charging method to charge the battery (188). According to one embodiment, the wearable electronic device (101) can wirelessly receive external power from a charging antenna module (540).

According to one embodiment, the power management module (188) may be electrically connected to components of the wearable electronic device (101) (e.g., memory (130), display module (160), audio module (170), sensor module (176), camera module (180), and/or communication module (190)). For example, the power management module (188) may provide power of the battery (189) to the components of the wearable electronic device (101) based on the control of the processor (120). According to one embodiment, the wearable electronic device (101) may receive power from a first battery (533) through the first power management module (531) and may receive power from a second battery (534) through the second power management module (532). According to one embodiment, the processor (120) can manage power consumption by at least partially changing the settings of the display module (160) based on information acquired using at least one camera (511, 513, 515, 517) included in the camera module (180).

The battery (189) can supply power to at least one component of the wearable electronic device (101). In one embodiment, the battery (189) can include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell. In one embodiment, the battery (189) can be supplied with power and charged or provided with power and discharged under the control of the power management module (188). In one embodiment, the battery (189) can include a plurality of batteries (e.g., a first battery (533), a second battery (534)). For example, the plurality of batteries (e.g., the first battery (533), the second battery (534)) can be arranged in a main frame (e.g., a main frame (240) of FIG. 2 or 3) and a support frame (e.g., a first support frame (260) and/or a second support frame (270) of FIG. 2 or 3). According to one embodiment, the first battery (533) may be placed in the first support frame (260), and the second battery (534) may be placed in the second support frame (270).

The charging antenna module (540) can receive power from the outside of the wearable electronic device (101). According to one embodiment, the charging antenna module (540) can include at least some of the configurations and/or functions of the antenna module (197). According to one embodiment, the charging antenna module (540) can include a loop-type antenna and/or a coil-type antenna. According to one embodiment, the charging antenna module (540) can include a plurality of antennas (e.g., a first antenna (541), a second antenna (542)). For example, the plurality of antennas (e.g., the first antenna (541), the second antenna (542)) can receive a magnetic field and/or an electromagnetic field from the outside, and can transmit a current generated through magnetic induction and/or magnetic resonance due to the external magnetic field and/or electromagnetic field to the power management module (188) and/or the battery (189). According to one embodiment, a plurality of antennas (e.g., a first antenna (541), a second antenna (542)) may be disposed on a main frame (e.g., a main frame (240) of FIG. 2 or FIG. 3) and a support frame (e.g., a first support frame (260) of FIG. 2 or FIG. 3, and/or a second support frame (270)). According to one embodiment, the first antenna (541) may be disposed on the first support frame (260), and the second antenna (542) may be disposed on the second support frame (270).

According to one embodiment of the present disclosure, the processor (120) can perform an application layer processing function requested by a user of the wearable electronic device (101). According to one embodiment, the processor (120) can provide control and commands of functions for various blocks of the wearable electronic device (101). According to one embodiment, the processor (120) can perform operations or data processing related to control and/or communication of each component of the wearable electronic device (101). For example, the processor (120) can include at least some of the configurations and/or functions of the processor (120) of FIG. 1. According to one embodiment, the processor (120) can be operatively connected with the components of the wearable electronic device (101). According to one embodiment, the processor (120) may load a command or data received from another component of the wearable electronic device (101) into the memory (130), process the command or data stored in the memory (130), and store result data.

According to one embodiment, the processor (120) may include processing circuitry and/or executable program elements. According to one embodiment, the processor (120) may control (or process) the overall operations related to supporting image capturing in the electronic device (101) based on the processing circuitry and/or executable program elements.

According to one embodiment, the processor (120) may perform an operation of recognizing an input device in a virtual environment. According to one embodiment, the processor (120) may perform an operation of determining a virtual space for determining a target virtual object based on a property of the input device. According to one embodiment, the processor (120) may perform an operation of determining a target virtual object based on the virtual space. According to one embodiment, the processor (120) may perform an operation of generating an affordance corresponding to each target virtual object. According to one embodiment, the processor (120) may perform an operation of displaying the affordance according to a specified arrangement method.

According to one embodiment, the processor (120) may perform an operation of displaying an execution screen for a virtual environment including a plurality of virtual objects through a display. According to one embodiment, the processor (120) may perform an operation of recognizing an input device while displaying the execution screen. According to one embodiment, the processor (120) may perform an operation of recognizing an indicator corresponding to the input device based on recognizing the input device. According to one embodiment, the processor (120) may perform an operation of determining an attribute of the input device based on recognizing the indicator.

According to one embodiment, the processor (120) may perform an operation of calculating a virtual space corresponding to a property of an input device based on an indicator. According to one embodiment, the processor (120) may perform an operation of determining a target virtual object included in the virtual space among a plurality of virtual objects of an execution screen based on the virtual space. According to one embodiment, the processor (120) may perform an operation of generating an affordance based on the target virtual object included in the virtual space. According to one embodiment, the processor (120) may perform an operation of displaying the affordance in association with the target virtual object.

According to one embodiment, the detailed operation of the processor (120) of the electronic device (101) is described with reference to the drawings below.

According to one embodiment, the operations performed by the processor (120) may be implemented as a recording medium (or a computer program product). For example, the recording medium may include a non-transitory computer-readable recording medium having recorded thereon a program for executing various operations performed by the processor (120).

The embodiments described in the present disclosure can be implemented in a computer-readable recording medium using software, hardware, or a combination thereof. In a hardware implementation, the operations described in one embodiment can be implemented using at least one of application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, micro-controllers, microprocessors, and/or other electrical units for performing functions.

In one embodiment, the recording medium (or the computer program product) may include a computer-readable recording medium having recorded thereon a program that executes operations for recognizing an input device in a virtual environment, determining a virtual space for determining a target virtual object based on a property of the input device, judging the target virtual object based on the virtual space, generating an affordance corresponding to each of the target virtual objects, and displaying the affordance according to a specified arrangement method.

In one embodiment, the recording medium (or the computer program product) may include a computer-readable recording medium storing a program for executing the following operations: displaying an execution screen for a virtual environment including a plurality of virtual objects through a display; recognizing an input device while displaying the execution screen; recognizing an indicator corresponding to the input device based on the recognition of the input device; determining a property of the input device based on the recognition of the indicator; calculating a virtual space corresponding to the property of the input device based on the indicator; determining a target virtual object included in the virtual space among the plurality of virtual objects of the execution screen based on the virtual space; generating an affordance based on the target virtual object included in the virtual space; and displaying the affordance in association with the target virtual object.

An electronic device (101, 201) according to one embodiment of the present disclosure may include a display (160), a camera (180), and a processor (120) operatively connected to the display (160) and the camera (180).

According to one embodiment, the processor (120) may be operable to recognize an input device (720) in a virtual environment (400, 700). According to one embodiment, the processor (120) may be operable to determine a virtual space (740) for determining a target virtual object (750) based on an attribute of the input device (720). According to one embodiment, the processor (120) may be operable to determine the target virtual object (750) based on the virtual space (740). According to one embodiment, the processor (120) may be operable to generate an affordance (760) corresponding to each of the target virtual objects (750). According to one embodiment, the processor (120) may be operable to display the affordance (760) according to a specified arrangement method.

According to one embodiment, the processor (120) may be operable to display an execution screen for the virtual environment including a plurality of virtual objects through the display. According to one embodiment, the processor (120) may be operable to recognize the input device while displaying the execution screen. According to one embodiment, the processor (120) may be operable to recognize an indicator (900) corresponding to the input device based on recognizing the input device.

According to one embodiment, the input device may include a device that supports user input for manipulating a virtual object in the virtual environment.

In one embodiment, the affordance may include a designated object that supports direct selection of the target virtual object among the plurality of virtual objects.

According to one embodiment, the processor (120) may be operable to determine a property of the input device and to calculate a virtual space corresponding to the property of the input device based on the indicator.

According to one embodiment, the virtual space may include a virtual sphere formed with a diameter having a resolution value based on a property of the input device centered on the indicator.

According to one embodiment, the resolution value may include a first resolution value predefined for the input device. According to one embodiment, the resolution value may include a second resolution value related to user noise generated by a user using the input device.

According to one embodiment, the processor (120) may be operable to calculate the resolution value based on the first resolution value and the second resolution value.

According to one embodiment, the processor (120) may operate to map the center of gravity of the virtual space based on the indicator and form the virtual space having the diameter based on the center of gravity.

According to one embodiment, the processor (120) may be operable to determine the size and/or spacing of the affordance based on the resolution value.

According to one embodiment, the processor (120) may operate to determine, based on the virtual space, the target virtual object included in the virtual space among the plurality of virtual objects of the execution screen.

According to one embodiment, the processor (120) may operate to determine whether the target virtual object included in the virtual space satisfies a specified condition. According to one embodiment, the processor (120) may operate to enter a guide generation mode for providing the affordance when the target virtual object satisfies the specified condition.

According to one embodiment, the specified condition may include a condition in which the target virtual object is a plurality of target virtual objects, at least two in number, and a distance between centers of gravity of each of the plurality of target virtual objects is less than or equal to a specified proximity distance.

According to one embodiment, the processor (120) may be operable to determine whether a plurality of target virtual objects are included in the virtual space. According to one embodiment, when the plurality of target virtual objects are included, the processor (120) may be operable to determine a center of gravity corresponding to each of the plurality of target virtual objects. According to one embodiment, the processor (120) may be operable to determine a distance between centers of gravity of each of the plurality of target virtual objects. According to one embodiment, the processor (120) may be operable to compare the distance between the centers of gravity with a specified proximity distance. According to one embodiment, when at least one of the distances between the centers of gravity is less than or equal to the specified proximity distance, the processor (120) may be operable to enter the guide generation mode.

According to one embodiment, the specified arrangement method may include a first method of providing an affordance by arranging it in a space in contact with the target virtual object and the surface of the virtual space. According to one embodiment, the specified arrangement method may include a second method of providing an affordance by arranging it in a list corresponding to each of the target virtual objects outside the virtual space.

According to one embodiment, the processor (120) may be operable to provide an object-based affordance based on a space where the target virtual object contacts a surface of the virtual space when the target virtual object spans the virtual space. According to one embodiment, the processor (120) may be operable to provide an object-based affordance based on a space along an extension line from the target virtual object to the surface of the virtual space when the target virtual object is inside the virtual space. According to one embodiment, the processor (120) may be operable to provide a list-based affordance to an outer space of the virtual space when a distance between the target virtual objects is smaller than a radius of the virtual space.

Hereinafter, the operation method of the electronic device (101) according to various embodiments will be described in detail. The operations performed in the electronic device (101) according to various embodiments may be executed by the processor (120) including various processing circuitry and/or executable program elements of the electronic device (101). According to one embodiment, the operations performed in the electronic device (101) may be stored in the memory (130) and executed by instructions that cause the processor (120) to operate when executed.

FIG. 6 is a flowchart illustrating a method of operating an electronic device according to one embodiment of the present disclosure.

According to one embodiment, FIG. 6 may illustrate an example of an operation in which an electronic device (101) according to one embodiment may support selection of a virtual object in a virtual environment without repositioning or enlarging the virtual object, so that the immersion of the user experience is not disturbed.

The operation method supported by the electronic device (101) according to one embodiment of the present disclosure may be performed according to, for example, the flowchart illustrated in FIG. 6. The flowchart illustrated in FIG. 6 is merely a flowchart according to one embodiment of the operation of the electronic device (101), and the order of at least some operations may be changed or performed in parallel, performed as independent operations, or at least some other operations may be performed complementarily to at least some operations. According to one embodiment of the present disclosure, operations 601 to 609 may be performed by at least one processor (120) of the electronic device (101) (e.g., the processor (120) of FIG. 1 or FIG. 5 ).

As illustrated in FIG. 6, an operation method performed by an electronic device (101) according to one embodiment may include an operation (601) of recognizing an input device in a virtual environment, an operation (603) of determining a virtual space for determining a target virtual object based on a property of the input device, an operation (605) of determining a target virtual object based on the virtual space, an operation (607) of generating an affordance corresponding to each target virtual object, and an operation (609) of displaying the affordance according to a specified arrangement method.

Referring to FIG. 6, in operation 601, the processor (120) of the electronic device (101) may perform an operation of recognizing an input device in a virtual environment. According to one embodiment, the processor (120) may display an execution screen (e.g., an AR screen or a VR screen) corresponding to the virtual environment through a user's display (e.g., the display module (160) of FIG. 1 or FIG. 5). According to one embodiment, the processor (120) may provide content overlapping at least a part of the real world (or the real world) or the virtual world through the execution screen. According to one embodiment, the content may include a plurality of virtual objects shown on the user's field of view. In one embodiment, the content may be graphic data corresponding to an application executed through the display of the electronic device (101) provided as a virtual object.

According to one embodiment, the processor (120) may recognize an input device while displaying an execution screen corresponding to a virtual environment. In one embodiment, the input device may include various devices (or means) for supporting a user input for manipulating content (e.g., a virtual object) in a virtual environment. According to one embodiment, the input device is connected to the electronic device (101) through a direct (e.g., wired) communication channel or wireless communication, and the connected input device may be recognized by the processor (120). According to one embodiment, the input device may enter an execution screen displayed through a display, and may be recognized (e.g., vision-based recognition) by the processor (120).

According to one embodiment, the input device may include a hardware device such as a mouse, a smart pen (or an electronic pen), a controller, a wearable device (e.g., a ring (e.g., a smart ring) and/or a watch) that is connected to the electronic device (101) in a designated communication manner. According to one embodiment, the input device may include an object (or object device) such as a user's hand (or finger), a stylus pen, a wearable device (e.g., a smart ring and/or a watch), and/or a glove that is recognized through an execution screen in a virtual environment. According to one embodiment, the input device may include a plurality of cameras (e.g., the eye tracking cameras 312-1 and 312-2 of FIG. 3). For example, the electronic device (101) may track the user's eye using the plurality of cameras, and may recognize the position and/or movement of the user's eye as an indicator corresponding to the input device.

In operation 603, the processor (120) may perform an operation of determining a virtual space for determining a target virtual object based on the properties of the input device. According to one embodiment, the processor (120) may determine a property (or type) corresponding to the input device based on recognizing the input device. According to one embodiment, the processor (120) may determine whether the recognized input device corresponds to a user's hand, a wearable device, a controller, a glove, a smart pen, a stylus pen, a mouse, and/or a camera for eye tracking. According to one embodiment, the processor (120) may determine a property of the input device, and determine a size (or range) of a virtual space (or a virtual sphere or a resolution sphere) based on the properties of the input device. According to one embodiment, the virtual space may include a virtual sphere (or a resolution sphere) formed with a diameter of a resolution value based on the properties of the input device centered on the input device (e.g., an indicator of the input device). According to one embodiment, the virtual space may be a space virtually (or internally) created to determine a target virtual object to be selected by the user based on the location of the input device among a plurality of virtual objects on the execution screen. For example, the virtual space may not be displayed on the execution screen.

In operation 605, the processor (120) may perform an operation of determining a target virtual object based on a virtual space. According to one embodiment, the processor (120) may extract (or infer) a virtual object included in a virtual space formed based on an input device among a plurality of virtual objects on an execution screen as a target virtual object. According to one embodiment, the target virtual object may include a plurality of virtual objects (e.g., at least two virtual objects).

In operation 607, the processor (120) may perform an operation of generating an affordance corresponding to each target virtual object. According to one embodiment, the processor (120) may generate an affordance corresponding to each target virtual object based on determining the target virtual object based on the virtual space. According to one embodiment, the affordance may include a designated object (e.g., a handle, an icon, an image, or a list) that supports direct selection of a target virtual object among a plurality of virtual objects on an execution screen. According to one embodiment, the affordance may be provided as graphic data.

In operation 609, the processor (120) may perform an operation of displaying the affordance according to a specified arrangement method. According to one embodiment, the specified arrangement method may include an object-based arrangement method (e.g., a first method) that provides an affordance by arranging it in a space in contact with a surface of a virtual space and a target virtual object, and a list-based arrangement method (e.g., a second method) that provides an affordance by arranging it in a list corresponding to each target virtual object outside the virtual space. According to one embodiment, an operation example of providing an affordance according to a specified arrangement method will be described with reference to the drawings below.

FIGS. 7A, 7B, 7C, and 7D are diagrams illustrating examples of operations of providing affordance in association with a virtual object in an electronic device according to one embodiment of the present disclosure.

According to one embodiment, FIGS. 7A, 7B, 7C, and 7D may illustrate examples of operations for providing an affordance that supports selection of a virtual object in association with a virtual object in a virtual environment in an electronic device (101) according to one embodiment.

Referring to FIGS. 7A to 7D, the example screen <701> may represent a state in which the electronic device (101) displays a virtual environment (700) (or an execution screen (700) of the virtual environment) through a display (e.g., a display module (160) of FIG. 1 or FIG. 5). As exemplified in the example screen <701>, the electronic device (101) may display a plurality of virtual objects (710) through the virtual environment (700).

Example screens <703> and <705> can illustrate examples of situations in which an input device (720) is recognized while displaying a virtual environment (700). According to one embodiment, in FIGS. 7B to 7D , the input device (720) is an example of a user's hand (e.g., a finger), and the electronic device (101) can recognize the user's hand (720) entering the virtual environment (700).

Example screen <705> may represent a state in which an input device (720) recognized in a virtual environment (700) is stopped at a certain location in the virtual environment (700) for a specified period of time (730). For example, a user may move the input device (720) to a space (or location) where a virtual object to be selected from among a plurality of virtual objects (710) in the virtual environment (700) exists and stop (or hold) the input device (720) for a specified period of time (e.g., about N seconds, where about N seconds is a time period in which holding of the input device (720) is recognizable). According to one embodiment, the electronic device (101) may detect a trigger for initiating an operation to select a virtual object based on the input device (720) when the input device (720) is stopped for a specified period of time in the virtual environment (700). According to one embodiment, the electronic device (101) may further perform an operation of determining an indicator (e.g., a pointing point) based on the input device (720).

Example screen <707> may represent an example of forming a virtual space (740) based on an input device (720). According to one embodiment, the virtual space (740) may include a virtual sphere (or resolution sphere) formed with a diameter of a resolution value based on an attribute of the input device (720) centered on the input device (720) (e.g., an indicator of the input device (720)). According to one embodiment, the virtual space (740) may be a space virtually (or internally) created to determine a target virtual object (e.g., a virtual object (750)) to be selected by the user based on the location of the input device (720) among a plurality of virtual objects (710) on a virtual environment (700). For example, the virtual space (740) may not be displayed on a display (or virtual environment (700)). According to one embodiment, the electronic device (101) can search for a virtual object (750) included in a virtual space (740) around an input device (720) (e.g., an indicator of the input device (720)) among a plurality of virtual objects (710).

The example screen <709> may show an example of a searched virtual object (750). For example, the electronic device (101) may determine a target virtual object (e.g., a virtual object (750) included in the virtual space (740)) for generating an affordance (760) (e.g., a handle) based on an input device (720) (e.g., an indicator of the input device (720)) among a plurality of virtual objects (710) based on the virtual space (740). In one embodiment, the example screen <709> may show an example in which four virtual objects (750) are extracted (or inferred) as target virtual objects for convenience of explanation. For example, in the example screen <709>, the four virtual objects (750) are shown with emphasis (e.g., highlighted) to be distinguished from other virtual objects, but this may be for convenience of explanation. For example, the target virtual object (750) may be provided with a graphical effect to distinguish it from the user's intuitiveness, or may be provided without any change and without a graphical effect.

Example screen <711> may represent an example of providing an affordance (760) in association with a target virtual object (750). According to one embodiment, the electronic device (101) may provide an affordance (760) corresponding to each target virtual object (750) by arranging it around the input device (720) (e.g., virtual space (740)). For example, the electronic device (101) may generate and provide four affordances (760), such as a first affordance (or first handle), a second affordance (or second handle), a third affordance (or third handle), and a fourth affordance (or fourth handle), corresponding to four virtual objects (750), such as a first virtual object, a second virtual object, a third virtual object, and a fourth virtual object, respectively.

Example screen <713> may illustrate an example in which an input device (720) (e.g., an indicator of the input device (720)) moves (e.g., changes position). For example, a user may move the input device (720) outside the provided area of the affordance (760) (e.g., outside the virtual space (740)). For example, the user may move (or change) the position of the input device (720) to select another virtual object among multiple virtual objects (710), perform another operation using the input device (720), or release the affordance (760).

Example screen <715> may represent an example of a state in which an affordance (760) is removed and not displayed on a virtual environment (700) according to movement of an input device (720). According to one embodiment, when the electronic device (710) detects movement of the input device (720), it may immediately remove the affordance (760) or maintain a state of displaying the affordance (760) for a specified period of time. According to one embodiment, the electronic device (101) may display the affordance (760) on the virtual environment (700) until another target virtual object is determined, and when another target virtual object is determined, the existing affordance (760) may disappear and another affordance may be generated and provided in association with another target virtual object.

FIG. 8 is a flowchart illustrating a method of operating an electronic device according to one embodiment of the present disclosure.

FIG. 9 is a diagram illustrating examples of indicators for various input devices in an electronic device according to one embodiment of the present disclosure.

FIG. 10 is a diagram illustrating an example of a virtual space generated by an electronic device and an indicator corresponding to an input device according to one embodiment of the present disclosure.

According to one embodiment, FIG. 8 may illustrate an example of an operation of providing an affordance that supports selection of a virtual object in association with a virtual object in a virtual environment in an electronic device (101) according to one embodiment.

The operation method supported by the electronic device (101) according to one embodiment of the present disclosure may be performed according to, for example, the flowchart illustrated in FIG. 8. The flowchart illustrated in FIG. 8 is merely a flowchart according to one embodiment of the operation of the electronic device (101), and the order of at least some operations may be changed or performed in parallel, performed as independent operations, or at least some other operations may be performed complementarily to at least some operations. According to one embodiment of the present disclosure, operations 801 to 815 may be performed by at least one processor (120) of the electronic device (101) (e.g., the processor (120) of FIG. 1 or FIG. 5 ).

According to one embodiment, the operations described in FIG. 8 may be performed heuristically, for example, in combination with the operations described in FIGS. 6 to 7d , or heuristically performed as a replacement for at least some of the operations described and combined with at least some other operations, or heuristically performed as a detailed operation of at least some of the operations described.

As illustrated in FIG. 8, an operation method performed by an electronic device (101) according to an embodiment may include an operation (801) of displaying an execution screen for a virtual environment including a plurality of virtual objects through a display, an operation (803) of recognizing an input device while displaying the execution screen, an operation (805) of recognizing an indicator corresponding to the input device, an operation (807) of determining a property of the input device, an operation (809) of calculating a virtual space corresponding to the property of the input device based on the indicator, an operation (811) of determining a target virtual object included in the virtual space among the plurality of virtual objects of the execution screen based on the virtual space, an operation (813) of generating an affordance based on the target virtual object included in the virtual space, and an operation (815) of displaying the affordance in association with the target virtual object.

Referring to FIG. 8, in operation 801, the processor (120) of the electronic device (101) may perform an operation of displaying an execution screen for a virtual environment including a plurality of virtual objects through a display. According to one embodiment, the processor (120) may display an execution screen (e.g., an AR screen or a VR screen) corresponding to the virtual environment through a user's display (e.g., the display module (160) of FIG. 1 or FIG. 5). According to one embodiment, the processor (120) may provide content (e.g., a plurality of virtual objects) overlapping at least a part of the real world (or the real world) or the virtual world through the execution screen. For example, the processor (120) may display an execution screen including a plurality of virtual objects (710) through the display, such as the example screen <701> of FIG. 7a.

In operation 803, the processor (120) may perform an operation of recognizing an input device while displaying an execution screen. In one embodiment, the input device may include various devices (or means) for supporting a user input for controlling an operation of content (e.g., a virtual object) in a virtual environment. According to one embodiment, when the processor (120) detects an operation (e.g., a pointer movement) in a virtual environment by an input device connected to the electronic device (101), the processor (120) may recognize the input device connected to the electronic device (101). According to one embodiment, when the processor (120) detects the entry of the input device onto an execution screen displayed through the display, the processor (120) may recognize the input device that has entered the execution screen. Examples of this are illustrated in example screen <703> of FIG. 7A and example screen <705> of FIG. 7B.

In operation 805, the processor (120) may perform an operation of recognizing an indicator corresponding to the input device based on recognizing the input device. According to one embodiment, the processor (120) may recognize (or determine) the indicator (900) based on the input shape of the input device (720) and/or the input coordinates of the input device (720). For example, the processor (120) may cognitively infer the indicator (900) based on the input shape for a vision-based input device (e.g., a user's hand, a glove), may recognize the coordinates of the input device (720) as the indicator (900) for an input device having sensor-based coordinates (e.g., a controller, a smart pen, a mouse), and may recognize the position and/or movement of the user's gaze as the indicator (900) for an eye-tracking-based input device (e.g., the eye-tracking recommendation cameras (312-1, 312-2) of FIG. 3). According to one embodiment, for an input device based on eye tracking, for example, a function such as a mouse cursor may be performed based on the position and/or movement of the user's gaze, and the position where the user's gaze remains may be recognized as an indicator (900).

Referring to FIG. 9, examples 901, 903, and 905 illustrate a case where the input device (720) is a user's hand, example 907 illustrates a case where the input device (720) is a controller, example 909 illustrates a case where the input device (720) is a smart pen, example 911 illustrates a case where the input device (720) is a mouse, and example 913 illustrates a case where the input device (720) is a smart ring. In one embodiment, example 913 illustrates and describes a ring-type wearable device (e.g., a smart ring) worn on a user's finger, but is not limited thereto. For example, an exemplary input device (720) of the present disclosure may include a bracelet-type wearable device or a watch-type wearable device.

In example 901, the processor (120) may vision-recognize an index finger as an input device (720), and recognize a tip (e.g., fingertip) (or a position of the tip) of the index finger as an indicator (900). In example 903, the processor (120) may vision-recognize a thumb and an index finger as input devices (720), and recognize an intermediate position between a first tip of the thumb and a second tip of the index finger as a position of the indicator (900). In example 905, the processor (120) may vision-recognize an entire finger as an input device (720), and recognize an intermediate position between the respective tips of five fingers as a position of the indicator (900).

In example 907, the processor (120) may recognize a controller connected to the electronic device (101) as an input device (720), and recognize coordinates of a pointing point indicated by the controller as the location of the indicator (900). In example 909, the processor (120) may recognize a smart pen connected to the electronic device (101) as an input device (720), and recognize coordinates of a pointing point indicated by the smart pen (or tip coordinates of the smart pen) as the location of the indicator (900). In example 911, the processor (120) may recognize a mouse connected to the electronic device (101) as an input device (720), and recognize pointer coordinates of the mouse as the location of the indicator (900).

In example 913, the processor (120) may recognize a smart ring connected to the electronic device (101) as an input device (720), and recognize the location of the smart ring as the location of the indicator (900). For example, the smart ring may include various components such as a motion sensor that detects movement of the smart ring, a communication circuit for connecting with the electronic device (101), and a biometric sensor (e.g., a PPG sensor) that may detect biometric information of the user. According to one embodiment, when the input device (720) is a smart ring (or a smart bracelet or a smart watch), the electronic device (101) may be connected to the smart ring through a designated wireless communication (e.g., a short-range communication such as UWB communication or Bluetooth communication). According to one embodiment, the processor (120) may recognize (or search for) the exact location of the smart ring through the designated wireless communication, and designate the location of the smart ring as the location of the indicator (900).

According to one embodiment, the processor (120) may determine the position of the smart ring by using vision recognition, or may determine the position of the smart ring by using vision recognition and motion sensing information obtained from the smart ring via wireless communication together. For example, the smart ring may include a light-emitting element, such as an LED, so that the electronic device (101) can easily recognize the vision of the smart ring. According to one embodiment, the electronic device (101) may determine the initial position of the smart ring by vision recognition of a light-emitting point on the smart ring, and thereafter, may receive motion sensing information from the smart ring to estimate movement from the initial position.

According to one embodiment, the processor (120) can use the position of the smart ring or its change as a function of the indicator (900). For example, the processor (120) can perform an interaction for manipulating the smart ring and content (e.g., a virtual object). For example, the processor (120) can determine a target virtual object based on the smart ring and provide an affordance corresponding to each target virtual object according to a specified arrangement method.

According to one embodiment, the processor (120) can monitor in real time the location of the smart ring or the change in the location of the smart ring (e.g., movement) based on the designated wireless communication and/or vision recognition, and select the virtual object at the location of the smart ring based on the smart ring being stopped on the virtual object for a certain period of time or receiving a selection signal (e.g., a user input using the smart ring (e.g., a designated touch input, a designated gesture input, or a muscle tension change input)) from the smart ring on the virtual object. According to one embodiment, the processor (120) can detect a change of the smart ring (e.g., movement information based on movement or a designated gesture) while the virtual object is selected, and move the virtual object in response to the change of the smart ring. For example, the processor (120) can operate to recognize the location of the smart ring as an indicator (900) in conjunction with the smart ring and manipulate the virtual object.

According to one embodiment, the processor (120) may utilize the position of the user's gaze or a change thereof as a function of the indicator (900). For example, the processor (120) may perform an interaction for manipulating content (e.g., a virtual object) based on the user's gaze. For example, the processor (120) may determine a target virtual object based on at least one virtual object at the position of the user's gaze through gaze tracking, and provide affordances corresponding to each target virtual object according to a specified arrangement method.

According to one embodiment, the processor (120) can monitor in real time the position or position change (e.g., movement) of the user's gaze based on gaze tracking, and select the virtual object at the position of the user's gaze based on whether the user's gaze stops on the virtual object for a certain period of time or whether the virtual object receives (or detects) a selection signal (e.g., detection of the user's eye blink or user input (e.g., selection gesture input)) from gaze tracking. According to one embodiment, the processor (120) can detect a change in the user's gaze (e.g., movement information based on movement or a designated gesture) while the virtual object is selected, and move the virtual object in response to the change in the user's gaze. For example, the processor (120) can operate to recognize the position of the user's gaze as an indicator (900) in conjunction with the user's gaze and manipulate the virtual object.

In operation 807, the processor (120) may perform an operation of determining an attribute of the input device. According to one embodiment, the processor (120) may determine a resolution (e.g., a predefined attribute (or capability) for the input device) according to a size (or type) of the input device. In one embodiment, the resolution may represent an ability to distinguish and select a specific virtual object using the input device. According to one embodiment, the resolution may vary depending on the input device, and the resolution (e.g., resolution value) for each input device may be stored in advance in the memory (130) of the electronic device (101). According to one embodiment, the resolution for each input device will be described with reference to the drawings described below.

In operation 809, the processor (120) may perform an operation of calculating a virtual space corresponding to a property of the input device based on the indicator. According to one embodiment, the processor (120) may determine a size (or range) of the virtual space (or virtual sphere or resolution sphere) based on a property (e.g., resolution) of the input device. According to one embodiment, the virtual space may include a virtual sphere (or resolution sphere) formed with a diameter of a resolution value of the input device centered on the input device (e.g., an indicator of the input device). An example of this is illustrated in FIG. 10.

Referring to FIG. 10, the virtual space (740) may be a space virtually (or internally) created to determine a target virtual object to be selected by the user based on the location of the input device (e.g., indicator (900)) among a plurality of virtual objects on the execution screen. For example, the virtual space (740) may not be displayed on the execution screen. An example of this is illustrated in the example screen <707> of FIG. 7b.

According to one embodiment, the virtual space (740) may represent a space that needs to be secured in order to accurately select one of a plurality of virtual objects of a virtual environment using the input device (720). According to one embodiment, the virtual space (740) may be expressed in the form of a virtual sphere having a specified distance (or length) (e.g., a length having a radius of about 1/2 of the resolution value) from the indicator (900) of the input device (720). According to one embodiment, the virtual space (740) expressed differently depending on the resolution value of each input device will be described with reference to the drawings described below.

In operation 811, the processor (120) may perform an operation of determining a target virtual object included in a virtual space among a plurality of virtual objects of an execution screen based on a virtual space. According to one embodiment, the processor (120) may extract (or infer) a virtual object included in a virtual space formed based on an input device among a plurality of virtual objects on the execution screen as a target virtual object. According to one embodiment, the target virtual object may include a plurality of virtual objects (e.g., at least two virtual objects). An example of this is illustrated in example screen <709> of FIG. 7c.

In operation 813, the processor (120) may perform an operation of generating an affordance based on a target virtual object included in the virtual space. According to one embodiment, the processor (120) may generate an affordance corresponding to each target virtual object based on determining the target virtual object based on the virtual space. According to one embodiment, the affordance may include a designated object (e.g., a handle, an icon, an image, or a list) that supports direct selection of a target virtual object among a plurality of virtual objects on an execution screen. According to one embodiment, the affordance may be provided as graphic data.

In operation 815, the processor (120) may perform an operation of displaying an affordance by associating it with a target virtual object. According to one embodiment, the processor (120) may provide an affordance by distinguishing it based on a virtual space corresponding to the secured space so as to support easy selection of the target virtual object. An example of this is illustrated in the example screen <711> of FIG. 7C. According to one embodiment, the specified arrangement method may include an object-based arrangement method (e.g., a first method) that provides an affordance by arranging it in a space in contact with a surface of a virtual space and a target virtual object, and a list-based arrangement method (e.g., a second method) that provides an affordance by arranging it in a list corresponding to each target virtual object outside the virtual space. According to one embodiment, an operation example of providing an affordance according to a specified arrangement method will be described with reference to the drawings described below.

FIG. 11 is a flowchart illustrating a method of operating an electronic device according to one embodiment of the present disclosure.

FIG. 12 is a diagram illustrating an example of providing a virtual space corresponding to an input device in an electronic device according to one embodiment of the present disclosure.

FIG. 13 is a diagram illustrating an example of resolution for each input device provided by an electronic device according to one embodiment of the present disclosure.

According to one embodiment, FIG. 11 may illustrate an example of an operation of providing a virtual space based on properties of an input device in an electronic device (101) according to one embodiment.

An operation method supported by an electronic device (101) according to one embodiment of the present disclosure may be performed, for example, according to a flowchart illustrated in FIG. 11. The flowchart illustrated in FIG. 11 is merely a flowchart according to one embodiment of an operation of the electronic device (101), and the order of at least some operations may be changed, performed in parallel, performed as independent operations, or at least some other operations may be performed complementarily to at least some operations. According to one embodiment of the present disclosure, operations 1101 to 1109 may be performed by at least one processor (120) of the electronic device (101) (e.g., the processor 120 of FIG. 1 or FIG. 5 ).

According to one embodiment, the operations described in FIG. 11 may be performed heuristically, for example, in combination with the operations described in FIGS. 6 to 10 , or heuristically performed as a replacement for at least some of the operations described and combined with at least some other operations, or heuristically performed as a detailed operation of at least some of the operations described.

As illustrated in FIG. 11, an operation method performed by an electronic device (101) according to one embodiment may include an operation of determining a property of an input device (1101), an operation of obtaining a resolution value corresponding to the property of the input device (1103), an operation of generating a virtual sphere having the resolution value as a diameter (1105), an operation of mapping a center of gravity of the virtual sphere (e.g., the center point of the sphere) to an indicator (1107), and an operation of forming a virtual space (1109).

Referring to FIG. 11, in operation 1101, the processor (120) of the electronic device (101) may perform an operation of determining an attribute of an input device. According to one embodiment, the processor (120) may determine an attribute (e.g., a type) of the input device based on recognizing the input device in a state of providing a virtual environment. According to one embodiment, the input device may include a hardware device such as a mouse, a smart pen (or an electronic pen), a controller, a wearable device (e.g., a ring and/or a watch) that is connected to the electronic device (101) via a specified communication method. According to one embodiment, the input device may include an object (or object device) such as a user's hand (or finger), a stylus pen, a wearable device, and/or a glove that is recognized through an execution screen in a virtual environment.

In operation 1103, the processor (120) may perform an operation of obtaining a resolution value corresponding to a property of the input device. According to one embodiment, the processor (120) may determine a resolution (e.g., a predefined property (or capability) for the input device) according to a size (or type) of the input device. In one embodiment, the resolution may represent an ability to distinguish and select a specific virtual object using the input device. According to one embodiment, the electronic device (101) may store a resolution (e.g., a resolution value) for each input device in advance in the memory (130). Examples thereof are illustrated in FIGS. 12 and 13 .

As illustrated in FIGS. 12 and 13, the resolution may vary for each input device (720). According to one embodiment, the resolution value for each input device may be stored in advance in the memory (130) of the electronic device (101). According to one embodiment, the resolution value for each input device may be stored externally (e.g., in a cloud server) of the electronic device (101), and the electronic device (101) may obtain (e.g., request and receive) the resolution value corresponding to the input device recognized from the external source and use it when necessary.

Referring to FIG. 12, examples <1201>, <1203>, and <1205> may represent examples in which the input device (720) is a user's hand (e.g., a thumb and an index finger), a controller, and a mouse. According to one embodiment, the processor (120) may determine a size (or range) of a virtual space (740) based on a predefined resolution value corresponding to a size (or type) of the input device (720). According to one embodiment, the processor (120) may form a virtual space (740) (e.g., a virtual sphere or a resolution sphere) with the resolution value of the input device (720) as the diameter centered on the input device (720) (e.g., an indicator (900) of the input device). For example, the processor (120) can express a virtual space (740) in the form of a virtual sphere having a specified distance (or length) (e.g., a length with a resolution value as a radius) from an indicator (900) of an input device (720).

According to one embodiment, example <1201> may represent an example in which a first resolution value (e.g., about 40 mm) is set for an input device (720) (e.g., a user's hand), a virtual space (740) having a diameter of the first resolution value is formed by the processor (120), and a center of gravity of the virtual space (740) is mapped and provided to an indicator (900). According to one embodiment, example <1203> may represent an example in which a second resolution value (e.g., about 25 mm) is set for an input device (720) (e.g., a controller), a virtual space (740) having a diameter of the second resolution value is formed by the processor (120), and a center of gravity of the virtual space (740) is mapped and provided to an indicator (900). According to one embodiment, example <1205> may represent an example in which a third resolution value (e.g., about 10 mm) is set for an input device (720) (e.g., a mouse), a virtual space (740) having a diameter of the third resolution value is formed by a processor (120), and a center of gravity of the virtual space (740) is mapped to an indicator (900) and provided.

According to one embodiment, the resolution value may be defined and/or learned based on the manipulation precision of object selection using the input device (720). For example, a relatively large (or high) resolution value may be set for an input device (e.g., a user's hand) with low accuracy for object selection, and a relatively small (or low) resolution value may be set for an input device (e.g., a mouse) with high accuracy for object selection. For example, a small resolution value may mean that the virtual space (740) forms a sphere with a small diameter. According to one embodiment, an input device (720) with a small resolution value may mean a device with high accuracy for object selection due to less noise (e.g., hardware noise for recognizing the input device (720) (e.g., camera noise) and/or user noise (e.g., hand tremors). According to one embodiment, an example of a resolution value that may be set for each input device (720) based on noise that may occur in the input device (720) is illustrated in FIG. 13.

Referring to FIG. 13, FIG. 13 may illustrate various examples of resolution values that are predefined for each type of input device (720). For example, a relatively large (or high) resolution value may be set for an input device (e.g., a user's hand) having low accuracy for object selection, and a relatively small (or low) resolution value may be set for an input device (e.g., a mouse) having high accuracy for object selection. For example, a lower resolution value may be assigned as the accuracy of object selection using the input device (720) increases. For example, as illustrated in FIG. 13, for a first input device (1301) (e.g., a user's hand) according to one embodiment, a first resolution value (e.g., approximately 40 mm) based on an experimental value may be set. For a second input device (1302) (e.g., a watch) according to one embodiment, a second resolution value (e.g., approximately 35 mm) based on an experimental value may be set. For a third input device (1303) (e.g., a ring) according to one embodiment, a third resolution value (e.g., about 30 mm) based on experimental values may be set. For a fourth input device (1304) (e.g., a controller) according to one embodiment, a fourth resolution value (e.g., about 25 mm) based on experimental values may be set. For a fifth input device (1305) (e.g., a glove) according to one embodiment, a fifth resolution value (e.g., about 20 mm) based on experimental values may be set. For a sixth input device (1306) (e.g., a smart pen) according to one embodiment, a sixth resolution value (e.g., about 15 mm) based on experimental values may be set. For a seventh input device (1307) (e.g., a mouse) according to one embodiment, a seventh resolution value (e.g., about 10 mm) based on experimental values may be set.

In operation 1105, the processor (120) may perform an operation of generating a virtual sphere having a resolution value as a diameter. According to one embodiment, the processor (120) may determine a size (or range) of the virtual space (or the virtual sphere or the resolution sphere) based on the resolution value of the input device. According to one embodiment, the processor (120) may generate a virtual sphere (or the resolution sphere) formed with a diameter of the resolution value of the input device (or a radius of about 1/2 of the resolution value from an indicator of the input device).

In operation 1107, the processor (120) may perform an operation of mapping a center of gravity of a virtual sphere to an indicator. In one embodiment, the processor (120) may map a center of gravity of a virtual space having a diameter of a resolution value specified for an input device to the indicator. An example of this is illustrated in FIG. 12.

In operation 1109, the processor (120) may perform an operation of forming a virtual space. According to one embodiment, the processor (120) may map a center of gravity of the virtual space based on an indicator of an input device, and form a virtual space having a diameter with a resolution value specified based on the center of gravity.

FIG. 14 is a flowchart illustrating a method of operating an electronic device according to one embodiment of the present disclosure.

According to one embodiment, FIG. 14 may illustrate an example of an operation for determining a resolution value for an input device in an electronic device (101) according to one embodiment.

An operation method supported by an electronic device (101) according to one embodiment of the present disclosure may be performed, for example, according to a flowchart illustrated in FIG. 14. The flowchart illustrated in FIG. 14 is merely a flowchart according to one embodiment of an operation of the electronic device (101), and the order of at least some operations may be changed or performed in parallel, performed as independent operations, or at least some other operations may be performed complementarily to at least some operations. According to one embodiment of the present disclosure, operations 1401 to 1405 may be performed by at least one processor (120) of the electronic device (101) (e.g., the processor (120) of FIG. 1 or FIG. 5 ).

According to one embodiment, the operations described in FIG. 14 may be performed heuristically, for example, in combination with the operations described in FIGS. 6 to 13 , or heuristically performed as a replacement for at least some of the operations described and combined with at least some other operations, or heuristically performed as a detailed operation of at least some of the operations described.

As illustrated in FIG. 14, an operation method performed by an electronic device (101) according to one embodiment may include an operation (1401) of obtaining a first resolution value related to an input device, an operation (1403) of obtaining a second resolution value related to user noise for the input device, and an operation (1405) of determining a resolution value for the input device based on the first resolution value and the second resolution value.

Referring to FIG. 14, at operation 1401, the processor (120) of the electronic device (101) may perform an operation of obtaining a first resolution value related to the input device. In one embodiment, the first resolution value may include a default resolution value predefined for the input device.

In operation 1403, the processor (120) may perform an operation of acquiring a second resolution value related to user noise for the input device. In one embodiment, the second resolution value may include a correction resolution value related to user noise (user nose) generated by a user using the input device. For example, the second correction value may learn about human errors, such as hand tremors, of a user when the user uses the input device, and may be updated as a correction value for the corresponding input device based on the learning result. For example, the second correction value may include a resolution value that is corrected by considering user noise using the corresponding input device for the first resolution value for the input device. According to one embodiment, the resolution value may include a first resolution value that is stored in advance in the electronic device (101) for each input device, and a second resolution value (e.g., a hand tremors correction value) that corrects user noise that is updated according to user learning.

In operation 1405, the processor (120) may perform an operation of determining a resolution value for the input device based on the first resolution value and the second resolution value. According to one embodiment, the processor (120) may calculate a final resolution value for the input device by considering the first resolution value of the input device and the second resolution value that compensates for human error (e.g., user noise) of a user using the input device.

FIG. 15 is a flowchart illustrating a method of operating an electronic device according to one embodiment of the present disclosure.

FIGS. 16A and 16B are diagrams illustrating examples of determining a target virtual object in an electronic device according to one embodiment of the present disclosure.

According to one embodiment, FIG. 15 may illustrate an example of an operation for determining entry into a guide generation mode for providing affordance in an electronic device (101) according to one embodiment.

An operation method supported by an electronic device (101) according to one embodiment of the present disclosure may be performed, for example, according to a flowchart illustrated in FIG. 15. The flowchart illustrated in FIG. 15 is merely a flowchart according to one embodiment of an operation of the electronic device (101), and the order of at least some operations may be changed or performed in parallel, performed as independent operations, or at least some other operations may be performed complementarily to at least some operations. According to one embodiment of the present disclosure, operations 1501 to 1509 may be performed by at least one processor (120) of the electronic device (101) (e.g., the processor (120) of FIG. 1 or FIG. 5 ).

According to one embodiment, the operations described in FIG. 15 may be performed heuristically, for example, in combination with the operations described in FIGS. 6 to 14 , or heuristically performed as a replacement for at least some of the operations described and combined with at least some other operations, or heuristically performed as a detailed operation of at least some of the operations described.

As illustrated in FIG. 15, an operation method performed by an electronic device (101) according to an embodiment may include an operation (1501) of identifying a target virtual object included in a virtual space, an operation (1503) of determining whether the target virtual object satisfies a specified condition, an operation (1505) of determining entry into a guide generation mode based on the target virtual object satisfying the specified condition, an operation (1507) of performing an operation related to affordance generation, and an operation (1509) of processing the performance of the corresponding operation based on the target virtual object not satisfying the specified condition.

Referring to FIG. 15, in operation 1501, the processor (120) of the electronic device (101) may perform an operation of identifying a target virtual object included in a virtual space. According to one embodiment, the processor (120) may extract (or infer) a virtual object included in a virtual space formed based on an input device among a plurality of virtual objects on an execution screen as a target virtual object. According to one embodiment, the processor (120) may identify a virtual object as a target virtual object when part or all of the virtual object is included in a virtual space centered on an indicator of the input device.

In operation 1503, the processor (120) may perform an operation of determining whether the target virtual object satisfies a specified condition. According to one embodiment, the specified condition may include a condition in which the target virtual object is a plurality of virtual objects, at least two of which are virtual objects, and a distance between centers of gravity of each of the plurality of virtual objects is less than or equal to a specified proximity distance. Examples of this are illustrated in FIGS. 16A and 16B.

Referring to FIG. 16A, FIG. 16A may illustrate an example in which both a first virtual object (1610) and a second virtual object (1620) are included in a virtual space (740). For example, in a case in which a plurality of virtual objects, such as at least two virtual objects (e.g., the first virtual object (1610) and the second virtual object (1620)), are included in a virtual space (740), it may be difficult for a user to distinguish and select the virtual objects. According to one embodiment, in a case in which at least two virtual objects (1610, 1620) are included in a virtual space (740), the processor (120) may operate to provide affordances for at least two virtual objects (1610, 1620) so that a user can easily select and manipulate the virtual objects.

Referring to FIG. 16B, FIG. 16B may illustrate an example in which a part of a third virtual object (1630) and a part of a fourth virtual object (1640) are included in a virtual space (740). For example, in a case in which a part of a plurality of virtual objects, such as at least two virtual objects (e.g., the third virtual object (1630) and the fourth virtual object (1640)), are included in a virtual space (740), it may be easy or difficult for a user to select a distinction between the virtual objects depending on the distance between the centers of gravity of each of the plurality of virtual objects (1630, 1640). According to one embodiment, as illustrated in FIG. 16B, when the distance (L) between the first center of gravity (1635) of the third virtual object (1630) and the second center of gravity (1645) of the fourth virtual object (1640) is greater than a specified proximity distance, it may be easy for a user to select a distinction between the virtual objects. For example, if the center of gravity of at least one virtual object exists outside the virtual space (740) and there is no overlap between the virtual objects (1630, 1640), the user's selection of a distinction for the virtual object can be easy.

In one embodiment, if the distance (L) between the first center of gravity (1635) of the third virtual object (1630) and the second center of gravity (1645) of the fourth virtual object (1640) in FIG. 16b is less than or equal to a specified proximity distance, the user may not easily select a distinct virtual object. For example, if the centers of gravity (1635, 1645) of a plurality of virtual objects (1630, 1640) exist within the virtual space (740) or there is overlap between the virtual objects (1630, 1640), the user may have difficulty selecting a distinct virtual object. According to one embodiment, the processor (120) may be operable to provide affordances for at least two virtual objects (1630, 1640) so that a user can easily select and manipulate the virtual objects when the distance (L) between centers of gravity (1635, 1645) of the plurality of virtual objects (1630, 1640) is less than or equal to a specified proximity distance.

In operation 1503, if the processor (120) determines that the target virtual object satisfies the specified condition (e.g., ‘Yes’ in operation 1503), in operation 1505, the processor (120) may perform an operation of determining whether to enter a guide generation mode. According to one embodiment, if the target virtual object in the virtual space is a plurality of virtual objects, which are at least two, and the distance between the centers of gravity of each of the plurality of virtual objects is less than or equal to a specified proximity distance, the processor (120) may determine that the specified condition is satisfied. According to one embodiment, the processor (120) may determine to enter a guide generation mode for generating an affordance based on determining that the specified condition is satisfied. An example of this is illustrated in FIG. 16A.

In operation 1507, the processor (120) may perform an operation related to affordance generation. According to one embodiment, the processor (120) may enter a guide generation mode and perform an operation of generating affordances associated with a plurality of virtual objects included in a virtual space. According to one embodiment, an operation example of generating and providing affordances is described with reference to the drawings described below.

In operation 1503, if the processor (120) determines that the target virtual object does not satisfy the specified condition (e.g., ‘No’ in operation 1503), in operation 1509, the processor (120) may perform an operation for processing the execution of the corresponding operation. According to one embodiment, the processor (120) may recognize a user input using an input device. According to one embodiment, the processor (120) may process an operation for selecting a virtual object based on the user input and executing a related function (e.g., object movement, object editing).

FIG. 17 is a flowchart illustrating a method of operating an electronic device according to one embodiment of the present disclosure.

According to one embodiment, FIG. 17 may illustrate an example of an operation for determining entry into a guide generation mode for providing affordance in an electronic device (101) according to one embodiment.

An operation method supported by an electronic device (101) according to one embodiment of the present disclosure may be performed, for example, according to a flowchart illustrated in FIG. 17. The flowchart illustrated in FIG. 17 is merely a flowchart according to one embodiment of an operation of the electronic device (101), and the order of at least some operations may be changed or performed in parallel, performed as independent operations, or at least some other operations may be performed complementarily to at least some operations. According to one embodiment of the present disclosure, operations 1701 to 1715 may be performed by at least one processor (120) of the electronic device (101) (e.g., the processor (120) of FIG. 1 or FIG. 5 ).

According to one embodiment, the operations described in FIG. 17 may be performed heuristically, for example, in combination with the operations described in FIGS. 6 to 16b , or heuristically performed as a replacement for at least some of the operations described and combined with at least some other operations, or heuristically performed as a detailed operation of at least some of the operations described.

As illustrated in FIG. 17, an operation method performed by an electronic device (101) according to an embodiment may include an operation of identifying a target virtual object included in a virtual space (1701), an operation of determining whether the virtual space includes a plurality of target virtual objects (1703), an operation of determining a center of gravity corresponding to each of the plurality of target virtual objects based on the inclusion of the plurality of target virtual objects in the virtual space (1705), an operation of determining a distance between the centers of gravity of each of the plurality of target virtual objects (1707), an operation of comparing the distance between the centers of gravity with a specified proximity distance (1709), an operation of determining whether there is a distance between the centers of gravity that is less than or equal to the specified proximity distance (1711), an operation of determining entry into a guide generation mode based on the existence of a distance less than or equal to the specified proximity distance (1713), and an operation of processing the execution of the corresponding operation when the virtual space does not include a plurality of target virtual objects or does not include a distance less than or equal to the specified proximity distance (1715).

Referring to FIG. 17, in operation 1701, the processor (120) of the electronic device (101) may perform an operation of identifying a target virtual object included in a virtual space. According to one embodiment, the processor (120) may extract (or infer) a virtual object included in a virtual space formed based on an input device among a plurality of virtual objects on an execution screen as a target virtual object. According to one embodiment, the processor (120) may identify a virtual object as a target virtual object when part or all of the virtual object is included in a virtual space centered on an indicator of the input device.

In operation 1703, the processor (120) may perform an operation of determining whether the virtual space includes a plurality of target virtual objects. According to one embodiment, the processor (120) may determine whether the virtual space includes at least two plurality of target virtual objects, as in the examples of FIG. 16A and/or FIG. 16B.

In operation 1703, if the processor (120) determines that the virtual space does not include multiple target virtual objects (e.g., ‘No’ in operation 1703), in operation 1715, the processor (120) may process the execution of the corresponding operation. According to one embodiment, the processor (120) may recognize a user input using an input device. According to one embodiment, the processor (120) may process an operation of selecting a virtual object based on the user input and executing a related function (e.g., object movement, object editing).

In operation 1703, if the processor (120) determines that the virtual space includes multiple target virtual objects (e.g., ‘Yes’ in operation 1703), in operation 1705, the processor may perform an operation of determining a center of gravity corresponding to each of the multiple target virtual objects.

In operation 1707, the processor (120) may perform an operation of determining a distance between centers of gravity of each of the plurality of target virtual objects. According to one embodiment, when some or all of the plurality of virtual objects, such as at least two virtual objects, are included in a virtual space, the processor (120) may determine the distance between the centers of gravity of each of the plurality of virtual objects. For example, the processor (120) may determine a distance (L) between a first center of gravity (1635) of a third virtual object (1630) and a second center of gravity (1645) of a fourth virtual object (1640), as illustrated in FIG. 16B .

In operation 1709, the processor (120) may perform an operation of comparing a distance between centers of gravity with a designated proximity distance. According to one embodiment, the processor (120) may compare distances between centers of gravity (e.g., multiple distances such as a first distance and a second distance) corresponding to each of a plurality of target virtual objects in the virtual space with the designated proximity distance. For example, it may be assumed that the virtual space includes a first virtual object, a second virtual object, and a third virtual object. According to one embodiment, the processor (120) may determine a first distance between centers of gravity of the first virtual object and the second virtual object, a second distance between centers of gravity of the first virtual object and the third virtual object, and a third distance between centers of gravity of the second virtual object and the third virtual object. According to one embodiment, the processor (120) may compare the first distance with the designated proximity distance, compare the second distance with the designated proximity distance, and compare the third distance with the designated proximity distance.

In operation 1711, the processor (120) may perform an operation of determining whether there is a distance between centers of gravity that is less than or equal to a specified close distance. According to one embodiment, it may be determined whether there is a distance less than or equal to a specified close distance among the first distance, the second distance, and/or the third distance.

In operation 1711, if the processor (120) determines that there is a distance less than or equal to a specified proximity distance (e.g., ‘Yes’ in operation 1711), in operation 1713, the processor (120) may perform an operation of determining entry into a guide generation mode. According to one embodiment, if there are at least two target virtual objects in a virtual space and the distance between the centers of gravity of each of the plurality of virtual objects is less than or equal to a specified proximity distance, the processor (120) may determine entry into a guide generation mode for generating an affordance.

In operation 1711, if the processor (120) determines that there is no distance less than the specified proximity distance (e.g., ‘No’ in operation 1711), in operation 1715, the processor (120) may process the execution of the corresponding operation. According to one embodiment, the processor (120) may recognize a user input using an input device. According to one embodiment, the processor (120) may process an operation of selecting a virtual object based on the user input and executing a related function (e.g., object movement, object editing).

FIG. 18 is a flowchart illustrating a method of operating an electronic device according to one embodiment of the present disclosure.

According to one embodiment, FIG. 18 may illustrate an example of an operation of providing affordance in an electronic device (101) according to one embodiment.

An operation method supported by an electronic device (101) according to one embodiment of the present disclosure may be performed, for example, according to a flowchart illustrated in FIG. 18. The flowchart illustrated in FIG. 18 is merely a flowchart according to one embodiment of an operation of the electronic device (101), and the order of at least some operations may be changed or performed in parallel, performed as independent operations, or at least some other operations may be performed complementarily to at least some operations. According to one embodiment of the present disclosure, operations 1801 to 1803 may be performed by at least one processor (120) of the electronic device (101) (e.g., the processor (120) of FIG. 1 or FIG. 5 ).

According to one embodiment, the operations described in FIG. 18 may be performed heuristically, for example, in combination with the operations described in FIGS. 6 to 17, or heuristically performed as a replacement for at least some of the operations described and combined with at least some other operations, or heuristically performed as a detailed operation of at least some of the operations described.

As illustrated in FIG. 18, an operation method performed by an electronic device (101) according to one embodiment may include an operation (1801) of determining an affordance arrangement method, and an operation (1803) of rendering an affordance based on the arrangement method and a resolution value.

Referring to FIG. 18, in operation 1801, the processor (120) of the electronic device (101) may perform an operation of determining an affordance arrangement method. According to one embodiment, the processor (120) may determine a designated arrangement method (or an affordance type or an affordance display type) for arranging an affordance on a virtual space based on a relationship between the virtual space and the target virtual object. According to one embodiment, the designated arrangement method may include a arrangement method (e.g., a first arrangement method) that provides an affordance by arranging it in a space in contact with a surface of the virtual space and the target virtual object. According to one embodiment, the designated arrangement method may include a arrangement method (e.g., a second arrangement method) that provides an affordance by arranging it in a list corresponding to each target virtual object outside the virtual space.

In operation 1803, the processor (120) may perform an operation of rendering an affordance based on a specified arrangement method and a resolution value. According to one embodiment, the processor (120) may generate an affordance with two-dimensional or three-dimensional graphic data (e.g., an icon or image of a specified shape (e.g., a cube, a circle, an oval, a triangle, a diamond), an image of an object shape, text based on text recognition included in an object). According to one embodiment, the processor (120) may generate an affordance realistically by considering object information (e.g., information about a location, a color, and/or a light source) related to a target virtual object in a virtual environment. For example, the processor (120) may generate an affordance with a color contrasting with a color of a target virtual object, and may place the affordance in a space that does not overlap with a location of the target virtual object in the virtual space. For example, the processor (120) can determine the size and/or spacing of the affordance based on the resolution value, and place the affordance based on the determined size and/or spacing. Examples of this are illustrated in FIGS. 19a, 19b, and 19c.

FIGS. 19A, 19B, and 19C are diagrams illustrating examples of providing affordance in an electronic device according to one embodiment of the present disclosure.

As illustrated in FIGS. 19a, 19b, and 19c, affordances may be provided in the form of designated objects each corresponding to a target virtual object based on the virtual space (740). According to one embodiment, when the interval (L) between the arranged affordances is smaller than the radius (r) of the virtual space (e.g., L < r), affordances may be provided in the form of designated lists each corresponding to a target virtual object.

Referring to FIG. 19A, FIG. 19A may illustrate an example of arranging an affordance when a target virtual object (e.g., a first virtual object (1910), a second virtual object (1920)) spans a virtual space (740). According to one embodiment, when the target virtual objects (1910, 1920) span a virtual space (740), the electronic device (101) may determine an affordance type as an internal arrangement method (or internal type) that places the affordance type inside the target virtual objects (1910, 1920).

According to one embodiment, the electronic device (101) may provide an affordance (e.g., a first affordance (1915), a second affordance (1925)) (e.g., an internal affordance disposed inside the virtual object) by placing it on a space (or surface) where the target virtual object (1910, 1920) contacts a surface of the virtual space (740). For example, the first affordance (1915) (e.g., a first internal handle) may be placed on a surface where the first virtual object (1910) contacts the virtual space (740), and the second affordance (1925) (e.g., a second internal handle) may be placed on a surface where the second virtual object (1920) contacts the virtual space (740).

Referring to FIG. 19B, FIG. 19B may illustrate an example of arranging an affordance when a target virtual object (e.g., a third virtual object (1930), a fourth virtual object (1940)) is inside a virtual space (740). According to one embodiment, when the target virtual objects (1930, 1940) are inside a virtual space (740), the electronic device (101) may determine an affordance type as an external arrangement method (or external type) that arranges the affordance type in a space (or surface) on an extension line to a surface of the virtual space (740).

According to one embodiment, the electronic device (101) may provide an affordance (e.g., a third affordance (1935), a fourth affordance (1945)) (e.g., an external affordance disposed on the surface of the virtual space (740)) by placing it in a space along an extension line to a surface of the virtual space (740). For example, the third affordance (1935) (e.g., a first external handle) may be placed in a space along an extension line from a third virtual object (1930) to a close surface (e.g., a surface at a short distance) of the virtual space (740) (or a surface contacting the extension line), and the fourth affordance (1945) (e.g., a second external handle) may be placed in a space along an extension line from a fourth virtual object (1940) to a close surface (e.g., a surface at a short distance) of the virtual space (740) (or a surface contacting the extension line).

Referring to FIG. 19c, FIG. 19c may illustrate an example of arranging affordances when some or all of target virtual objects (e.g., the fifth virtual object (1950), the sixth virtual object (1960), the seventh virtual object (1970), and the eighth virtual object (1980)) exist within the virtual space (740) or span the virtual space (740), and a gap (L) between at least some affordances is smaller than a radius of the virtual space (740). According to one embodiment, the electronic device (101) may arrange affordances (e.g., the fifth affordance (1955), the sixth affordance (1965), the seventh affordance (1975), and the eighth affordance (1985)) based on the virtual space (740), and when the interval (L) between each of the affordances (1955, 1965, 1975, 1985) arranged in the virtual space (740) is smaller than the radius of the virtual space (740), the affordance types may be determined in a list arrangement manner (or list type) that arranges them in a list form on the periphery of the virtual space (740) within the virtual environment.

According to one embodiment, the electronic device (101) may provide an affordance (1990) (e.g., a list-based affordance) by placing it in an area outside the virtual space (740) within the virtual environment. For example, the electronic device (101) may generate an affordance (1990) in a list form and place it in an area within the virtual environment if a gap (L) between at least two of the affordances (e.g., the fifth affordance (1955) and the sixth affordance (1965)) among the affordances (e.g., the fifth affordance (1955), the sixth affordance (1965), the seventh affordance (1975), and the eighth affordance (1985)) placed based on the virtual space (740) is smaller than a radius of the virtual space (740) (e.g., half of the resolution value).

According to one embodiment, the list-based affordance (1990) may be provided in the form of a list including items (e.g., a fifth item, a sixth item, a seventh item, and an eighth item) each corresponding to a target virtual object (e.g., a fifth virtual object (1950), a sixth virtual object (1960), a seventh virtual object (1970), and an eighth virtual object (1980)).

FIG. 20 is a flowchart illustrating a method of operating an electronic device according to one embodiment of the present disclosure.

According to one embodiment, FIG. 20 may illustrate an example of an operation of providing affordance in an electronic device (101) according to one embodiment.

An operation method supported by an electronic device (101) according to one embodiment of the present disclosure may be performed, for example, according to a flowchart illustrated in FIG. 20. The flowchart illustrated in FIG. 20 is merely a flowchart according to one embodiment of an operation of the electronic device (101), and the order of at least some operations may be changed, performed in parallel, performed as independent operations, or at least some other operations may be performed complementarily to at least some operations. According to one embodiment of the present disclosure, operations 2001 to 2015 may be performed by at least one processor (120) of the electronic device (101) (e.g., the processor (120) of FIG. 1 or 5 ).

According to one embodiment, the operations described in FIG. 20 may be performed heuristically, for example, in combination with the operations described in FIGS. 6 to 19c, or heuristically performed as a replacement for at least some of the operations described and combined with at least some other operations, or heuristically performed as a detailed operation of at least some of the operations described.

As illustrated in FIG. 20, the operation method performed by the electronic device (101) according to one embodiment includes an operation of identifying a state between a target virtual object and a virtual space (2001), an operation of determining whether the target virtual object is in a state spanning the virtual space (2003), an operation of providing an affordance based on a space where the target virtual object contacts a surface of the virtual space if the target virtual object is in a state spanning the virtual space (2005), an operation of determining whether the target virtual object is in a state included in the virtual space (2007), an operation of providing an affordance based on a space in an extension line from the target virtual object to the surface of the virtual space if the target virtual object is in a state included in the virtual space (2009), an operation of determining whether a gap between the target virtual objects is smaller than a radius of the virtual space (2011), an operation of providing a list-based affordance based on a specified space of the virtual space if the gap between the target virtual objects is smaller than a radius of the virtual space (2013), and an operation of determining whether a state between the target virtual object and the virtual space is the above state (e.g., operation 2003, operation If not included in the state of 2007 and operation 2011, it may include an operation (2015) that provides an affordance based on the contact space and/or the space along the extension.

Referring to FIG. 20, in operation 2001, the processor (120) of the electronic device (101) may perform an operation of identifying a state between a target virtual object and a virtual space. According to one embodiment, the processor (120) may determine whether the state between the target virtual object and the virtual space corresponds to a first state, a second state, or a third state. In one embodiment, the first state may include a state in which the target virtual object spans the virtual space. In one embodiment, the second state may include a state in which the target virtual object is included within the virtual space. In one embodiment, the third state may include a state in which a gap between affordances is smaller than a radius of the virtual space.

In operation 2003, the processor (120) may perform an operation to determine whether a target virtual object is in a state (e.g., a first state) spanning the virtual space. An example of the first state is illustrated in FIG. 19a.

In operation 2003, if the target virtual object is in a state (e.g., a first state) extending over the virtual space (e.g., ‘yes’ in operation 2003), in operation 2005, the processor (120) may perform an operation of providing an affordance based on a space where the target virtual object contacts a surface of the virtual space. According to one embodiment, if the target virtual object extends over the virtual space, the processor (120) may determine an affordance type as an internal placement method (or internal type) that places the affordance inside the target virtual object. According to one embodiment, the processor (120) may place and provide an affordance in a space (or surface) where the target virtual object contacts a surface of the virtual space.

In operation 2007, the processor (120) may perform an operation of determining whether the target virtual object is in a state (e.g., a second state) included within the virtual space. An example of the second state is illustrated in FIG. 19b.

In operation 2007, if the target virtual object is in a state (e.g., a second state) included in the virtual space (e.g., ‘yes’ in operation 2007), in operation 2009, the processor (120) may perform an operation of providing an affordance based on a space on an extension line from the target virtual object to a surface of the virtual space. According to one embodiment, if the target virtual object exists in the virtual space, the processor (120) may determine an affordance type as an external placement method (or external type) that places the affordance in a space (or plane) on an extension line to the surface of the virtual space. According to one embodiment, the processor (120) may place and provide the affordance in a space on an extension line to the surface of the virtual space.

In operation 2011, the processor (120) may perform an operation of determining whether a state (e.g., a third state) is such that the gap between target virtual objects is smaller than a radius of the virtual space. An example of the third state is illustrated in FIG. 19c.

In operation 2011, if the gap between target virtual objects is smaller than the radius of the virtual space (e.g., the third state), then in operation 2013, the processor (120) may perform an operation of providing a list-based affordance based on a specified space in the virtual space. According to one embodiment, the processor (120) may arrange the affordances based on the virtual space, and if the gap (L) between each of the affordances arranged in the virtual space is smaller than the radius of the virtual space, the affordance type may be determined as a list arrangement method (or list type) that arranges the affordance type in a list form on the outskirts of the virtual space within the virtual environment. According to one embodiment, the processor (120) may arrange and provide an affordance (e.g., a list-based affordance) in an area outside the virtual space within the virtual environment. For example, the processor (120) may generate a list of affordances and place them in an area within the virtual environment if the distance (L) between at least two affordances arranged based on the virtual space is smaller than the radius of the virtual space (e.g., half of the resolution value).

In operation 2003, operation 2005, and operation 2011, if the state between the target virtual object and the virtual space is not included in the first state, the second state, and the third state, in operation 2015, the processor (120) may perform an operation of providing an affordance based on the contact space and/or the space on the extension line. According to one embodiment, the processor (120) may provide a part or all of the affordance corresponding to each target virtual object by arranging it within the corresponding object in the virtual space, as in the example of FIG. 19a. According to one embodiment, the processor (120) may provide a part or all of the affordance corresponding to each target virtual object by arranging it on a plane on the extension line that contacts the virtual space, as in the example of FIG. 19b. According to one embodiment, the processor (120) may provide a part of the affordance corresponding to each target virtual object by arranging it within the corresponding object in the virtual space, and provide another part of the affordance by arranging it on a plane on the extension line that contacts the virtual space.

FIGS. 21A, 21B, and 21C are diagrams illustrating examples of determining the size of an affordance in an electronic device according to one embodiment of the present disclosure.

In one embodiment, the electronic device (101) may determine the size and/or spacing of the affordance based on a resolution value for the virtual space (e.g., a diameter of the virtual space).

Referring to FIG. 21A, the electronic device (101) may form a virtual space (740) having a resolution value of the input device (720) as a diameter (D) based on an indicator (900) according to the input device (720) (e.g., a user's hand). According to one embodiment, FIG. 21A may show an example in which the input device (720) is a user's hand. According to one embodiment, the electronic device (101) may obtain a resolution value (e.g., a first resolution value set for the user's hand) (e.g., about 40 mm) of the input device (720). According to one embodiment, the electronic device (101) may form a virtual space (740) having a diameter (e.g., D = about 40 mm) equal to the first resolution value (e.g., about 40 mm) of the input device (720). According to one embodiment, the electronic device (101) can determine a size (e.g., width) of the affordance (2110) based on a half (e.g., radius (r), where r = D/2) of the first resolution value (or diameter (D) of the virtual space (740)) (e.g., about 40 mm). For example, the affordance (2110) can be provided in a designated shape (e.g., cube, circle, oval, triangle, diamond) having a width of half (e.g., about 20 mm) of the first resolution value (or diameter (D) of the virtual space (740)) (e.g., about 40 mm).

Referring to FIG. 21b, the electronic device (101) may form a virtual space (740) having a resolution value of the input device (720) as a diameter (D) based on an indicator (900) according to the input device (720) (e.g., a controller). According to one embodiment, FIG. 21b may show an example in which the input device (720) is a controller. According to one embodiment, the electronic device (101) may obtain a resolution value (e.g., a second resolution value set for the controller) (e.g., about 25 mm) of the input device (720). According to one embodiment, the electronic device (101) may form a virtual space (740) having a diameter (e.g., D = about 25 mm) equal to the second resolution value (e.g., about 25 mm) of the input device (720). In one embodiment, the electronic device (101) can determine a size (e.g., width) of the affordance (2120) based on a half (e.g., radius (r), where r = D/2) of the second resolution value (or diameter (D) of the virtual space (740)) (e.g., about 25 mm). For example, the affordance (2120) can be provided in a designated shape (e.g., cube, circle, oval, triangle, diamond) having a width of half (e.g., about 12.5 mm) of the second resolution value (or diameter (D) of the virtual space (740)) (e.g., about 25 mm).

Referring to FIG. 21c, the electronic device (101) may form a virtual space (740) having a resolution value of the input device (720) as a diameter (D) based on an indicator (900) according to the input device (720) (e.g., a controller). According to one embodiment, FIG. 21c may show an example in which the input device (720) is a controller and provides an affordance (2130) in the form of a list. According to one embodiment, the electronic device (101) may obtain a resolution value (e.g., a second resolution value set for the controller) (e.g., about 25 mm) of the input device (720). According to one embodiment, the electronic device (101) may form a virtual space (740) having a diameter (e.g., D = about 25 mm) of the second resolution value (e.g., about 25 mm) of the input device (720). In one embodiment, the electronic device (101) can determine the size of the affordance (2130) (e.g., spacing between items) based on half (e.g., radius (r), where r = D/2) of the second resolution value (or diameter (D) of the virtual space (740)) (e.g., about 25 mm). For example, the affordance (2130) can be provided in the form of a list having a spacing of half (e.g., about 12.5 mm) of the second resolution value (or diameter (D) of the virtual space (740)) (e.g., about 25 mm).

FIGS. 22A and 22B are diagrams illustrating examples of operations for providing affordance in association with a virtual object in an electronic device according to one embodiment of the present disclosure.

According to one embodiment, FIGS. 22A and 22B may illustrate an example of an operation of providing an affordance (2250) that can support selection of at least one of a target virtual object in a virtual environment (2200) in association with a target virtual object in an electronic device (101) according to one embodiment.

Referring to FIGS. 22A and 22B , the electronic device (101) can recognize an input device (720) while displaying a virtual environment (2200) (or an execution screen (2200) of the virtual environment) through a display (e.g., a display module (160) of FIG. 1 or 5 ). According to one embodiment, assuming that the input device (720) is a user's hand (e.g., a finger) in FIGS. 22A and 22B , the electronic device (101) can recognize the user's hand (720) entering the virtual environment (2200).

According to one embodiment, the electronic device (101) may generate and display affordances (2250) corresponding to target virtual objects that are targets of user selection based on a virtual space formed based on an input device (720) (e.g., an indicator (900) of the input device (720).

According to one embodiment, the electronic device (101) may display an affordance (2250) (e.g., a guide handle) related to a target virtual object (e.g., a chess piece) in a virtual space formed around the input device (720) (e.g., an indicator (900) of the input device (720)) when the input device (720) is stationary for a predetermined period of time based on at least one virtual object among a plurality of virtual objects in the virtual environment (2200). According to one embodiment, the user may select and manipulate (e.g., move, delete, copy, and/or edit) at least one target virtual object (e.g., a chess piece) based on the affordance (2250).

FIGS. 23A, 23B, and 23C are diagrams illustrating examples of operations for providing affordance in association with a virtual object in an electronic device according to one embodiment of the present disclosure.

According to one embodiment, FIGS. 23A through 23C may illustrate examples of an operation of providing an affordance (2350) that can support selection of at least one of a target virtual object in a virtual environment (2300) in association with a target virtual object in an electronic device (101) according to one embodiment.

Referring to FIGS. 23A to 23B , the electronic device (101) may recognize the input device (720) while displaying the virtual environment (2300) (or the execution screen (2300) of the virtual environment) through a display (e.g., the display module (160) of FIG. 1 or 5 ). According to one embodiment, assuming that the input device (720) is a controller in FIGS. 23A to 23C , the electronic device (101) may recognize the indicator (900) of the input device (720) connected to the electronic device (101).

According to one embodiment, the electronic device (101) may generate and display affordances (2350) corresponding to target virtual objects that are targets of user selection based on a virtual space formed based on an input device (720) (e.g., an indicator (900) of the input device (720).

According to one embodiment, the electronic device (101) may display an affordance (2350) (e.g., a guide handle) related to a target virtual object (e.g., a shape) in a virtual space formed around the input device (720) (e.g., the indicator (900) of the input device (720)) when the input device (720) (or the indicator (900) of the input device (720)) stops for a predetermined period of time based on at least one virtual object among a plurality of virtual objects in the virtual environment (2300). According to one embodiment, the user may select and manipulate (e.g., move, delete, copy, and/or edit) at least one target virtual object (e.g., a shape) based on the affordance (2350).

Referring to FIG. 23c, a user may select one of the affordances (2360) among the affordances (2350) corresponding to each target virtual object by using the input device (720). According to one embodiment, when the input device (720) (e.g., the indicator (900) of the input device (720)) stops (e.g., holds) for a predetermined period of time specified for one of the affordances (2360), the electronic device (101) may determine the depth (e.g., depth menu) of the affordance (2360). For example, the electronic device (101) may determine whether menu expansion of the affordance (2360) is possible. In one embodiment, the electronic device (101) may provide a depth menu (2370) associated with (or adjacent to or around) the selected affordance (2360) when there is a depth menu related to the affordance (2360) selected by the input device (720). In one embodiment, the depth menu (2370) may include multiple sub-depths, such as n-depth, and the electronic device (101) may provide up to n-depth.

FIGS. 24A, 24B, and 24C are diagrams illustrating examples of operations for providing affordance in association with a virtual object in an electronic device according to one embodiment of the present disclosure.

According to one embodiment, FIGS. 24A to 24C may illustrate examples of an operation of providing an affordance (2450) that can support selection of at least one of target virtual objects in a virtual environment (2400) by associating the target virtual object with the target virtual object in an electronic device (101) according to one embodiment. According to one embodiment, FIGS. 24A to 24C may illustrate examples of a case where an input device (720) is a user's hand and provides an affordance (2450) in the form of a list.

Referring to FIGS. 24A to 24B , the electronic device (101) may recognize an input device (720) while displaying a virtual environment (2400) (or an execution screen (2400) of the virtual environment) through a display (e.g., a display module (160) of FIG. 1 or 5 ). According to one embodiment, assuming that the input device (720) is a user's hand as an example in FIGS. 24A to 24C , the electronic device (101) may recognize the user's hand (720) entering the virtual environment (2400).

According to one embodiment, the electronic device (101) may generate and display affordances (2450) corresponding to target virtual objects that are targets of user selection based on a virtual space formed based on an input device (720) (e.g., an indicator (900) of the input device (720).

According to one embodiment, the electronic device (101) may determine a list arrangement method (or list type) for arranging affordances corresponding to each target virtual object based on the virtual space, and arranging them in a list form on the outskirts of the virtual space within the virtual environment (2400) when the interval (L) between each of the affordances arranged in the virtual space is smaller than the radius of the virtual space. According to one embodiment, the electronic device (101) may provide an affordance (2450) (e.g., a list-based affordance) by arranging it in an area outside the virtual space within the virtual environment (2400). For example, when the interval (L) between at least two of the affordances arranged based on the virtual space is smaller than the radius of the virtual space (e.g., half of the resolution value), the electronic device (101) may generate an affordance (2450) in a list form and place it in an area within the virtual environment (2400).

According to one embodiment, the electronic device (101) may display, in a list format, an affordance (2450) (e.g., a guide handle) related to a target virtual object (e.g., a book) in a virtual space formed around the input device (720) (e.g., the indicator (900) of the input device (720)), when the input device (720) (or the indicator (900) of the input device (720)) stops for a predetermined period of time based on at least one virtual object among a plurality of virtual objects in the virtual environment (2400). According to one embodiment, the user may select and manipulate (e.g., move, delete, copy, and/or edit) at least one target virtual object (e.g., a book) based on the affordance (2450).

Referring to FIG. 24c, a user may select one of the affordances (2460) among the affordances (2450) corresponding to each target virtual object by using the input device (720). According to one embodiment, when the input device (720) (e.g., the indicator (900) of the input device (720)) stops (e.g., holds) for a predetermined period of time for one of the affordances (2460), the electronic device (101) may highlight (2470) (e.g., highlight, select box) and provide the target virtual object corresponding to the affordance (2360) in the virtual environment (2400). For example, when an item (2460) is selected from an affordance (2450) in a list form via an input device (720), the electronic device (101) can distinguish a target virtual object corresponding to the selected item (2460) from other target virtual objects in a virtual environment (2400), thereby increasing the visibility and/or legibility of the target virtual object. According to one embodiment, the electronic device (101) can configure each item in the affordance (2450) with two-dimensional or three-dimensional graphic data (e.g., an icon or image of a specified shape (e.g., a cube, a circle, an oval, a triangle, a diamond) or an image of an object shape or a text based on text recognition included in an object) to increase the visibility and/or legibility of the corresponding target virtual object for each item.

FIGS. 25A and 25B are diagrams illustrating examples of operations for providing affordance in association with a virtual object in an electronic device according to one embodiment of the disclosure.

Referring to FIGS. 25A and 25B , the electronic device (101) may provide an affordance (2500) including a plurality of individual affordances (e.g., items). According to one embodiment, when providing the affordance (2500), the electronic device (101) may determine the depth (e.g., depth menu) of the affordance (2500). For example, the electronic device (101) may determine whether menu expansion of the affordance (2500) is possible. According to one embodiment, when menu expansion is possible, the electronic device (101) may provide a menu expansion button (e.g., “>”) for each affordance for which menu expansion is possible.

According to one embodiment, a user may select one of the affordances (2510) corresponding to each target virtual object by using an input device (720). For example, the user may select a menu expansion button of one of the affordances (2510) by using the input device (720) (e.g., an indicator (900) of the input device (720)). According to one embodiment, the electronic device (101) may expand the menu of the affordance (2510) based on the user input to provide a depth menu (2550). According to one embodiment, if there is a depth menu related to the affordance (25100) selected by the input device (720), the electronic device (101) may provide the depth menu (2550) associated with (or adjacent to or around) the selected affordance (2510).

FIG. 26 is a diagram illustrating an example of an operation of providing affordance in an electronic device according to one embodiment of the present disclosure.

According to one embodiment, FIG. 26 may illustrate an example of a method for arranging list affordances considering a three-dimensional object layout.

According to one embodiment, the electronic device (101) may provide a list that takes into account an actual 3D layout when selecting a plurality of overlapping 3D affordance objects in a virtual environment. For example, the electronic device (101) may provide a list based on a 3D object layout (e.g., a 3D affordance object (2610)) so as to improve intuitiveness in selecting a plurality of overlapping 3D virtual objects.

According to one embodiment, in example <2601>, when the gap between a plurality of affordances in a virtual environment is smaller than the resolution value and it is necessary to generate affordances in the form of a list, the electronic device (101) can project the corresponding 3D affordance object (2610) onto a plane from the user's viewpoint and calculate the center of gravity (2620) of each affordance object (2610).

According to one embodiment, in example <2603>, the electronic device (101) may place the overlapping affordance objects (2610) in a three-dimensionally expanded form by keeping the sizes (e.g., size ratios) of the affordance objects (2610) the same and magnifying only the distance between each affordance object (2610) by a specified magnification multiple (e.g., approximately N times) based on the center of gravity (2620). According to one embodiment, the specified magnification multiple may be set to a minimum value (e.g., approximately 1.2 times, approximately 2 times, or approximately 3 times) that satisfies a specified condition (e.g., spacing between centers of gravity, minimum width) according to a resolution value of the input device.

FIGS. 27A, 27B, and 27C are diagrams illustrating examples of operations for providing affordance in association with a virtual object in an electronic device according to one embodiment of the present disclosure.

According to one embodiment, FIGS. 27A to 27C may illustrate examples of an operation of providing an affordance (2750) capable of supporting selection of at least one of target virtual objects in a virtual environment (2700) by associating the target virtual objects with each other in an electronic device (101) according to one embodiment. According to one embodiment, FIGS. 27A to 27C may illustrate examples of a case where multiple input devices (270) (e.g., a mouse and a user's hand or a user's left hand and a user's right hand) operate in a virtual environment (2700).

Referring to FIGS. 27A to 27C , the electronic device (101) may recognize a first input device (720) (e.g., a mouse) while displaying a virtual environment (2700) (or an execution screen (2700) of the virtual environment) through a display (e.g., a display module (160) of FIG. 1 or 5 ). According to one embodiment, FIGS. 27A to 27B may indicate a state in which a virtual object specified by the first input device (720) in the virtual environment (2700) does not exist. For example, this may indicate a state in which an affordance corresponding to the first input device (720) is not provided. According to one embodiment, when a virtual object specified by the first input device (720) exists, an affordance may be provided based on the first input device (720).

In one embodiment, a user may move a second input device (720) (e.g., a user's hand) into a virtual environment (2700) while manipulating a first input device (720). In one embodiment, the electronic device (101) may recognize the first input device (720) (e.g., a mouse) in the virtual environment (2700) while simultaneously recognizing the second input device (720) (e.g., a user's hand) entering the virtual environment (2700).

According to one embodiment, the electronic device (101) can generate and display affordances (2750) corresponding to target virtual objects that are targets of user selection based on a virtual space formed based on the second input device (720).

According to one embodiment, the electronic device (101) may display an affordance (2750) (e.g., a guide handle) related to a target virtual object in a virtual space formed around the second input device (720) when the second input device (720) stops for a predetermined period of time based on at least one virtual object among a plurality of virtual objects in the virtual environment (2700). According to one embodiment, the user may select and manipulate (e.g., move, delete, copy, and/or edit) at least one target virtual object based on the affordance (2750).

According to one embodiment, while the electronic device (101) displays an affordance (2750) centered on the second input device (720), if the first input device (720) stops for a predetermined period of time based on at least one virtual object among a plurality of virtual objects within the virtual environment (2700), the electronic device (101) may simultaneously provide another affordance (not shown) related to another target virtual object within the virtual space formed centered on the first input device (720). For example, the electronic device (101) may simultaneously provide affordances respectively corresponding to a plurality of input devices of different types.

FIGS. 28A and 28B are diagrams illustrating examples of operations for providing affordance in association with a virtual object in an electronic device according to one embodiment of the present disclosure.

According to one embodiment, FIGS. 28A and 28B may illustrate an example of an operation of providing an affordance (2850) that can support selection of at least one of target virtual objects by associating it with a target virtual object of the virtual environment (2800) when a plurality of electronic devices (101) (e.g., a first electronic device (2801), a second electronic device (2803)) are connected to each other, share a virtual environment (2800), and collaborate. For example, FIGS. 28A and 28B may illustrate an example of a scenario in which a first electronic device (2801) of a first user and a second electronic device (2803) of a second user are linked to work together in a virtual environment (2800).

Referring to FIG. 28A, a first electronic device (2801) or a second electronic device (2803) may recognize at least one input device while displaying a virtual environment (2800) (or an execution screen (2800) of the virtual environment) through their respective displays (e.g., the display module (160) of FIG. 1 or FIG. 5). According to one embodiment, in the first electronic device (2801) and the second electronic device (2803), an electronic device (101) executing the virtual environment (2800) may be a master device. According to one embodiment, an electronic device (101) that shares the virtual environment (2800) executed by the master device may be a slave device. According to one embodiment, recognition of the input device may be recognized by a master device (e.g., the first electronic device (2801) or the second electronic device (2803)) that provides the virtual environment (2800).

According to one embodiment, a master device (e.g., a first electronic device (2801)) may generate and display an affordance (2850) corresponding to each target virtual object to be selected by a user based on a virtual space formed based on an input device. According to one embodiment, the affordance (2850) generated by the master device may also be shared and displayed on a virtual environment (2800) provided through a display of a slave device (e.g., a second electronic device (2803)).

According to one embodiment, the master device may display an affordance (2850) (e.g., a guide handle) related to a target virtual object within a virtual space formed around the input device when the input device remains stationary for a predetermined period of time based on at least one virtual object among a plurality of virtual objects within the virtual environment (2800). According to one embodiment, the first user or the second user may select and manipulate (e.g., move, delete, copy, and/or edit) at least one target virtual object based on the affordance (2850).

Referring to FIG. 28b, the first user or the second user may select one of the affordances (2860) among the affordances (2850) corresponding to the target virtual object, respectively, by using the input device. According to one embodiment, the master device may determine a depth menu of the affordance (2860) when the input device is stopped (e.g., held) for a predetermined period of time designated for one of the affordances (2860). For example, the master device may determine whether menu expansion of the affordance (2860) is possible. According to one embodiment, when there is a depth menu related to the affordance (2860) selected by the input device, the master device may provide a depth menu (2870) associated with (or adjacent to or around) the selected affordance (2860). In one embodiment, a depth menu (2870) generated by a master device may also be shared and displayed on a virtual environment (2800) provided through the display of a slave device.

An operation method performed in an electronic device (101, 201) according to one embodiment of the present disclosure may include an operation of recognizing an input device (720) in a virtual environment (400, 700). According to one embodiment, the operation method may include an operation of determining a virtual space (740) for determining a target virtual object (750) based on a property of the input device (720). According to one embodiment, the operation method may include an operation of determining the target virtual object (750) based on the virtual space (740). According to one embodiment, the operation method may include an operation of generating an affordance (760) corresponding to each of the target virtual objects (750). According to one embodiment, the operation method may include an operation of displaying the affordance (760) according to a specified arrangement method.

According to one embodiment, the operation of recognizing the input device may include an operation of displaying an execution screen for the virtual environment including a plurality of virtual objects through a display, an operation of recognizing the input device while displaying the execution screen, and an operation of recognizing an indicator (900) corresponding to the input device based on recognizing the input device.

According to one embodiment, the operation of determining the virtual space may include an operation of determining a property of the input device, an operation of calculating the resolution value based on a first resolution value predefined according to the property of the input device and a second resolution value related to user noise generated by a user using the input device, an operation of calculating the virtual space having the resolution value as a diameter, and an operation of mapping a center of gravity of the virtual space based on the indicator.

According to one embodiment, the virtual space may include a virtual sphere formed with a diameter having a resolution value based on a property of the input device centered on the indicator.

According to one embodiment, the operation of determining the virtual object may include an operation of determining, based on the virtual space, the target virtual object included in the virtual space among the plurality of virtual objects of the execution screen, an operation of determining whether the target virtual object included in the virtual space satisfies a specified condition, and an operation of entering a guide generation mode for providing the affordance when the target virtual object satisfies the specified condition.

According to one embodiment, the specified condition may include a condition in which the target virtual object is a plurality of target virtual objects, at least two in number, and a distance between centers of gravity of each of the plurality of target virtual objects is less than or equal to a specified proximity distance.

According to one embodiment, the operation of displaying the affordance may include an operation of providing an object-based affordance based on a space in which the target virtual object contacts a surface of the virtual space when the target virtual object spans the virtual space, an operation of providing an object-based affordance based on a space along an extension from the target virtual object to the surface of the virtual space when the target virtual object is inside the virtual space, and an operation of providing a list-based affordance in an outer space of the virtual space when a distance between the target virtual objects is smaller than a radius of the virtual space.

According to one embodiment, the affordance may include a virtual object that supports direct selection of the target virtual object among the plurality of virtual objects.

The various embodiments of the present disclosure disclosed in this specification and drawings are merely specific examples presented to easily explain the technical content of the present disclosure and to help understand the present disclosure, and are not intended to limit the scope of the present disclosure. Therefore, the scope of the present disclosure should be interpreted to include all changes or modified forms derived based on the technical idea of the present disclosure in addition to the embodiments disclosed herein.

101, 201: Electronic Devices
120: Processor
130: Memory
160: Display module
180: Camera module
400, 700: Virtual environment (running screen)
710: Multiple Virtual Objects
720: Input Device
740: Virtual space (resolution sphere)
750: Target Virtual Object
760: Affordance
900: Indicator
760: Affordance

Claims (20)

In an electronic device (101, 201),
display (160); and
A processor (120) operatively connected to the display (160) is included, and the processor (120) comprises:
Recognizes an input device (720) in a virtual environment (400, 700),
Determine a virtual space (740) for determining a target virtual object (750) based on the properties of the input device (720),
Based on the above virtual space (740), the target virtual object (750) is determined,
Generate an affordance (760) corresponding to each of the above target virtual objects (750),
An electronic device set to display the above affordance (760) according to a specified arrangement method.
In the first paragraph, the processor,
Displaying an execution screen for the virtual environment including a plurality of virtual objects through the above display,
Recognize the input device while displaying the above execution screen,
Based on recognizing the input device, it is set to recognize an indicator (900) corresponding to the input device,
The input device is an electronic device including a device that supports user input for manipulating a virtual object in the virtual environment.
In the second paragraph, the affordance is an electronic device including a designated object that supports direct selection of the target virtual object among the plurality of virtual objects.
In the second paragraph, the processor,
Determine the properties of the above input device,
An electronic device set to produce a virtual space corresponding to the properties of the input device based on the above indicator.
In the fourth paragraph, the virtual space is
An electronic device comprising a virtual sphere formed with a diameter having a resolution value based on properties of the input device centered on the indicator.
In the fifth paragraph, the resolution value is,
A first resolution value predefined for the above input device, and
An electronic device comprising a second resolution value related to user noise generated by a user using the input device.
In the sixth paragraph, the processor,
An electronic device configured to calculate the resolution value based on the first resolution value and the second resolution value.
In the fifth paragraph, the processor,
Mapping the center of gravity of the virtual space based on the above indicator,
An electronic device set to form a virtual space having the diameter based on the center of gravity.
In the fifth paragraph, the processor,
An electronic device configured to determine the size and/or spacing of the affordance based on the above resolution value.
In the fourth paragraph, the processor,
An electronic device set to determine, based on the above virtual space, the target virtual object included in the virtual space among the plurality of virtual objects of the above execution screen.
In the 10th paragraph, the processor,
Determine whether the target virtual object included in the above virtual space satisfies a specified condition,
An electronic device set to enter a guide generation mode for providing the affordance when the target virtual object satisfies the specified condition.
In the 11th paragraph, the specified conditions are:
An electronic device including a condition that the target virtual object is a plurality of target virtual objects, at least two in number, and the distance between the centers of gravity of each of the plurality of target virtual objects is less than or equal to a specified proximity distance.
In the 11th paragraph, the processor,
Determine whether the above virtual space contains multiple target virtual objects,
When the above multiple target virtual objects are included, the center of gravity corresponding to each of the above multiple target virtual objects is determined,
Determine the distance between the centers of gravity of each of the plurality of target virtual objects,
Compare the distance between the centers of gravity with the specified proximity distance,
An electronic device set to enter the guide generation mode when at least one of the distances between the centers of gravity is less than or equal to the specified proximity distance.
In the second paragraph, the specified arrangement method is,
A first method of providing an affordance by arranging it in a space that comes into contact with the target virtual object and the surface of the virtual space;
An electronic device including a second method for providing affordances by arranging them in a list corresponding to each target virtual object outside the virtual space.
In the 14th paragraph, the processor,
When the target virtual object is over the virtual space, an object-based affordance is provided based on the space where the target virtual object is in contact with the surface of the virtual space.
If the target virtual object is inside the virtual space, an object-based affordance is provided based on the space along the extension line from the target virtual object to the surface of the virtual space.
An electronic device configured to provide a list-based affordance to an outer space of the virtual space when the distance between the target virtual objects is smaller than the radius of the virtual space.
In a method of operating an electronic device (101, 201),
An action of recognizing an input device (720) in a virtual environment (400, 700);
An operation of determining a virtual space (740) for determining a target virtual object (750) based on the properties of the input device (720);
An operation of determining the target virtual object (750) based on the virtual space (740);
An operation of generating an affordance (760) corresponding to each of the above target virtual objects (750); and
A method including an action of displaying the above affordance (760) according to a specified arrangement method.
In the 16th paragraph, the operation of recognizing the input device is,
An action of displaying an execution screen for a virtual environment including a plurality of virtual objects through a display;
An action of recognizing the input device while displaying the above execution screen, and
A method comprising an action of recognizing an indicator (900) corresponding to the input device based on recognizing the input device.
In the 17th paragraph, the operation of determining the virtual space is:
An action for determining the properties of the above input device;
An operation of calculating the resolution value based on a first resolution value predefined according to the properties of the input device and a second resolution value related to user noise generated by a user using the input device.
An operation for producing a virtual space having the above resolution value as a diameter;
Including an action of mapping the center of gravity of the virtual space based on the above indicator,
The above virtual space is,
A method comprising a virtual sphere formed with a diameter having a resolution value based on properties of the input device centered on the indicator.
In the 17th paragraph, the operation of determining the virtual object is:
An operation of determining, based on the above virtual space, the target virtual object included in the virtual space among the plurality of virtual objects of the above execution screen;
An action for determining whether the target virtual object included in the virtual space satisfies a specified condition;
If the target virtual object satisfies the specified condition, the operation of entering a guide generation mode for providing the affordance is included.
The above specified conditions are,
A method comprising a condition that the target virtual object is a plurality of target virtual objects, at least two in number, and the distance between the centers of gravity of each of the plurality of target virtual objects is less than or equal to a specified proximity distance.
In the 17th paragraph, the action of indicating the affordance is:
An action of providing an object-based affordance based on a space where the target virtual object is in contact with a surface of the virtual space, when the target virtual object is overlapping the virtual space.
An operation of providing an object-based affordance based on a space along an extension line from the target virtual object to the surface of the virtual space, when the target virtual object is inside the virtual space.
Including an action of providing a list-based affordance to an outer space of the virtual space when the gap between the target virtual objects is smaller than the radius of the virtual space,
The above affordance is,
A method including a virtual object that supports direct selection of a target virtual object among the plurality of virtual objects.

Images