KR20250150917A - An electronic apparatus and a method of operating the electronic apparatus - Google Patents

Abstract

The disclosed electronic device includes a game application corresponding to the execution of cloud game content and a game driving unit for executing the game application, and receives input data while executing the game application by the game driving unit, and when an application corresponding to an activated window corresponds to the game application, controls to transmit the received input data to a server transmitting the cloud game content or to provide the input data to the game application depending on whether the state of the player for executing the cloud game content is a play state.

Description

{An electronic apparatus and a method of operating the electronic apparatus}

Various embodiments relate to electronic devices and methods of operating the same, and more particularly, to electronic devices for executing cloud gaming applications and methods of operating the same.

As networks advance, enabling the rapid transmission of large amounts of data without delay, various industries are thriving. One such industry is cloud computing. Cloud gaming services, which integrate cloud computing into the gaming industry, are gaining popularity. The number of providers is increasing, and the market size is growing globally.

Cloud gaming is a gaming service where key computations, such as graphics processing, required for games are performed on cloud servers. User inputs from user devices, such as smartphones or smart TVs, are transmitted to the server, and the rendered game screen is streamed and displayed on the user device. Because cloud servers handle these computational processes, which typically require expensive equipment, cloud gaming allows users to de-clutter their hardware. Therefore, users can enjoy high-spec games anytime, anywhere at a low cost, using devices capable of receiving streaming video or transmitting input signals.

In one embodiment, an electronic device and an operating method thereof are provided that enable efficient use of network resources by eliminating logic for unnecessarily transmitting data and transmitting data directly from the game driving unit to the game server in order to reduce the work of the game driving unit.

According to one embodiment, an electronic device may include a communication interface, a memory storing a game application corresponding to execution of cloud game content and a game driver for executing the game application, and a processor executing one or more instructions stored in the memory. The processor may receive input data while executing the game application by executing one or more instructions by the game driver. The processor may control the received input data to be transmitted to a server transmitting the cloud game content or provided to the game application depending on whether the state of a player for executing the cloud game content is a play state when an application corresponding to an activated window corresponds to the game application by executing one or more instructions.

According to one embodiment, a method for operating an electronic device 100 including a game application corresponding to execution of cloud game content and a game driver for executing the game application may include an operation of receiving input data while executing the game application by the game driver. The method may include an operation of transmitting the received input data to a server transmitting the cloud game content or providing the received input data to the game application depending on whether the state of a player for execution of the cloud game content is a play state when an application corresponding to an activated window corresponds to the game application.

According to one embodiment, a non-transitory computer-readable recording medium storing one or more instructions, wherein the one or more instructions are executed by a processor of an electronic device 100 including a game application corresponding to execution of cloud game content and a game driving unit for executing the game application, so that the electronic device receives input data while executing the game application by the game driving unit, and when an application corresponding to an activated window corresponds to the game application, transmits the received input data to a server transmitting the cloud game content or provides the received input data to the game application depending on whether the state of a player for execution of the cloud game content is a play state.

According to the disclosed embodiment, the electronic device and its operating method can reduce latency and improve overall performance by efficiently performing frequent and continuous transmission tasks of small data performed in a game driving unit that causes a bottleneck in the execution of a cloud game.

The present invention can be readily understood by the following detailed description and its accompanying drawings, wherein reference numerals refer to structural elements.
FIG. 1 is a reference diagram for explaining the operation of a server and an electronic device according to one embodiment.
FIG. 2 illustrates an example of a system including an electronic device, an input device, and a server according to one embodiment.
FIG. 3 is an example of a block diagram of an electronic device according to one embodiment.
FIG. 4 illustrates an example functional block diagram of an electronic device for explaining the operation of the electronic device according to one embodiment.
FIG. 5 is a flowchart according to an example of a method of operating an electronic device according to one embodiment.
FIG. 6 is a reference diagram for explaining an operation of selecting a game application in a user interface that allows selection of a game application according to one embodiment.
FIG. 7 is a reference diagram for explaining an operation of selecting a game content item in a game application user interface according to one embodiment.
Figure 8 illustrates an example of a media player pipeline according to one embodiment.
FIG. 9 illustrates an example of a flowchart of a method of operating an electronic device according to one embodiment.
FIG. 10 illustrates an example of a flowchart of a method of operating an electronic device during execution of a game application according to one embodiment.
FIG. 11 illustrates an example of a flowchart of a method of operating an electronic device during execution of a game application according to an embodiment.
FIG. 12 is a reference diagram for explaining an example of displaying a UI during game play according to one embodiment.
FIG. 13 illustrates an example of a flowchart of a method of operating an electronic device during execution of a game application according to one embodiment.
Figure 14 illustrates an example of a quick panel according to one embodiment.
Figure 15 illustrates an example of a user interface according to one embodiment.
Figure 16 illustrates an example of a window stack according to one embodiment.

The terms used in this specification will be briefly explained, and the present invention will be described in detail.

The terms used in this invention have been selected from widely used, current terms, taking into account the functions of the invention. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, in which case their meanings will be described in detail in the relevant description of the invention. Therefore, the terms used in this invention should not be defined simply as names, but rather based on their inherent meanings and the overall content of the invention.

When a part of the specification is said to "include" a component, this does not exclude other components, but rather implies the inclusion of other components, unless otherwise specifically stated. Furthermore, terms such as "part," "module," etc., used throughout the specification refer to a unit that processes at least one function or operation, which may be implemented in hardware, software, or a combination of hardware and software.

Below, with reference to the attached drawings, embodiments of the present invention are described in detail so that those skilled in the art can easily implement the present invention. However, the present invention can be implemented in various different forms and is not limited to the embodiments described herein. In the drawings, parts irrelevant to the description have been omitted to clearly explain the present invention, and similar parts have been designated with similar reference numerals throughout the specification.

The term "user" in this disclosure means a person who controls a function or operation of a computing device or electronic device using a control device, and may include a viewer, an administrator, or an installer.

In this disclosure, a "game application" refers to an application that enables the playing of cloud games. The user interface upon execution of the game application may provide items that enable the execution of one or more cloud game contents. The game application may primarily include a portion that provides a user interface and processes user input upon execution of such a game application, and a portion that plays audio and video data received in a streaming manner from a cloud server.

FIG. 1 is a reference diagram for explaining the operation of a server and an electronic device according to one embodiment.

Referring to FIG. 1, server 300 may execute game content upon a request from electronic device 100 and transmit a result screen according to the execution of the game content to the electronic device 100 via a communication network. For example, when server 300 receives a request to execute game content from electronic device 100, server 300 may execute the game content and transmit a result screen or result image rendered according to the execution of the game content to the electronic device 100 via a communication network.

Electronic device 100 transmits a request for executing game content to server 300, and according to this execution request, receives a result screen according to the execution of the game content from server 300 and can display the received result screen 30 on a display.

The input device 200 can transmit an operation command to the electronic device 100 for controlling a game screen, which is a result of executing game content displayed on the electronic device 100. A user can operate the input device 200 to control the displayed game screen while viewing the game screen displayed on the electronic device 100, and the input device 200 can transmit an operation command according to such user operation to the electronic device 100. The server 300 can receive a content operation command from the input device 200 from the electronic device 100, execute the game content according to the content operation command, and transmit a game content execution result screen to the electronic device 100.

Because cloud gaming uses streaming, player input signals are not immediately reflected on the device, resulting in latency. Cloud gaming services need to provide a gaming experience comparable to what regular users experience on high-performance PCs and consoles. However, unlike typical OTT (Over the Top) video content, latency can be a problem in games like fighting games and FPS (First-Person Shooters), where fast response times are crucial. Therefore, minimizing latency for high responsiveness is a critical challenge in cloud gaming. Latency can be caused by a variety of factors, one of which is input signal processing delay.

Electronic device 100 may include a game application and a game driver for cloud gaming. The game application provides a game application user interface that enables game content selection and user settings, and manages the playback of game content received from server 300. The game driver may process or transmit user input between the game application and the interface module. To achieve this, the game driver shares resources for tasks such as audio/video data processing, network events, and video output handled by the interface module, which can cause bottlenecks. Most web-based services, currently in service form, handle UI (User Interface) event processing and rendering in addition to the aforementioned tasks on a single thread. Furthermore, game controllers, among input devices, require periodic reading and processing of values rather than event-based input, which can exacerbate bottlenecks. This bottleneck in the game driver can increase latency in cloud gaming. Therefore, efficient management of the game driver's operations is desirable.

According to an embodiment of the present disclosure, when processing user input in a game driving unit, the game driving unit can efficiently operate and reduce latency by appropriately providing the user input to a destination for processing the user input while considering the environment in which the game is executed.

According to one embodiment, the electronic device 100 may receive input data while executing a game application that provides a user interface for executing cloud game content.

According to one embodiment, the electronic device 100 can identify an application corresponding to an activated window.

According to one embodiment, the electronic device 100 can identify whether the state of the player for playing the cloud game content is a play state by identifying that the application corresponding to the activated window is the game application.

According to one embodiment, the electronic device 100 may transmit the input data to a server transmitting the cloud game content or provide the input data to the game application depending on whether the state of the player is the play state.

According to one embodiment, the electronic device 100 may transmit the input data to a server transmitting the cloud game content when the state of the player is identified as being in the play state.

In one embodiment, the electronic device 100 may provide the input data to the game application so that the game application can operate based on the user input when the state of the player is identified as not being in the play state.

According to one embodiment, the electronic device 100 may receive status information from the game application indicating that a user interface based on the game application is being displayed while the player's status is in the play state.

According to one embodiment, the electronic device 100 may provide the input data to the game application based on the state information so that the game application can operate based on the user input.

According to one embodiment, the electronic device 100 may process input data differently depending on whether focus is located within the system user interface, as the electronic device 100 identifies that the application corresponding to the activated window is an application corresponding to the system user interface.

According to one embodiment, the electronic device 100 can identify whether focus is located within the system user interface.

According to one embodiment, the electronic device 100 may transmit the input data to the system UI module upon identifying that focus is located within the system user interface.

In one embodiment, the electronic device 100 may transmit the input data to the server upon identifying that focus is not located within the system user interface.

In one embodiment, the electronic device 100 can identify an application at the top layer in the window stack as the application corresponding to the activated window.

According to one embodiment, the electronic device 100 can generate the player for executing the cloud game content, and identify whether the state of the player is the play state based on the operating state of at least one of an audio decoder or a video decoder of the generated player.

According to one embodiment, the electronic device 100 can create the player based on metadata of the cloud game content received from the server.

According to one embodiment, the electronic device 100 transmits the input data to the server, receives a result screen of executing the cloud game content based on the input data from the server, and provides the input data to the game application, thereby controlling the game application to operate based on the input data.

FIG. 2 illustrates an example of a system including an electronic device, an input device, and a server according to one embodiment.

Referring to FIG. 2, the system may include an electronic device 100, an input device 200, and a server 300 connected to a communication network.

The electronic device 100 is a device that can display images or data according to a user's request, and may include a communication unit 110, a display 120, a memory 130, and a processor 140.

The communication unit 110 may perform communication with at least one external device. The communication unit 110 may include a communication circuit. Here, "communication" may refer to an operation of transmitting and/or receiving data, signals, requests, and/or commands.

The communication unit 110 can perform wired or wireless communication with at least one external device. The external device may be a server 300 or an input device 200, etc.

For example, the communication unit 110 may include at least one of a communication module, a communication circuit, a communication device, an input/output port, and an input/output plug for performing wired or wireless communication with at least one external device.

For example, the communication unit 110 may include at least one wireless communication module, wireless communication circuit, or wireless communication device that performs wireless communication with at least one external device.

For example, the communication unit 110 may include a short-range communication module, such as an IR (infrared) communication module, that can receive control commands from a remote controller located in close proximity, such as an input device 200. In this case, the communication unit 110 may receive a control signal from the remote control device.

As another example, the communication unit 110 may include at least one communication module that performs communication according to a wireless communication standard such as Bluetooth, Wi-Fi, BLE (Bluetooth Low Energy), NFC/RFID, Wi-Fi Direct, UWB, or ZIGBEE. Alternatively, the communication unit 110 may further include a communication module that performs communication with a server to support long-distance communication according to a long-distance communication standard. For example, the communication unit 110 may include a communication module that performs communication through a network for Internet communication. In addition, the communication unit 110 may include a communication module that performs communication through a communication network according to a communication standard such as 3G, 4G, 5G, and/or 6G.

As another example, the communication unit 110 may include at least one port for connecting to an external device via a wired cable in order to communicate with the external device via a wired connection. For example, the communication unit 110 may include at least one of an HDMI port (High-Definition Multimedia Interface port), a component jack, a PC port, and a USB port. Accordingly, the communication unit 110 may communicate with a wired external device via at least one port. Here, the port may refer to a physical device configuration into which a cable, communication line, or plug can be connected or inserted.

As described above, the communication unit 110 may include at least one support element for supporting communication between the electronic device 100 and an external device. Here, the support element may include the communication module, the communication circuit, the communication device, the port (for input/output of data), the cable port (for input/output of data), the plug (for input/output of data), etc., as described above. For example, the at least one support element included in the communication unit 110 may include an Ethernet communication module, a Wi-Fi communication module, a Bluetooth communication module, an IR communication module, a USB port, a tuner (or a broadcast receiver), an HDMI port, a DP (display port), a DVI (digital visual interface) port, etc.

The display 120 can output images or data processed in the electronic device 100.

The memory 130 can store a program for processing and controlling the processor 140, and can store data input to or output from the electronic device 100. In addition, the memory 130 can store data necessary for the operation of the electronic device 100.

The memory 130 may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a RAM (Random Access Memory), a SRAM (Static Random Access Memory), a ROM (Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a PROM (Programmable Read-Only Memory), a magnetic memory, a magnetic disk, and an optical disk.

The processor 140 controls the overall operation of the electronic device 100. For example, the processor 140 may perform the functions of the electronic device 100 described in the present disclosure by executing one or more instructions stored in the memory 130.

Processor 140 may include various processing circuits and/or multiple processors. For example, the term "processor" as used herein, including in the claims, may include various processing circuits, including at least one processor. At least one processor, one or more processors may be individually and/or collectively configured to perform various functions described herein in a distributed fashion. As used herein, "processor," "at least one processor," and "one or more processors" may be configured to perform multiple functions. However, these terms encompass, for example, but not limited to, situations where one processor performs some of the functions and other processor(s) perform other parts of the functions, and situations where a single processor may perform all of the functions. Furthermore, the at least one processor may include a combination of processors that perform various of the disclosed functions in a distributed manner. At least one processor may execute program instructions to achieve or perform various functions.

In an embodiment of the present disclosure, the processor 140 can store one or more instructions in an internally provided memory, and control the execution of operations of the display device by executing one or more instructions stored in the internally provided memory. That is, the processor 140 can perform a predetermined operation by executing at least one instruction or program stored in the internal memory or memory 130 provided within the processor 140.

According to one embodiment, the processor 140 may perform operations of the electronic device 100 disclosed in the present disclosure by executing one or more instructions stored in the memory 130.

According to one embodiment, the processor 140 may receive input data while executing a game application that provides a user interface for executing cloud game content by executing one or more instructions stored in the memory 130.

According to one embodiment, the processor 140 may identify an application corresponding to an activated window by executing one or more instructions stored in the memory 130.

According to one embodiment, the processor 140 can identify whether the state of the player for executing or playing the cloud game content is a play state by identifying that the application corresponding to the activated window is the game application by executing one or more instructions stored in the memory 130.

According to one embodiment, the processor 140 may transmit the input data to a server transmitting the cloud game content or provide the input data to the game application, depending on whether the state of the player is the play state, by executing one or more instructions stored in the memory 130.

According to one embodiment, the processor 140 may transmit the input data to a server transmitting the cloud game content by executing one or more instructions stored in the memory 130, upon identifying the player's state as being in the play state.

In one embodiment, the processor 140 may provide the input data to the game application so that the game application can operate based on the user input by executing one or more instructions stored in the memory 130 when the state of the player is identified as not being the play state.

According to one embodiment, the processor 140 may receive state information from the game application indicating that a user interface based on the game application is being displayed while the state of the player is in the play state by executing one or more instructions stored in the memory 130.

According to one embodiment, the processor 140 may provide the input data to the game application so that the game application can operate based on the user input based on the state information by executing one or more instructions stored in the memory 130.

According to one embodiment, the processor 140 may process input data differently depending on whether focus is located on the system user interface, by identifying that the application corresponding to the activated window is an application corresponding to the system user interface by executing one or more instructions stored in the memory 130.

In one embodiment, the processor 140 may identify whether focus is located on the system user interface by executing one or more instructions stored in the memory 130.

According to one embodiment, the processor 140 may transmit the input data to the system UI module upon identifying that focus is located on the system user interface by executing one or more instructions stored in the memory 130.

According to one embodiment, the processor 140 may transmit the input data to the server by executing one or more instructions stored in the memory 130 when it is determined that the focus is not located on the system user interface.

According to one embodiment, the processor 140 may identify an application at the top layer in the window stack as the application corresponding to the activated window by executing one or more instructions stored in the memory 130.

According to one embodiment, the processor 140 may generate the player for executing the cloud game content by executing one or more instructions stored in the memory 130, and identify whether the state of the player is the play state based on the operational state of at least one of an audio decoder or a video decoder of the generated player.

According to one embodiment, the processor 140 may generate the player based on metadata of the cloud game content received from the server by executing one or more instructions stored in the memory 130.

According to one embodiment, the processor 140 may transmit the input data to the server by executing one or more instructions stored in the memory 130, thereby receiving a result screen of executing the cloud game content based on the input data from the server, and may control the game application to operate based on the input data by providing the input data to the game application.

The electronic device 100 may be any type of device that performs a function, including a processor and memory. The electronic device 100 may be a fixed or portable device. For example, the electronic device 100 may be a device that has a display and can display image content, video content, game content, graphic content, etc. The electronic device 100 may output or display images or content received from the server 300. The electronic device 100 may include various types of electronic devices that can receive and output content, such as televisions such as network TVs, smart TVs, Internet TVs, web TVs, and IPTVs; computers such as desktops, laptops, and tablets; smartphones, cellular phones, game players, music players, video players, medical equipment, home appliances, and various other smart devices. The electronic device 100 may be referred to as a display device in that it receives and displays content, and may also be referred to as a content receiving device, a sync device, an electronic device, a computing device, etc.

The block diagram of the electronic device 100 illustrated in FIG. 2 is a block diagram for one embodiment. Each component of the block diagram may be integrated, added, or omitted depending on the specifications of the electronic device 100 actually implemented. For example, two or more components may be combined into one component, or one component may be subdivided into two or more components, as needed. In addition, the functions performed by each block are for the purpose of explaining embodiments, and the specific operations or devices thereof do not limit the scope of the present invention.

Although FIG. 2 illustrates the electronic device 100 as including a display, the present invention is not limited thereto. For example, the electronic device 100 may be a device that provides content to an external display device including a display. The content may include, but is not limited to, images, videos, audio, text, games, applications, etc. The electronic device 100 may include a set-top box (STB), a Blu-ray Disc (Blu-ray Disc) player, a Digital Versatile Disk (DVD) player, a game device, a digital camera, a camcorder, a streaming device, a home theater, etc. In this case, the electronic device 100 may be connected to the external display device through an input/output port such as an HDMI port, and configured to transmit video/audio signals to the external display device. Alternatively, the electronic device 100 may be connected to the external display device through wired communication, short-range wireless communication such as wireless LAN (W-LAN), Wi-Fi, or Bluetooth, or long-range wireless communication.

Now let's describe input device 200.

The input device 200 may include a communication unit 210, a user input unit 220, a memory 230, and a processor 240. However, the electronic device 200 may be implemented with more components than the illustrated components and is not limited to the examples described above.

The communication unit 210 may include various communication circuits included in one or more modules that enable wireless communication between the input device 200 and a wireless communication system or between the input device 200 and a network in which another device is located. In one embodiment, the communication unit 210 may perform communication with the electronic device 100 according to a short-range communication technology. The short-range communication technology may include, for example, Bluetooth communication, Wi-Fi communication, infrared communication, etc. In one embodiment, the communication unit 210 may perform communication with the server 300 according to an Internet protocol.

The user input unit 220 may be any form of interface means capable of receiving user input. For example, the user input unit 220 may include an operation button arranged on a part of the input device 200 capable of receiving user input, a touch-sensitive display configured to detect touch input, a microphone capable of receiving user voice input, etc.

Memory 230 can store a program for processing and controlling the processor 240, and can store data input to or output from the input device 200.

The memory 230 may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a RAM (Random Access Memory), a SRAM (Static Random Access Memory), a ROM (Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a PROM (Programmable Read-Only Memory), a magnetic memory, a magnetic disk, and an optical disk.

Processor 240 controls the overall operation of input device 200. Processor 240 may include various processing circuits. For example, processor 240 may perform the functions of input device 200 described in this disclosure by executing one or more instructions stored in memory 230.

In an embodiment of the present disclosure, the processor 240 can store one or more instructions in an internally provided memory, and control the execution of the one or more instructions stored in the internally provided memory so that the aforementioned operations are performed. That is, the processor 240 can perform a predetermined operation by executing at least one instruction or program stored in an internal memory provided within the processor 240 or in the memory 230.

According to one embodiment, the processor 240 may perform a communication connection with the electronic device 100 using a short-range wireless communication technology by executing one or more instructions stored in the memory 230. The short-range wireless communication technology may include Bluetooth communication technology or Wi-Fi Direct technology.

According to one embodiment, the processor 240 may control the communication unit 210 to establish a communication connection with the electronic device 100 by executing one or more instructions stored in the memory 230.

According to one embodiment, the processor 240 may transmit an input signal corresponding to a user input received through the user input unit 220 to the electronic device 100 by executing one or more instructions stored in the memory 230.

The input device 200 may be any type of device that performs a function, including a processor and memory. The input device 200 may include various electronic devices, such as a remote controller, a game controller, or a smartphone.

Meanwhile, the block diagram of the input device 200 illustrated in FIG. 2 is a block diagram for one embodiment. Each component of the block diagram may be integrated, added, or omitted depending on the specifications of the input device 200 actually implemented. For example, two or more components may be combined into one component, or one component may be subdivided into two or more components, as needed. In addition, the functions performed by each block are for the purpose of explaining embodiments, and the specific operations or devices thereof do not limit the scope of the present invention.

Now let's describe server 300.

Server 300 may include a communication unit 310, memory 320, and processor 330. However, server 300 may be implemented with more components than those illustrated, and is not limited to the examples described above. For example, server 300 may include a separate image processing unit for image processing of application images executed on server 300.

The communication unit 310 may include various communication circuits included in one or more modules that enable wireless communication between the server 300 and a wireless communication system or between the server 300 and a network in which other devices are located. In one embodiment, the communication unit 310 may perform communication with the electronic device 100 according to the Internet Protocol.

Memory 320 can store a program for processing and controlling the processor 330, and can store data input to or output from the server 300.

The memory 320 may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a RAM (Random Access Memory), a SRAM (Static Random Access Memory), a ROM (Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a PROM (Programmable Read-Only Memory), a magnetic memory, a magnetic disk, and an optical disk.

Processor 330 controls the overall operation of server 300. Processor 330 may include various processing circuits. For example, processor 330 may perform the functions of server 300 described in this disclosure by executing one or more instructions stored in memory 320.

In an embodiment of the present disclosure, the processor 330 can store one or more instructions in an internally provided memory, and control the execution of the one or more instructions stored in the internally provided memory so that the aforementioned operations are performed. That is, the processor 330 can perform a predetermined operation by executing at least one instruction or program stored in an internal memory provided within the processor 330 or in the memory 320.

According to one embodiment, the processor 330 may receive a request to execute game content from the electronic device 100 by executing one or more instructions stored in the memory 320, and transmit a game screen resulting from executing the requested game content to the electronic device 100.

According to one embodiment, the processor 330 may execute one or more instructions stored in the memory 320 to receive an input signal from the electronic device 100, execute game content based on the received input signal, and transmit a game screen resulting from the execution to the electronic device 100.

The block diagram of server 300 illustrated in FIG. 2 is a block diagram for one embodiment. Each component in the block diagram may be integrated, added, or omitted depending on the specifications of the server 300 actually implemented. For example, two or more components may be combined into a single component, or a single component may be subdivided into two or more components, as needed. Furthermore, the functions performed by each block are intended to illustrate embodiments, and their specific operations or devices do not limit the scope of the present invention.

FIG. 3 is an example of a block diagram of an electronic device according to one embodiment.

Referring to FIG. 3, the electronic device 100 may include a communication unit 110, a display 120, a memory 130, a processor 140, an image processing unit 150, an audio processing unit 160, an audio output unit 170, a receiving unit 180, and a detection unit 190.

The communication unit 110 may include various communication circuits included in one or more modules that enable wireless communication between the electronic device 100 and a wireless communication system or between the electronic device 100 and a network in which another electronic device is located. For example, the communication unit 110 may include a mobile communication module 111, a wireless Internet module 112, and a short-range communication module 113.

The mobile communication module 111 transmits and receives wireless signals with at least one of a base station, an external terminal, and a server on a mobile communication network. The wireless signals may include various types of data, such as voice call signals, video call signals, or text/multimedia message transmission and reception.

The wireless Internet module 112 refers to a module for wireless Internet access, which may be built into or external to the device. Wireless Internet technologies that may be used include WLAN (Wireless LAN) (Wi-Fi), Wibro (Wireless broadband), Wimax (World Interoperability for Microwave Access), and HSDPA (High Speed Downlink Packet Access). Through the wireless Internet module 122, the device can establish a Wi-Fi P2P (Peer to Peer) connection with other devices.

Short-range communication module 113 refers to a module for short-range communication. Short-range communication technologies that can be used include Bluetooth, BLE (Bluetooth Low Energy), RFID (Radio Frequency Identification), IrDA (Infrared Data Association), UWB (Ultra Wideband), and ZigBee.

Display 120 can display video signals received from server 300 on the screen.

Memory 130 can store programs related to the operation of the electronic device 100 and various data generated during the operation of the electronic device 100.

Memory 130 may store at least one instruction. Furthermore, memory 140 may store at least one instruction to be executed by processor 150. Furthermore, memory 140 may store at least one program to be executed by processor 150. Furthermore, memory 150 may store an application for providing a specific service.

Specifically, the memory 130 may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a RAM (Random Access Memory), a SRAM (Static Random Access Memory), a ROM (Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a PROM (Programmable Read-Only Memory), a magnetic memory, a magnetic disk, and an optical disk.

In one embodiment, memory 130 may store a game application 131 and a game driver 132.

The processor 140 controls the overall operation of the electronic device 100. For example, the processor 140 may perform the functions of the electronic device 100 described in the present disclosure by executing one or more instructions stored in the memory 130.

Processor 140 may include various processing circuits and/or multiple processors. For example, the term "processor" as used herein, including in the claims, may include various processing circuits, including at least one processor. At least one processor, one or more processors may be individually and/or collectively configured to perform various functions described herein in a distributed fashion. As used herein, "processor," "at least one processor," and "one or more processors" may be configured to perform multiple functions. However, these terms encompass, for example, but not limited to, situations where one processor performs some of the functions and other processor(s) perform other parts of the functions, and situations where a single processor may perform all of the functions. Furthermore, the at least one processor may include a combination of processors that perform various of the disclosed functions in a distributed manner. At least one processor may execute program instructions to achieve or perform various functions.

In an embodiment of the present disclosure, the processor 140 can store one or more instructions in an internally provided memory, and control the execution of operations of the display device by executing one or more instructions stored in the internally provided memory. That is, the processor 140 can perform a predetermined operation by executing at least one instruction or program stored in the internal memory or memory 130 provided within the processor 140.

According to one embodiment, the processor 140 may receive input data while executing a game application that provides a user interface for executing cloud game content by the game driver 132 by executing one or more instructions stored in the memory 130.

According to one embodiment, the processor 140 may identify an application corresponding to an activated window by the game driver 132 by executing one or more instructions stored in the memory 130.

According to one embodiment, the processor 140 can identify whether the state of the player for playing the cloud game content is a play state by executing one or more instructions stored in the memory 130, thereby identifying, by the game driver 132, that the application corresponding to the activated window is the game application.

According to one embodiment, the processor 140 may transmit the input data to a server transmitting the cloud game content or provide the input data to the game application, depending on whether the state of the player is the play state, by executing one or more instructions stored in the memory 130, by the game driver 132.

According to one embodiment, the processor 140 may transmit the input data to the server transmitting the cloud game content by executing one or more instructions stored in the memory 130, as determined by the game driver 132 that the player's state is in the play state.

According to one embodiment, the processor 140 may provide the input data to the game application so that the game application can operate based on the user input when the state of the player is identified by the game driver 132 as not being in the play state by executing one or more instructions stored in the memory 130.

According to one embodiment, the processor 140 may receive, by the game driver 132, status information from the game application indicating that a user interface based on the game application is being displayed while the player's status is in the play state, by executing one or more instructions stored in the memory 130.

According to one embodiment, the processor 140 may provide the input data to the game application by executing one or more instructions stored in the memory 130, based on the state information, by the game driver 132, so that the game application may operate based on the input data.

According to one embodiment, the processor 140 may process input data differently depending on whether focus is located on the system user interface, by executing one or more instructions stored in the memory 130, and by the game driver 132 identifying that the application corresponding to the activated window is an application corresponding to the system user interface.

According to one embodiment, the processor 140 may, by executing one or more instructions stored in the memory 130, determine, by the game driver 132, whether focus is located on the system user interface.

According to one embodiment, the processor 140 may transmit the input data to the system UI module by executing one or more instructions stored in the memory 130, upon the game driver 132 identifying that the system user interface has focus.

According to one embodiment, the processor 140 may transmit the input data to the server by executing one or more instructions stored in the memory 130, upon the game driver 132 identifying that the system user interface does not have focus.

According to one embodiment, the processor 140 can, by executing one or more instructions stored in the memory 130, cause the game driver 132 to identify the application in the top layer as the application corresponding to the activated window by referencing the window stack.

According to one embodiment, the processor 140 may generate the player for executing the cloud game content by executing one or more instructions stored in the memory 130, and identify whether the state of the player is the play state based on the operational state of at least one of an audio decoder or a video decoder of the generated player.

According to one embodiment, the processor 140 may generate the player based on metadata of the cloud game content received from the server by the game driver 132 by executing one or more instructions stored in the memory 130.

According to one embodiment, the processor 140 may control the game application to operate based on the input data by transmitting the input data to the server through the game driver 132 by executing one or more instructions stored in the memory 130, thereby receiving a result screen of executing the cloud game content based on the input data from the server, and by providing the input data to the game application.

The image processing unit 150 can process an image signal received from the receiving unit 110 or the communication unit 120 and output it to the display 130 under the control of the processor 140. The image processing unit 150 can include various image processing circuits.

The audio processing unit 160 may, under the control of the processor 140, convert an audio signal received from the receiving unit 110 or the communication unit 120 into an analog audio signal and output it to the audio output unit 180. The audio processing unit 160 may include various audio processing circuits.

The audio output unit 170 can output audio (e.g., voice, sound) input through the communication unit 110 or the receiver unit 180. In addition, the audio output unit 170 can output audio stored in the memory 130 under the control of the processor 140. The audio output unit 170 can include at least one of a speaker, a headphone output terminal, or an S/PDIF (Sony/Philips Digital Interface:) output terminal, or a combination thereof.

The receiving unit 180 can receive video (e.g., moving images, etc.), audio (e.g., voice, music, etc.), and additional information (e.g., EPG, etc.) from the outside of the electronic device 100 under the control of the processor 140. The receiving unit 180 may include one of an HDMI port (High-Definition Multimedia Interface port, 181), a component jack (component jack, 182), a PC port (PC port, 183), and a USB port (USB port, 184), or a combination of one or more thereof. In addition to the HDMI port, the receiving unit 180 may further include a DisplayPort (DP), Thunderbolt, or MHL (Mobile High-Definition Link).

The detection unit 190 detects the user's voice, the user's image, or the user's interaction, and may include a microphone, a camera unit, and a light receiving unit.

The microphone receives the user's spoken voice. The microphone can convert the received voice into an electrical signal and output it to the processor 140. The user's voice may include, for example, a voice corresponding to a menu or function of the electronic device 100.

The camera can receive images (e.g., consecutive frames) corresponding to a user's motion, including a gesture, within the camera's recognition range. The processor 140 can use the recognition results of the received motion to select a menu displayed on the electronic device 100 or perform a control corresponding to the motion recognition results.

The optical receiver receives an optical signal (including a control signal) from an external control device. The optical receiver can receive an optical signal corresponding to a user input (e.g., touch, press, touch gesture, voice, or motion) from the control device. A control signal can be extracted from the received optical signal under the control of the processor 140.

FIG. 4 illustrates an example functional block diagram of an electronic device for explaining the operation of the electronic device according to one embodiment.

Referring to FIG. 4, server 300 may include a game server 301 that runs game content and a media streaming server 302 that streams a game screen resulting from the running of the game content.

The game server 301 can receive a game content execution request from the game driver 132, execute the requested game content, and transmit audio/video data resulting from the execution to the media streaming server 302. In addition, the game server 301 can receive a user input for game control from the game driver 132 after the game is executed, execute the game content based on the received user input, and transmit audio/video data resulting from the execution to the media stream server 302. The user input that the game server 301 receives from the game driver 132 can also include time information, such as information on which button of the input device 200 was pressed, time information on selecting a specific button of the input device 200, or time information on how long a specific button was pressed. Part or all of the code of the game application of the electronic device 100 can be included in the game server 301. Although FIG. 4 illustrates that the game server 301 and the media streaming server 302 are configured as separate servers, the game server 301 and the media streaming server 302 may be configured as a single server.

The electronic device 100 may include a game application 131, a game driving unit 132, and an output unit 121.

Game application 131 may provide a user interface for selecting or setting game content and may play a video corresponding to a game screen received from server 300.

The game driver 132 is a process that runs the game application 131 and can generally handle communication with the server 300, media processing, event transmission from input devices, and rendering of the user interface of the game application.

The output unit 121 can serve to output audio data and video data corresponding to game content received from the server 300.

In general, whether the input data of the input device 200 is used to control the UI of the game application or transmitted to the server 300 to be reflected in the play of the game content depends on the scenario of the game application 131, so the game driver 132 operates to transmit the input data received from the input device 200 to the game application 131 without any particular judgment. In other words, since the game application 131 has a UI and processes the user's operations, the game driver 132 only processes the functions called by the game application and does not know the scenario of the game application 131. Therefore, the game driver 132 has no choice but to transmit the input data coming from the input device 200 to the game application 131 to be processed by the game application 131. However, in the case of a cloud game application, the input data is largely used in one of two ways: to control the UI of the game application or to transmit it to the server 300 to have the server 300 use it to execute the game content. Even when input data from input device 200 is transmitted to server 300, conventionally, game driver 132 transmits it to game application 131, and then, depending on the scenario of game application 131, it is transmitted from game application 131 to game driver 132 again, and only then can game driver 132 transmit the input data to server 300. Therefore, input data to be transmitted to game server 300 is not transmitted directly from game driver 132 to server 300, but may be transmitted to game application 131 and then to game driver 132 before being transmitted to server 300, which inevitably causes input delay.

Accordingly, in the disclosed embodiment, input delay can be reduced by having the game driver 132 determine how to process input data received from the input device 200, thereby directly transmitting the input data to an appropriate destination that will consume the input data, i.e., the server 300 or the game application 131.

The game driver 132 can transmit input for game play to the game server 301. The input for game play can also include time information, such as information about which button of the input device 200 was pressed, information about the time a specific button of the input device 200 was selected, or information about how long a specific button was pressed.

In one embodiment, the game driver 132 may receive input data while executing a game application that provides a user interface for executing cloud game content.

In one embodiment, the game driver 132 may identify an application corresponding to an activated window.

According to one embodiment, the game driver 132 can identify whether the player's status for playing the cloud game content is a play status by identifying that the application corresponding to the activated window is the game application.

According to one embodiment, the game driver 132 may transmit the input data to a server transmitting the cloud game content or provide the input data to the game application depending on whether the player's status is the play status.

According to one embodiment, the game driver 132 may transmit the input data to a server transmitting the cloud game content when the player's state is identified as being in the play state.

In one embodiment, the game driver 132 may provide the input data to the game application so that the game application can operate based on the user input when the state of the player is identified as not being the play state.

According to one embodiment, the game driver 132 may receive status information from the game application indicating that a user interface based on the game application is being displayed while the player's status is in the play state.

According to one embodiment, the game driver 132 may provide the input data to the game application based on the status information so that the game application can operate based on the user input.

In one embodiment, the game driver 132 may process input data differently depending on whether the system user interface has focus, as the application corresponding to the activated window is identified as an application corresponding to the system user interface.

In one embodiment, the game driver 132 may identify whether the system user interface has the focus.

In one embodiment, the game driver 132 may transmit the input data to the system UI module when the system user interface is identified as having the focus.

In one embodiment, the game driver 132 may transmit the input data to the server when the system user interface is identified as not having the focus.

In one embodiment, the game driver 132 may identify an application corresponding to the activated window by referencing the window stack.

FIG. 5 is a flowchart according to an example of a method of operating an electronic device according to one embodiment.

Referring to FIG. 5, in operation 510, the electronic device 100 may receive input data while executing a game application corresponding to execution of cloud game content.

In one embodiment of the present disclosure, a game application may represent an application that provides a user interface for executing and configuring one or more game contents and also operates a media player that plays a video stream corresponding to the game contents.

FIG. 6 is a reference diagram for explaining an operation of selecting a game application in a user interface that allows selection of a game application according to one embodiment.

Referring to FIG. 6, the electronic device 100 may provide a user interface 600 for selecting one or more game applications.

The user interface 600 may include, for example, an item 610 for executing a first game application, an item 620 for executing a second game application, and an item 630 for selecting a third game application. The first game application may correspond to a first game server. In response to a user input selecting item 610 for executing the first game application, the electronic device 100 may execute the first game application. The second game application may correspond to a second game server. In response to a user input selecting item 620 for executing the second game application, the electronic device 100 may execute the second game application. The third game application may correspond to a third game server. In response to a user input selecting item 630 for executing the third game application, the electronic device 100 may execute the third game application.

For example, when receiving a user input from the user to select an item 610 that can execute a first game application, the electronic device 100 can execute the first game application. In addition, the electronic device 100 can provide a user interface 700 according to the execution of the first game application. The user interface 700 can include items that can select one or more game contents provided by the first game server. In FIG. 7, for example, the user interface 700 can include items that can select game contents from game #1 to game #7 that can be provided by the first game server. In addition, the user interface 700 can also include a search window 720 that can search for game contents.

FIG. 7 is a reference diagram for explaining an operation of selecting a game content item in a game application user interface according to one embodiment.

Referring to FIG. 7, when a user input for selecting a game #1 item 710 is received in a game application user interface 700, the electronic device 100 may request the first game server to execute game content corresponding to game #1 (hereinafter, game #1 content) and receive and display a game screen 800 according to the execution of the game #1 content from the first game server.

Returning to FIG. 5, the electronic device 100 may receive input data, for example, user input, while executing a game application. Executing the game application may include both a state in which the game application user interface 700 is displayed, as illustrated in FIG. 6, and a state in which a specific game content is selected and a game screen 800 is displayed, as illustrated in FIG. 7. Accordingly, the electronic device 100 may receive user input while the game application user interface 700 is displayed, or while the game screen 800 is displayed.

In operation 520, the electronic device 100 can identify an application corresponding to an activated window. The application corresponding to the activated window refers to an application in the topmost window layer among the applications running on the electronic device 100. The application in the topmost window layer can indicate the application corresponding to the activated window. The electronic device 100 can identify the application in the topmost window layer by referencing information about the window stack.

The electronic device 100 can manage windows using at least one window stack. A window stack may refer to a logical arrangement of windows that are activated and/or inactive on a multi-screen device. For example, an active window may refer to a window that is currently displayed on the display. An inactive window may refer to a window that was once activated and displayed, but is currently not displayed because it is behind the active window. An inactive window may refer to a suspended application. An application corresponding to an activated window will be described with reference to FIG. 16.

Figure 16 illustrates an example of a window stack according to one embodiment.

Referring to FIG. 16, the electronic device 100 can execute a first application 1610, a second application 160, and a third application 1630. The first application 1610, the second application 1620, and the third application 1630 are all in a running state. What is currently displayed on the display screen may represent a window corresponding to the third application 1630. In addition, the first application 1610 and the second application 1620 may correspond to windows that are behind the third application 1630 and thus are not displayed on the display screen. User input received in this window stack state may correspond to data used in the third application. In this way, the third application 1630 in the topmost window layer in the window stack may represent an application corresponding to an activated window. On the other hand, the first application 1610 and the second application 1620 can be referred to as applications corresponding to the inactive window.

In operation 530, the electronic device 100 can identify whether the state of the player for playing cloud game content is a play state by identifying that the application corresponding to the activated window is a game application.

The electronic device 100 can identify that the application corresponding to the activated window is a game application. Based on this identification, the electronic device 100 can further identify whether the player's state is a play state. That is, as described above with reference to FIGS. 6 and 7, even while the game application is running, it may include various states. User input entered while the game application user interface 700 is displayed while the game application is running should be used to control the user interface 700. Therefore, such user input should be provided to the game application. On the other hand, user input received while selecting game content on the user interface 700 and actually playing the game should be provided to the game server, so that the game server can control the game content based on the user input. Therefore, even while the game application is running, the destination to which the user input is provided varies depending on the specific state, i.e., whether it is in a play state or not. Therefore, the electronic device 100 can identify whether the playback state of the game content is in a play state in order to confirm this state. The player's play state is specifically described with reference to FIGS. 8 and 9.

When an electronic device 100 sends a request to a game server to execute a specific game content, the game server may transmit metadata corresponding to the requested game content to the electronic device 100. This is to enable the electronic device 100 to prepare in advance the creation of a player for playing the game content in order to play the game content. The player may be referred to as a media player. The metadata corresponding to the media content transmitted to the electronic device 100 may include information such as resolution information, frame rate information, and codec type. The electronic device 100 may receive the metadata corresponding to the game content from the game server and create a player using the information included in the received metadata. Next, when the electronic device 100 receives a stream corresponding to a game screen corresponding to the requested game content from the game server, the electronic device 100 may play the received stream using the created player.

FIG. 8 illustrates an example of a media player pipeline, which is an example of a player, according to one embodiment.

Referring to FIG. 8, a media player pipeline 800 according to one embodiment may include a file source 810, a demultiplexer 820, a video decoder 830, an audio decoder 840, a video sink 850, and an audio sink 860.

Media Source 810 brings data into the pipeline. The media source can obtain data from a video file or a network stream.

Demultiplexer 820 demultiplexes audio packets and video packets interleaved in a media content stream.

Video decoder 830 processes video packet data from the stream.

Audio decoder 840 processes audio packet data from the stream.

The Video Sync 850 displays video on the display based on the decoded video data.

Audio Sync 860 plays audio based on the decoded audio data.

The game application controls the flow of data through the media player pipeline, and controls the media player pipeline through a media session that handles tasks such as quality control, audio/video synchronization, and responding to format changes, and also receives events from the media player pipeline. Therefore, the game application can identify whether the player is in the play state or the playback state by monitoring the media player pipeline. That is, the game application can determine that the player is in the play state or the playback state when an operation is performed according to the media player pipeline 800, for example, when at least one of the video decoder 830 and the audio decoder 840 is in operation. As illustrated in FIG. 7, if the game application is running and the game application user interface is displayed and a stream corresponding to game content has not yet been received from the game server, the player can be identified as not being in the play state.

Returning to FIG. 5 again, at operation 540, the electronic device 100 may transmit user input to a server transmitting cloud gaming content or provide it to a gaming application, depending on whether the player's state is in a play state.

According to one embodiment, if the electronic device 100 determines that the player's state is a play state, the electronic device 100 may transmit input data to a server transmitting cloud game content, thereby causing the server to proceed with the cloud game content based on the input data. For example, as illustrated in FIG. 7, if the electronic device 100 receives a game screen 750 from a game server and operates a media player, this may mean that the game content is already being played. Accordingly, since the input data received in this state must be used by the server for calculating the game content, the electronic device 100 may transmit the input data to the server.

According to one embodiment, if the electronic device 100 determines that the player's state is not a play state, the electronic device 100 may provide input data to the game application, thereby allowing the game application to control the user interface based on the input data. For example, as illustrated in FIG. 7, if the game application is running but the state of the media player is not yet a play state, this may mean that the game application user interface 700 is being output. Since the input data received in this state must be consumed by the game application for UI control, the input data may be provided to the game application.

FIG. 9 illustrates an example of a flowchart of a method of operating an electronic device according to one embodiment.

Referring to FIG. 9, in operation 901, the electronic device 100 may receive a request to execute a game application from the input device 200.

The input device 200 may be communicatively connected to the electronic device 100. When a user presses a button provided on the input device 200, the input device 200 may transmit an input signal corresponding to the pressed button to the electronic device 100.

According to one embodiment, the electronic device 100 may provide a user interface for requesting execution of a game application. The user interface for requesting execution of a game application may include items that enable execution of one or more game applications. The input device 200 may transmit a request for execution of the game application based on this user interface. For example, the electronic device 100 may display a user interface 600 as illustrated in FIG. 6.

When a user selects an item 610 capable of executing, for example, a first game application on a user interface 600 using an input device 200, the input device 200 may transmit an input signal indicating that the item 610 has been selected to the electronic device 100. This input signal may correspond to a request for executing the first game application.

In operation 902, the electronic device 100 may access a game server using a URL related to the corresponding game server in response to a game application execution request received from the input device 200 and request a server home UI screen provided by the game server. For example, when an input for selecting item 610 for execution of a first game application is received from the input device 200, the electronic device 100 may access a first game server using a URL related to the first game server and request a game application UI provided by the first game server.

In operation 903, the server 300 may transmit the game application UI to the electronic device 100 at the request of the electronic device 100. A portion of the game application UI screen may be pre-saved by the electronic device 100, and a portion of the game application UI screen may be received from the first game server. For example, the electronic device 100 may display the entire layout information of the pre-saved game application UI screen, or may receive and display only the updated contents among the contents displayed on the game application UI screen from the first game server.

In operation 904, the electronic device 100 may display a game application UI received from the server 300. The game application UI may include items for selecting one or more game contents, and may be a user interface 700 as illustrated in FIG. 7, for example.

In operation 905, the input device 200 may transmit an input for selecting a game content execution icon displayed on a game application UI screen to the electronic device 100. For example, the game application UI screen may include one or more items for executing game content, and the user may select a desired item for executing game content from among the displayed one or more items for executing game content using the input device 200. For example, the user may select the game #1 item 710 on the user interface 700 as illustrated in FIG. 7. Then, the input device 200 may transmit an input for selecting game #1 to the electronic device 100.

In operation 906, the electronic device 100 may receive an input for selecting a game content execution item from the input device 200, and may transmit a request for executing game content corresponding to the selected game content execution item to the server 300.

In operation 907, the server 300 may transmit metadata corresponding to the requested game content to the electronic device 100 in response to a game content execution request received from the electronic device 100. This is to enable the electronic device 100 to prepare in advance the creation of a media player for playing the game content in order to play the game content. The metadata corresponding to the media content transmitted to the electronic device 100 may include media-type attributes that specify encoding parameters such as an encoding algorithm, a video frame size, a video frame rate, an audio bit rate, and an audio sample rate, descriptive information of the media content such as a title, an artist, a genre, and encoding parameters, and DRM characteristic information such as usage restrictions.

In operation 908, the electronic device 100 may receive metadata corresponding to game content from the server 300 and generate a media player using information included in the received metadata. That is, the electronic device 100 may generate a media player suitable for playing a stream of content to be provided by the server 300 by referring to the received metadata.

In operation 909, the server 300 executes the game content requested to be executed by the electronic device 100, and in operation 910, the server 300 can stream audio and video streams corresponding to the game screen resulting from the execution of the game content to the electronic device 100.

In operation 911, the electronic device 100 may receive an audio/video stream corresponding to a game screen resulting from the execution of media content from the server 300 and play the received audio/video stream using a media player. Based on the playback of the audio/video stream, the electronic device 100 may display the game screen.

In operation 912, the input device 200 can transmit input for game play to the electronic device 100. The user can use the input device 200 to input for game play while viewing the game screen displayed on the display of the electronic device 100. Accordingly, the input device 200 can transmit the user's input for game play to the electronic device 100.

In operation 913, the electronic device 100 may transmit input for game play received from the input device 200 to the server 300. The input for game play may also include time information, such as information about which button of the input device 200 was pressed, information about the time a specific button of the input device 200 was selected, or information about how long a specific button was pressed.

In operation 914, the server 300 can control the play of the game content based on the input received from the electronic device 100. That is, the server 300 can control the game content by performing operations for the play of the game content based on the user's input for the game play.

In operation 915, the server 300 can transmit a result screen resulting from the control of the game play to the electronic device 100. Accordingly, the electronic device 100 can display a game screen controlled according to the user's game play input.

FIG. 10 illustrates an example of a flowchart of a method of operating an electronic device during execution of a game application according to one embodiment.

At various points during the execution of a game application as illustrated in FIG. 9, the electronic device 100 may receive user input from the input device 200, and it is necessary to efficiently transmit the received user input to a destination that will consume the input, i.e., the game application or server.

Referring to FIG. 10, at operation 1010, electronic device 100 may receive input data from input device 200.

According to one embodiment, the electronic device 100 and the input device 200 may be connected using short-range communication, such as Bluetooth communication or Wi-Fi communication. The electronic device 100 may receive input data transmitted from the input device 200 through a communication interface, and the input data received through the communication interface may be transmitted to the game driver 132.

At operation 1020, the game driver 132, upon receiving input data, can identify whether the application in the topmost window layer is a game application. If the identification result shows that the application in the topmost window layer is a game application, the process proceeds to operation 1030. If the application in the topmost window layer is not a game application, the process proceeds to operation 1040.

At operation 1030, the game driver 132 can identify whether the state of the media player is a play state.

If the status of the media player is identified as not being in a play state as a result of the identification of operation 1030, the process can proceed to operation 1040.

In operation 1040, the game driver 132 may provide input data to the game application 132. That is, when the game application is running and the state of the media player is not in the play state, it may mean that the game application has started running but has not yet received an actual stream from the server 300 and has not started playing, or that the playing has already been completed and the game application user interface is displayed. Accordingly, since the input data received from the input device 200 in this state is used to control the UI of the game application, such input data may be provided to the game application 131 so that it can be used by the game application 131. The UI of the game application may include, for example, the user interface 700 illustrated in FIG. 7.

Then, in operation 1050, the game application 131 can control the user interface based on input data provided from the game driver 132. For example, if the user inputs an item 720 for searching game content in the user interface 700, the game application 131 can perform a game content search operation, such as displaying a virtual keyboard and performing a search operation corresponding to a keyword later entered by the user, based on this input.

If it is determined in operation 1030 that the state of the media player is in a play state, the process may proceed to operation 1060. In other words, when the game application is running and the state of the media player is in a play state, this may mean that the electronic device 100 receives an audio/video stream corresponding to the game screen from the server 300 and plays it. Therefore, since the input data received from the input device 200 in this state is for controlling the game play, it is preferable to provide the input data to the server 300 so that the server 300 can utilize the input data.

In operation 1060, the game driver 132 can transmit input data to the server 300. Conventionally, the game application has all scenarios for how to process input data received while the game application is running. Therefore, even if the input data is data to be used directly in the server 300, the game driver 132 can first transmit the input data to the game application 131, and when the game application 131 determines that the input data should be transmitted to the server 300, the game driver 131 can then receive the input data from the game application 131 and transmit the received input data to the server 300. However, according to the embodiment of the present disclosure, the game driver 132 determines that the input data is to be consumed in the server 300 based on its own judgment, and transmits it directly to the server 300 without having to transmit it to the game application 131, thereby significantly reducing latency.

In operation 1070, server 300 may receive input data from game driver 132 and execute game content based on the input data.

FIG. 11 illustrates an example of a flowchart of a method of operating an electronic device during execution of a game application according to an embodiment.

The operation illustrated in FIG. 11 is mostly the same as the operation illustrated in FIG. 10, except that it further includes operations 1110 and 1120. That is, there may be a situation in which the electronic device 100 receives an audio/video stream corresponding to the game content from the server 300 and outputs a user interface by the game application's own operation even while playing the game content. In such a case, since the user interface is being displayed on the display screen even when the state of the media player is in the playing state, it is preferable that the user input at this time is not transmitted to the server 300, but rather is transmitted to the game application 131 so that it can be UI controlled by the game application 131.

Referring to Fig. 11, operations 1010 to 1040 are the same as in Fig. 10, so their description will be omitted.

As a result of the identification of operation 1030, if it is determined that the state of the media player is in a play state, the process may proceed to operation 1110. That is, when the game application is running and the state of the media player is in a play state, it may mean that the electronic device 100 receives an audio/video stream corresponding to the game screen from the server 300 and plays it. Even when the electronic device 100 is playing the game screen, there may be a state in which a UI by the game application itself is displayed. In this situation, it is preferable that the received input data be consumed by the game application 131 instead of the server 300, and therefore, a judgment operation such as in operation 1110 may be provided.

In operation 1110, the game driver 132 can identify whether the game application status information is displaying a UI. For example, the game application 131 may display a UI for some reason of its own even when the media player status is in a play state. In such a case, the game application 131 can display the UI and notify the game driver 132 of the UI display status. Therefore, the game driver 132 can identify whether the game application status information is displaying a UI.

FIG. 12 is a reference diagram for explaining an example of displaying a UI during game play according to one embodiment.

For example, when the electronic device 100 displays a game screen 1200 by receiving and playing an audio/video stream corresponding to game content from the server 300 and no input is received from the user for a certain period of time while the game is being played, the electronic device 100 may display a user interface 1210 as illustrated in FIG. 12. That is, when no user input is received for a certain period of time while the game is being played, the electronic device 100 may output the user interface 1210 to determine whether the user intends to continue playing the game. The user interface 1210 may include a QUIT GAME item 1211 and a NEVER MIND item 1212 along with a message asking the user whether he or she wants to quit the game. If the user wants to continue playing the game, he or she can continue playing the game by selecting the NEVER MIND item 1212. If the user wants to stop playing the game, he or she can exit the game by selecting the QUIT GAME item 1211. Therefore, while this user interface 1210 is displayed, any user input received needs to be used by the game application 131 to control the UI.

Returning to Figure 11 again, if it is identified in operation 1110 that the game application status information is being displayed on the UI, operation 1120 can be performed.

At operation 1120, the game driver 132 may transmit input data to the game application 131.

Then, in operation 1050, the game application 131 can perform operations such as UI control based on the received input data.

If the identification result of operation 1110 determines that the status information of the game application is not being displayed on the UI, the process can proceed to operation 1060. The operations of operations 1060 and 1070 are the same as those described in Fig. 10, and therefore further description is omitted.

FIG. 13 illustrates an example of a flowchart of a method of operating an electronic device during execution of a game application according to one embodiment.

The operation of FIG. 13 can be performed when it is determined in the judgment result of operation 1020 of FIG. 10 that the application in the top window layer is not a game application.

In operation 1310, the game driver 132 can identify that the application in the topmost window layer is a system user interface application. The system user interface application may include an application generated by the system of the electronic device 100. For example, during game play, the user may cause the system user interface to be displayed by pressing a specific button for control by the system of the electronic device 100 rather than for settings or controls within the game application. This system user interface may include a quick panel.

Figure 14 illustrates an example of a quick panel according to one embodiment.

A user can display a quick panel by performing a long press action of pressing a specific button provided on an input device 100 while playing a game to call up the quick panel. When an electronic device 100 receives an input signal corresponding to a long press of a specific button, which is input data from the input device 100, this should be operated by the system UI module of the electronic device 100, not by the operation of a game application. Therefore, this input signal can be transmitted to the system UI module 1300, and the system UI module 1300 can display a quick panel 1400 as illustrated in FIG. 14. Since the system UI module 1300 is triggered and operated in this way, a system UI application can be located on the topmost window layer.

Referring to FIG. 14, the quick panel 1400 may include, for example, an item 1410 for managing applications, an item 1420 for playing music, an item 1430 for broadcasting a game screen, an item 1440 for changing settings, an item 1450 for displaying a list of recently played games, and an item 1460 for going to the home screen.

Returning to FIG. 13, at operation 1320, the game driver 132 can identify whether the system user interface has focus. If the system user interface has focus, this may indicate that the corresponding system user application is performing some action or control based on input data. For example, a system user interface with focus may include a quick panel. If a system user interface does not have focus, this may indicate that the system user application is not performing any action but is simply displaying the system user interface. For example, a system user interface without focus may be an interface that simply has a notification function, such as a message indicating that the volume of the user electronic device 100 is increasing when an input is made to increase the volume. For example, a volume message 1500 as illustrated in FIG. 15 may indicate a user interface that does not have focus and only displays information.

If it is determined in operation 1320 that the system user interface does not have focus, the process may proceed to operation 1330. In this state, since the input data received from the input device 200 is for controlling game play, it is desirable to provide such input data to the server 300 so that the server 300 can utilize it.

In operation 13300, the game driver 132 can transmit input data to the server 300. Conventionally, the game application has all scenarios for how to process input data received while the game application is running. Therefore, even if the input data is data to be used directly in the server 300, the game driver 132 can first transmit the input data to the game application 131, and when the game application 131 determines that the input data should be transmitted to the server 300, the game driver 131 can then receive the input data from the game application 131 and transmit the received input data to the server 300. However, according to the embodiment of the present disclosure, the game driver 132 determines that the input data is to be consumed in the server 300 based on its own judgment, and transmits it directly to the server 300 without having to transmit it to the game application 131, thereby significantly reducing latency.

At operation 1340, the server 300 may receive input data from the game driver 132 and execute game content based on the input data.

If the system user interface is identified as having focus as a result of the judgment in operation 1320, the process can proceed to operation 1350.

In operation 1350, the game driver 132 can transmit input data to the system UI module. That is, by transmitting the input data to the system UI module that outputs the system user interface, the system UI module can control the UI based on the input data.

At operation 1360, the system UI module may control the UI based on input data transmitted by the game driver 132. For example, if the input data represents an input for selecting an item 1460 that can go to the home screen from the quick panel 1400, the system UI module may perform an operation to display the home screen on the display.

Some embodiments may also be implemented in the form of a recording medium containing computer-executable instructions, such as program modules, executed by a computer. Computer-readable media may be any available media that can be accessed by a computer, and includes both volatile and nonvolatile media, removable and non-removable media. Computer-readable media may also include computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, program modules, or other data.

The disclosed embodiments can be implemented as a S/W program including commands stored in a computer-readable storage media.

A computer may include an electronic device according to the disclosed embodiments, as a device capable of calling a command stored from a storage medium and performing an operation according to the called command according to the disclosed embodiments.

Computer-readable storage media may be provided in the form of non-transitory storage media. Here, "non-transitory" simply means that the storage medium does not contain signals and is tangible, but does not distinguish between whether data is stored semi-permanently or temporarily on the storage medium.

Additionally, the control method according to the disclosed embodiments may be provided as a computer program product. The computer program product may be traded as a product between a seller and a buyer.

A computer program product may include a software program and a computer-readable storage medium on which the software program is stored. For example, the computer program product may include a product in the form of a software program (e.g., a downloadable app) distributed electronically by a device manufacturer or through an electronic marketplace (e.g., Google Play Store, App Store). For electronic distribution, at least a portion of the software program may be stored on a storage medium or temporarily created. In this case, the storage medium may be a storage medium of a manufacturer's server, an electronic marketplace server, or a relay server that temporarily stores the software program.

In a system comprising a server and a device, the computer program product may include a storage medium of the server or a storage medium of the device. Alternatively, if a third device (e.g., a smartphone) exists that is communicatively connected to the server or device, the computer program product may include a storage medium of the third device. Alternatively, the computer program product may include a software program itself that is transmitted from the server to the device or the third device, or from the third device to the device.

In this case, one of the server, the device, and the third device may execute the computer program product to perform the method according to the disclosed embodiments. Alternatively, two or more of the server, the device, and the third device may execute the computer program product to perform the method according to the disclosed embodiments in a distributed manner.

For example, a server (e.g., a cloud server or an artificial intelligence server, etc.) may execute a computer program product stored on the server to control a device in communication with the server to perform a method according to the disclosed embodiments.

As another example, a third device may execute a computer program product to control a device in communication with the third device to perform a method according to the disclosed embodiments. When the third device executes the computer program product, the third device may download the computer program product from a server and execute the downloaded computer program product. Alternatively, the third device may execute a computer program product provided in a preloaded state to perform the method according to the disclosed embodiments.

Additionally, in this specification, a “part” may be a hardware component such as a processor or circuit, and/or a software component executed by a hardware component such as a processor.

The above description of the present disclosure is provided for illustrative purposes only, and those skilled in the art will readily appreciate that modifications to other specific forms can be made without altering the technical spirit or essential features of the present disclosure. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, components described as being single may be implemented in a distributed manner, and similarly, components described as being distributed may be implemented in a combined manner.

The scope of the present disclosure is indicated by the claims described below rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present disclosure.

Claims (20)

In electronic device 100,
Communication interface 110,
Memory 130 for storing a game application corresponding to the execution of cloud game content and a game driving unit for executing the game application, and
A processor 140 comprising one or more instructions stored in the memory,
The above processor 140 executes one or more of the instructions,
By the above game driver, input data is received while executing the game application,
An electronic device that controls whether the application corresponding to the activated window corresponds to the game application, and transmits the received input data to a server transmitting the cloud game content or provides the received input data to the game application, depending on whether the player's status for executing the cloud game content is a play status. In the first paragraph,
The above processor 140 executes one or more instructions,
If the state of the player is the play state, the input data is transmitted to the server transmitting the cloud game content,
An electronic device that controls the input data to be provided to the game application so that the game application operates based on the input data when the state of the player is not the play state. In claim 1 or 2,
The above processor 140 executes one or more instructions,
While the state of the above player is the above play state, receiving state information corresponding to a user interface output related to the above game application from the above game application,
An electronic device that controls, based on the received status information, to provide the input data to the game application so that the game application can operate based on the input data. In any one of the first to third paragraphs,
The above processor 140 executes one or more instructions,
An electronic device that controls the processing of input data differently depending on whether the focus is located within the system user interface, if the application corresponding to the activated window is an application corresponding to the system user interface. In any one of the first to fourth paragraphs,
The above processor 140 executes one or more instructions,
When the focus is located within the above system user interface, the input data is transmitted to the system UI module,
An electronic device that controls the transmission of the input data to the server when the focus is not located within the system user interface. In any one of the first to fifth paragraphs,
An electronic device, wherein the system user interface on which the focus is located includes a user interface corresponding to a quick panel. In any one of claims 1 to 6,
The above processor 140 executes one or more instructions,
An electronic device that controls the identification of an application at the top layer in a window stack as the application corresponding to the activated window. In any one of the first to seventh paragraphs,
The above processor 140 executes one or more instructions,
Create the player for executing the above cloud game content,
An electronic device that identifies whether the state of the player is the play state based on the operational state of at least one of the audio decoder or video decoder of the generated player. In any one of claims 1 to 8,
The above processor 140 executes one or more instructions,
An electronic device that creates the player based on metadata of the cloud game content received from the server. In any one of claims 1 to 9,
By transmitting the above input data to the server, a result screen for executing the cloud game content based on the above input data is received from the server,
An electronic device that controls the game application to operate based on the input data by providing the input data to the game application. A method for operating an electronic device 100 including a game application corresponding to the execution of cloud game content and a game driving unit for executing the game application,
An operation of receiving input data while executing the game application by the above game driving unit, and
A method comprising an operation of transmitting the received input data to a server transmitting the cloud game content or providing the received input data to the game application, depending on whether the state of the player for executing the cloud game content is a play state, when the application corresponding to the activated window corresponds to the game application. In Article 11,
If the state of the player is the play state, an operation of transmitting the input data to a server transmitting the cloud game content, and
A method comprising an action of providing the input data to the game application so that the game application operates based on the input data if the state of the player is not the play state. In claim 11 or 12,
An operation of receiving status information corresponding to a user interface output related to the game application from the game application while the state of the player is the play state, and
A method comprising an action of providing the input data to the game application so that the game application can operate based on the input data, based on the received status information. In any one of Articles 11 to 13,
A method comprising an operation for controlling the processing of the input data differently depending on whether the focus is located within the system user interface, if the application corresponding to the activated window is an application corresponding to the system user interface. In any one of Articles 11 to 14,
When the focus is located within the above system user interface, an operation of transmitting the input data to the system UI module, and
A method comprising an action of transmitting the input data to the server when the focus is not located within the system user interface. In any one of Articles 11 to 15,
A method wherein the system user interface on which the focus is located includes a user interface corresponding to a quick panel. In any one of Articles 11 to 16,
A method comprising an operation of identifying an application corresponding to the activated window based on a window stack. In any one of Articles 11 to 17,
An action to create the player for executing the cloud game content, and
A method comprising an operation of identifying whether the state of the player is the play state based on the operational state of at least one of an audio decoder or a video decoder of the generated player. In any one of Articles 11 to 18,
An operation of receiving a result screen of executing the cloud game content based on the input data from the server by transmitting the input data to the server, and
A method comprising an action of controlling the game application to operate based on the input data by providing the input data to the game application. In a non-transitory computer-readable recording medium having one or more instructions stored thereon, the one or more instructions are executed by a processor of an electronic device 100 including a game application corresponding to the execution of cloud game content and a game driving unit for executing the game application, so that the electronic device 100,
By the above game driver, input data is received while executing the game application,
A non-transitory computer-readable recording medium that transmits the received input data to a server transmitting the cloud game content or provides the received input data to the game application, depending on whether the state of the player for executing the cloud game content is a play state when the application corresponding to the activated window corresponds to the game application.