KR102932370B1 - Method for performing a video calling, display device for performing the same method, and computer readable medium storing a program for performing the same method - Google Patents

Abstract

A method for performing a video call through a display device according to the disclosed embodiment includes: a step of recognizing at least one object included in a first image acquired for performing a video call; a step of receiving a user input for selecting at least one of the recognized at least one object; a step of acquiring a second image including the selected object among the at least one object included in the first image based on the user input; and a step of transmitting the second image to a counterpart device.

Description

Method for performing a video call, a display device for performing the method, and a computer-readable storage medium storing a program for performing the method {Method for performing a video call, a display device for performing the same method, and a computer-readable medium storing a program for performing the same method}

The disclosed embodiment relates to a method for performing a video call, a display device for performing the method, and a computer-readable storage medium storing a program for performing the method.

As wired and wireless communication networks and technologies advance, the use of video calling services between electronic devices is increasing. Specifically, video calling services between electronic devices are widely used to enable remote users to communicate with each other without face-to-face interaction.

Specifically, for video call services, one electronic device and another electronic device can be interconnected via a wired or wireless communication network. Here, the electronic device can be any electronic device that includes a display capable of providing a video call screen and can connect to a wired or wireless communication network to communicate with another electronic device located remotely. Examples of electronic devices include portable computers such as laptops, netbooks, or tablet PCs; portable terminals such as smartphones or PDAs; and televisions.

When a video call is made between multiple electronic devices, for example, a first electronic device and a second electronic device, the first electronic device acquires an image of a user and transmits the acquired image to the second electronic device. Accordingly, the second electronic device can make a call while viewing the image of the user of the first electronic device. Furthermore, the second electronic device acquires an image of the user and transmits the acquired image to the first electronic device. Accordingly, the first electronic device can make a call while viewing the image of the user of the second electronic device.

Additionally, the first electronic device or the second electronic device can be used by multiple users. Thus, when interconnecting multiple electronic devices to conduct video calls, there is a need to provide devices and methods that enhance user satisfaction and convenience.

Additionally, when providing services through electronic devices, it is necessary to provide electronic devices and methods of operating electronic devices in a way that protects personal information or privacy of users.

Therefore, there is a need to provide electronic devices and methods that can protect users' privacy in accordance with their intentions while increasing user satisfaction and convenience.

The disclosed embodiment aims to provide a method for performing a video call, a display device performing the method, and a computer-readable storage medium storing a program performing the method, which can protect the privacy of a user using a plurality of electronic devices when they are interconnected and used.

A display device according to the disclosed embodiment includes a display; a communication unit for performing communication with an external device; a user interface for receiving a user input; and a processor for executing at least one instruction. The processor recognizes at least one object included in a first image acquired for performing a video call, and, based on a user input for selecting at least one of the recognized at least one object, acquires a second image including the selected object among the at least one object included in the first image, and controls the communication unit so that the second image is transmitted to the other party's device.

Additionally, the processor may control a user interface screen for selecting at least one object included in the first image to be output through the display before transmitting the image to be included in the video call screen to the other party's device.

Additionally, the processor may initiate transmission of the second image after the user input is received.

Additionally, the processor may stop transmitting images to be included in the video call screen until the user input is received.

Additionally, the processor determines whether a new object is recognized in the updated first image when the camera acquires the updated first image, and if the new object is recognized, controls a user interface screen for selecting the new object to be output through the display, and based on a user input for selecting the new object, updates the second image to include the new object.

Additionally, if there is an object among at least one object included in the second image that is not selected by the user input, the processor can generate the second image by deleting the unselected object from the first image.

In addition, if there is an object among at least one object included in the first image that is not selected by the user input, the processor may generate the second image by performing image processing so that the first image includes a virtual object corresponding to the unselected object.

Additionally, the display device according to the disclosed embodiment may further include a camera. The processor may activate the camera in response to a video call request and acquire the first image through the activated camera.

In addition, the processor can recognize at least one person, a background of the space, and an object included in the space included in the first image, and control a user interface screen for selecting or deselecting each of the recognized at least one person, the background of the space, and the object included in the space to be output through the display.

In addition, the processor can perform object recognition by obtaining at least one object output from the neural network when the neural network that has received the first image performs an operation for object recognition to extract and output at least one object included in the first image.

The disclosed embodiment is a method for performing a video call via a display device. The method for performing a video call includes: a step of recognizing at least one object included in a first image acquired for performing a video call; a step of receiving a user input for selecting at least one of the recognized at least one object; a step of acquiring a second image including the selected object from among the at least one object included in the first image based on the user input; and a step of transmitting the second image to a counterpart device.

The disclosed method for performing a video call, a display device performing the method, and a computer-readable storage medium storing a program for performing the method can protect the privacy of multiple users within the front space of the display device according to their respective intentions. Accordingly, user satisfaction with the display device can be enhanced.

Figure 1 is a drawing for explaining a video call service performed between multiple electronic devices.
Figure 2 is a drawing for explaining a video call service performed on a display device.
FIG. 3 is a block diagram illustrating a display device according to the disclosed embodiment.
FIG. 4 is another box diagram illustrating a display device according to the disclosed embodiment.
FIG. 5 is a drawing for explaining a plurality of electronic devices used to perform a video call service according to the disclosed embodiment.
FIG. 6 is a flowchart illustrating a method for performing a video call according to the disclosed embodiment.
FIG. 7 is a drawing for explaining an object recognition operation according to the disclosed embodiment.
FIG. 8 is another flowchart illustrating a method for performing a video call according to the disclosed embodiment.
FIG. 9 is a drawing showing an example of a user interface screen output in the disclosed embodiment.
FIG. 10 is a drawing showing another example of a user interface screen output in the disclosed embodiment.
FIG. 11 is a drawing showing an example of an image generated by a display device according to the disclosed embodiment.
FIG. 12 is a drawing showing another example of an image generated by a display device according to the disclosed embodiment.
FIG. 13 is a drawing showing another example of an image generated by a display device according to the disclosed embodiment.
FIG. 14 is another flowchart illustrating a method for performing a video call according to the disclosed embodiment.
Figure 15 is a drawing for explaining the operations shown in Figure 14.
FIG. 16 is a drawing showing another example of an image generated by a display device according to the disclosed embodiment.

Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings so that those skilled in the art can easily practice the present invention. However, the present invention may be implemented in various different forms and is not limited to the embodiments described herein. In addition, in the drawings, parts irrelevant to the description are omitted to clearly explain the present invention, and similar parts are designated with similar reference numerals throughout the specification. In addition, throughout the drawings, the same reference numerals are assigned to the same components.

Throughout the specification, when a part is said to be "connected" to another part, this includes not only the cases where the parts are "directly connected" but also the cases where the parts are "electrically connected" with other elements intervening. Furthermore, when a part is said to "include" a component, this does not exclude other components, but rather includes other components, unless otherwise stated.

The appearances of phrases such as “in some embodiments” or “in one embodiment” in various places throughout this specification are not necessarily all referring to the same embodiment.

Some embodiments may be represented by functional box configurations and various processing steps. Some or all of these functional boxes may be implemented by various hardware and/or software configurations that perform specific functions. For example, the functional boxes of the present disclosure may be implemented by one or more processors or microprocessors, or by circuit configurations for performing the intended functions. Furthermore, for example, the functional boxes of the present disclosure may be implemented by various programming or scripting languages. The functional boxes may be implemented by algorithms that execute on one or more processors. Furthermore, the present disclosure may employ conventional techniques for electronic configuration, signal processing, and/or data processing. Terms such as "module" and "configuration" may be used broadly and are not limited to mechanical and physical configurations.

Additionally, the connecting lines or connecting members between components depicted in the drawings are merely exemplary representations of functional connections and/or physical or circuit connections. In an actual device, connections between components may be represented by various functional connections, physical connections, or circuit connections that may be replaced or added.

Additionally, the description 'at least one of A, B, and C' means that it can be any one of 'A', 'B', 'C', 'A and B', 'A and C', 'B and C', and 'A, B, and C'.

The disclosed embodiments relate to a method for performing a video call, a display device for performing the method, and a computer-readable storage medium storing a program for performing the method. Specifically, the disclosed embodiments relate to a method for performing a video call, which can protect a user's privacy according to the user's intention when performing a video call between multiple electronic devices, a display device for performing the method, and a computer-readable storage medium storing a program for performing the method.

In the disclosed embodiment, the electronic device for performing a video call includes a display for displaying a video call screen, and can be any electronic device capable of connecting to a wired or wireless communication network to perform communication with another electronic device located remotely. Furthermore, the electronic device is a computing device capable of providing a video call service, including a display, and can exist in a wide variety of forms. For example, the electronic device may include, but is not limited to, a wearable device, a smart phone, a personal digital assistant (PDA), a media player, a tablet PC, a laptop computer, a media player, a TV, a digital TV, a smart TV, a digital signage, and a digital signage.

Hereinafter, the "mobile device" and "display device" may be electronic devices capable of performing video calls by displaying a video call screen on a display. That is, the "mobile device" and "display device" may be wearable devices, smartphones, PDAs (personal digital assistants), media players, tablet PCs, laptop computers, media players, TVs, digital TVs, smart TVs, digital signage, digital signs, etc., respectively.

However, for convenience of explanation, "mobile device" and "display device" are distinguished. For example, an electronic device with a relatively large display or capable of capturing video call screens with a relatively wide angle of view may be referred to as a "display device," while an electronic device with a relatively small display or capable of capturing video call screens with a relatively narrow angle of view may be referred to as a "mobile device."

For convenience of explanation, electronic devices with a portable form factor will be referred to as "mobile devices" below. Furthermore, electronic devices that provide a larger screen through a larger display than mobile devices or that can capture video call screens with a wider field of view than mobile devices may be referred to as "display devices."

Additionally, the mobile device may be portable and user-portable. Furthermore, the display device may be portable and installed in a fixed location. Alternatively, the display device may be portable and installed in a fixed location.

The following description illustrates and explains the mobile device as a smartphone and the display device as a TV, using the attached drawings as examples. Furthermore, in the attached drawings, identical components are depicted using identical drawing symbols. Furthermore, throughout the detailed description, identical components are described using identical terminology.

Hereinafter, with reference to the attached drawings, a method for performing a video call according to the disclosed embodiment, a display device performing the method, and a computer-readable storage medium storing a program performing the method will be described in detail.

Figure 1 is a drawing for explaining a video call service performed between multiple electronic devices.

In the disclosed embodiment, a video call means that multiple users located remotely from each other make a call while each user sees the other user's face on the screen through multiple electronic devices. In addition, the 'video call' mentioned in the disclosed embodiment can be applied to all fields in which users located remotely communicate while providing each other's images, such as video conferencing, non-face-to-face communication, non-face-to-face education, etc., and can refer to all cases in which multiple users located remotely from each other communicate while seeing each other's images. Here, the image output from the electronic device through a video call need not necessarily be the face of the user making the video call, but may also be an image representing the environment in which the user making the video call is located, an image provided by the user, etc.

Typically, video calls can be made between two different electronic devices. Alternatively, three or more users located remotely can make video calls using three or more electronic devices.

Below, we will illustrate and illustrate an example of a video call between one user (hereinafter, "User") and another user (hereinafter, "Party") via two remotely located electronic devices. Here, "User" does not refer to a single person; it may refer to multiple users, and "Party" may also refer to multiple people.

Referring to FIG. 1, a video call can be performed between a display device (103) and a counterpart device (102), which are electronic devices located at a remote location (S191). Furthermore, the video call can be performed by transmitting and receiving data required for the video call using a communication network, such as 3G, 4G, or 5G. Alternatively, the video call can be performed through a telecommunications company's call service. Furthermore, the video call can be performed through a predetermined application (e.g., a video call application, a non-face-to-face communication application, a video education application, a video conferencing application, etc.) provided through a service provider's server.

In the following description, including FIG. 1, the user's electronic device is referred to as a display device (103), and the electronic device of the other party with whom the user wishes to make a video call is referred to as the other party's device (102). In FIG. 1, the other party's device (102) is illustrated as a smartphone as an example. In addition, the display device (103) and the mobile device (101) may be electronic devices owned by user A2 (117), and the other party's device (102) may be an electronic device owned by the other party, user B (127).

Referring to FIG. 1, the display device (103) can perform a video call with the other party's electronic device (102) located at a remote location (S191). The display device (103) can perform the video call by executing its own calling function or an application for non-face-to-face communication.

Alternatively, a video call executed on a mobile device (101) may be performed on a display device (103) through device switching. Specifically, a video call may be performed between a mobile device (101) and a counterpart device (102) (S181). Subsequently, the device executing the video call may be switched or changed from the mobile device (101) to the display device (103) (S182). For example, a user A2 (117) may own a plurality of electronic devices capable of mutual communication, namely, a mobile device (101) and a display device (103). Then, in order to conveniently use the video call on a larger screen, the user A2 (117) may want to change or switch the electronic device executing the video call from the mobile device (101) to the display device (103). When the execution device of the video call is switched from a mobile device (101) to a display device (103), user A2 (117) can be provided with a video call screen (151) through the display (131) of the display device (103) that provides a larger screen. In FIG. 1, the case where the video call screen (151) includes a message notifying preparation for a video call is illustrated as an example, but the video call screen (151) can be a screen that includes an image (112) of the other party, user B (127), as the main screen and an image (for example, 113) of user A2 (117) as the sub screen, as in the video call screen (111) output from the mobile device (101).

As described in Fig. 1, when a video call is performed through a display device (103), an image of the user is captured through a camera (105) included in the display device (103) (or connected to the display device (103)).

Specifically, when a video call starts, the camera (105) of the display device (103) photographs the front of the display device (103) to obtain an image of the user included in the video call screen.

The display device (103) has a display (131) that can provide a larger screen than the mobile device (101) placed on the front, and generally, the camera (105) captures the entire space in front of the display device (103). Therefore, if there are multiple people in the space in front of the display device (103), the camera (105) captures all of the multiple people. For example, if user A1 (116), user A2 (117), and user A3 (118) are located in the space in front of the display device (103), user A1 (116), user A2 (117), and user A3 (118) can all be imaged in the image captured by the camera (105). Here, 'imaging' may mean visually expressing or displaying through shooting or scanning. Specifically, the images captured by the camera (105) may all display user A1 (116), user A2 (117), and user A3 (118).

The image captured by the camera (105) is described in detail below with reference to FIG. 2.

Figure 2 is a drawing illustrating a video call service performed on a display device. In Figure 2, the same configuration as in Figure 1 is depicted using the same drawing symbols.

Referring to FIG. 2, a display device (103) capable of performing a video call according to the disclosed embodiment may be located within a home. The display device (103) may include a camera (105) on its own. Alternatively, if the display device (103) does not include a camera (105), the camera (105) may be formed as an external camera (105) that can be electrically connected to the display device (103). Hereinafter, a case in which the camera (105) is provided in a form that is included in the display device (103) will be described as an example.

The display device (103) has a larger display (i.e., a large-screen display) than the mobile device (101), and the overall size of the device is also larger. In addition, the camera (105) included or connected to the display device (103) generally has a larger angle of view (210) than the camera (not shown) included in the mobile device (101). Therefore, the camera (105) can capture a wider space than the camera (not shown) included in the mobile device (101).

For example, a mobile device (101) can capture only a portion of the face and upper body of user A2 (117) holding the device. In contrast, a camera (105) can capture the entire space in front of the display device (103). Accordingly, the image acquired by the display device (103) can represent multiple users (116, 117, 118) located in the space in front of the display and the entire space.

That is, when a camera (105) captures an image to perform a video call on a display device (103), multiple users (116, 117, 118) and spaces may all appear in the captured image.

As described above, when multiple people are present within the shooting space of the display device (103), there may be some who do not want to be included in the video call screen. For example, user A1 (116), who did not request a video call with the other party's device (102 in FIG. 1), may not want user B (127 in FIG. 1) of the other party's device (102 in FIG. 1) to see his or her appearance. In addition, user A2 (117), who requested a video call with the other party's device (102 in FIG. 1), may not want user B (127 in FIG. 1) to see the space in which he or she is present.

Accordingly, the disclosed embodiment provides a device and method that can protect the privacy of each of a plurality of users according to the intention of each of the users existing in a space where a display device (103) is placed.

FIG. 3 is a block diagram illustrating a display device according to the disclosed embodiment. The display device (300) illustrated in FIG. 3 may correspond to the display device (103) illustrated in FIGS. 1 and 2 . Therefore, in describing the operations of the display device (300), any descriptions that overlap with those in FIGS. 1 and 2 will be omitted.

Referring to FIG. 3, the display device (300) includes a processor (310), a display (320), a communication unit (330), and a user interface (350).

In the disclosed embodiment, the processor (310) executes at least one instruction to recognize at least one object included in a first image acquired for performing a video call. Then, based on a user input selecting at least one of the recognized objects, the processor acquires a second image including the selected object among the at least one object included in the first image. Subsequently, the processor controls the communication unit (330) to transmit the second image to the other party's device.

In the disclosed embodiment, the "first image" may be an image acquired by the display device (300) for a video call. Specifically, if the display device (300) includes a camera (e.g., 105 of FIG. 2) internally or externally, the first image may be an original image captured by the camera when the image of the user to be included in the video call screen is captured by the camera. Alternatively, the first image may be an image in which objects included in the image captured by the camera are not edited.

For example, if the display device (300) internally includes a camera (not shown), the processor (310) of the display device (300) may activate the camera (not shown) when a video call is requested. Accordingly, the camera (not shown) may capture a front space of the display device (300) to obtain a first image.

As another example, a display device (300) may be connected to an external camera (not shown) and receive a first image acquired from the external camera (not shown). Specifically, when a video call is requested, the processor (310) may transmit a control signal requesting activation to the external camera (not shown) and receive the first image acquired from the external camera (not shown) through the communication unit (330).

And, the ‘second image’ may be an image generated to include (or display) only objects selected by user input among the objects displayed in the first image. Specifically, when the display device (300) includes a camera (e.g., 105 of FIG. 2) internally or externally, the ‘second image’ may be an image generated by editing the first image acquired from the camera so that only objects selected by user input are displayed. However, when all objects included in the first image are objects selected by user input, the second image may be identical to the first image.

Specifically, the processor (310) controls the execution of at least one instruction so that an intended operation is performed. Here, the at least one instruction may be stored in an internal memory (not shown) included in the processor (310) or in a memory (not shown) included in the display device (300) separately from the processor (310).

Specifically, the processor (310) can control at least one component included in the display device (300) so that an intended operation is performed by performing at least one instruction. Therefore, even if the processor (310) performs predetermined operations as an example, it may mean that the processor (310) controls at least one component included in the display device (300) so that the predetermined operations are performed. In addition, even if the display device (300) is described as performing a predetermined operation, it may mean that it is performed under the control of the processor (310). In addition, although the processor (310) is described and illustrated as an example in which it is formed as one processor, it may be formed in a form in which a plurality of processors are included.

Specifically, the processor (310) may include a RAM (not shown) that stores signals or data input from the outside of the display device (300) or is used as a storage area corresponding to various tasks performed in the display device (300), a ROM (not shown) that stores a control program and/or a plurality of instructions for controlling the display device (300), and at least one processor (not shown). The processor (not shown) may include a graphic processor (Graphic Processing Unit, not shown) for graphic processing corresponding to video. The processor (not shown) may be implemented as a SoC (System On Chip) that integrates a core (not shown) and a GPU (not shown). In addition, the processor (310) may include multiple cores more than a single core. For example, the processor (310) may include a dual core, a triple core, a quad core, a hexa core, an octa core, a deca core, a dodeca core, a hexa double core, etc.

In the disclosed embodiment, the processor (310) may perform a video call with a counterpart device (e.g., 102 of FIG. 1) in response to a video call request. Here, the video call may be performed through the video call function of the display device (300) itself, or through an application for providing a video call service. For example, an application for providing a video call service may be referred to as a video call application.

In this case, the processor (310) may store a video call application. The video call application may be distributed (e.g., downloaded or uploaded) online to each of two user devices (e.g., a smartphone and/or a smart TV) via an application store (e.g., Play Store™). In the case of online distribution, the video call application may be distributed via a manufacturer's server, an application store's server, or a relay server.

In the disclosed embodiment, a video call application is installed on at least two of a mobile device (101), a counterpart device (102), and a display device (300), and a video call can be performed between the two installed devices using the video call application.

The display (320) outputs an image on the screen. Specifically, the display (320) can output an image corresponding to the video data through an internally included display panel (not shown) so that a user can visually recognize the video data.

In the disclosed embodiment, in order to perform a video call, the display (320) can output a video call screen.

The communication unit (330) communicates with another electronic device (not shown) via at least one wired or wireless communication network. In the disclosed embodiment, the communication unit (330) communicates with at least one of the mobile device (101) and the counterpart device (102). In addition, the communication unit (330) may communicate with a server (not shown in FIG. 3) (e.g., the server (500) shown in FIG. 5) to transmit and receive predetermined data with at least one of the mobile device (101) and the counterpart device (102).

The communication unit (330) can communicate with an external device (e.g., at least one of a mobile device (101), a counterpart device (102), and a server (not shown)) via a wired or wireless communication network (301). Specifically, the communication unit (330) can be formed in a form including at least one communication module, a communication circuit, etc., and can transmit and receive data with the external device via the communication module and/or the communication circuit.

Specifically, the communication unit (330) may include at least one short-range communication module (not shown) that performs communication according to a communication standard such as Bluetooth, Wi-Fi, BLE (Bluetooth Low Energy), NFC/RFID, Wi-Fi Direct, UWB, or ZIGBEE. Here, the short-range communication module (not shown) may also be referred to as a 'short-range communication unit (not shown)'.

In addition, the communication unit (330) may further include a remote communication module (not shown) that performs communication with a server (not shown) for supporting remote communication according to a remote communication standard. Specifically, the communication unit (330) may include a remote communication module (not shown) that performs communication via a network for Internet communication. In addition, the communication unit (330) may include a communication network that complies with communication standards such as 3G, 4G, and/or 5G. In addition, the remote communication module (not shown) may also be referred to as a 'remote communication unit (not shown)'.

Additionally, the communication unit (330) may include at least one port (not shown) for connecting to the mobile device (101) via a wired cable in order to communicate with the mobile device (101) via a wire. For example, the communication unit (330) may include a cable connection port, such as an HDMI port (not shown).

The user interface (350) may receive user input for controlling the display device (300). Specifically, the user interface (350) may receive an input for selecting at least one object included in a first image. Alternatively, the user interface (350) may receive an input for excluding at least one object included in the first image.

Specifically, when the user interface (350) receives an input to exclude at least one object from among at least one object included in the first image, the processor (310) may consider the remaining objects, excluding the object excluded by the user input, among the at least one object included in the first image, as selected. For example, when the first image includes a first, a second, and a third object, and the second object is excluded by the user input, the first and third objects may be considered to have been selected by the user.

The user interface (350) may include, but is not limited to, a user input device including a touch panel that detects a user's touch, a button that receives a user's push operation, a wheel that receives a user's rotation operation, a keyboard, and a dome switch.

Additionally, the user interface (350) may include a voice recognition device (not shown) for voice recognition. For example, the voice recognition device (not shown) may be a microphone, and the voice recognition device may receive a user's voice command or voice request. Accordingly, the processor (310) may control the execution of an operation corresponding to the voice command or voice request.

Additionally, the user interface (350) may include a motion detection sensor (not shown). For example, the motion detection sensor (not shown) may detect movement of the display device (3100) and receive the detected movement as user input. Additionally, the voice recognition device (not shown) and the motion detection sensor (not shown) described above may not be included within the user interface (350), but may be included within the display device (300) as modules independent of the user interface (350).

In addition, the user interface (350) may receive user input through a remote control device, etc. In this case, the user interface (350) may include a communication module for receiving a signal corresponding to the user input from the remote control device (not shown). For example, if the remote control device (not shown) transmits an IR signal corresponding to the user input, the user interface (350) may include a communication module (not shown) capable of receiving the IR signal.

In addition, when the display device (400) and an external mobile device (101) are wirelessly connected to each other, the user interface (350) can receive user input received through a touch panel (not shown) of the mobile device (101).

Alternatively, user input received through the touch panel (not shown) of the mobile device (101) may be received through the communication unit (330) of the display device (400) and transmitted to the processor (310). For example, when the display device (400) and an external mobile device (101) are wirelessly connected to each other through a Wi-Fi network or a Bluetooth network, user input received through the touch panel (not shown) of the mobile device (101) may be received through the Wi-Fi communication module (not shown) or the Bluetooth communication module (not shown) of the communication unit (330).

FIG. 4 is another box diagram illustrating a display device according to the disclosed embodiment. The display device (400) illustrated in FIG. 4 may correspond identically to the display device (300) illustrated in FIG. 3. Referring to FIG. 4, the display device (400) may further include at least one of a camera (340), a memory (370), and an audio input/output unit (365) compared to the display device (300).

The camera (340) includes an image sensor (not shown) and can capture image frames, such as still images or moving images, through the image sensor in video call mode or shooting mode. In the disclosed embodiment, the camera (340) can be activated based on the receipt of a video call request. Accordingly, the activated camera (340) can perform shooting to capture an image of the user who requested the video call.

Specifically, the camera (340) can acquire video frames at predetermined time intervals according to a set FPS (Frames per second). For example, the camera (340) can acquire 30 video frames per second. Accordingly, the first image acquired by the camera (330) can refer to a set of images or each image that is continuously acquired and updated.

Additionally, the display device (400) may include a single camera. Furthermore, the display device (400) may include multiple cameras positioned at different locations. When the display device (400) includes multiple cameras, the display device (400) may allow a user to select one of the multiple cameras for video calls based on its own settings or user input.

In addition, the camera for obtaining an image of the user may be provided separately from the display device (400), rather than being included within the display device (400). For example, it may be an external camera (not shown) positioned on one side of or adjacent to the display device (400) so as to be able to photograph the front of the display device (400). In this case, the external camera (not shown) and the display device (400) may be connected via a wired or wireless signal line, and the external camera (not shown) may obtain an image of the user of the video call under the control of the display device (400).

For convenience of explanation, the following description will be given as an example of a case in which a camera for obtaining an image of a user for a video call is included in a display device (400).

The memory (370) may store at least one instruction. Furthermore, the memory (370) may store at least one instruction executed by the processor (310). Furthermore, the memory (370) may store at least one program executed by the processor (310). Furthermore, the memory (370) may store an application for providing a specific service. For example, the memory (370) may store a video call application.

Specifically, the memory (370) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a RAM (Random Access Memory), a SRAM (Static Random Access Memory), a ROM (Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a PROM (Programmable Read-Only Memory), a magnetic memory, a magnetic disk, and an optical disk.

The user interface (350) can receive user input for controlling the display device (400). The user interface (350) can include, but is not limited to, a user input device including a touch panel that detects a user's touch, a button that receives a user's push operation, a wheel that receives a user's rotation operation, a keyboard, and a dome switch.

Additionally, the user interface (350) may include a voice recognition device (not shown) for voice recognition. For example, the voice recognition device (not shown) may be a microphone, and the voice recognition device may receive a user's voice command or voice request. Accordingly, the processor (310) may control the execution of an operation corresponding to the voice command or voice request.

Additionally, the user interface (350) may include a motion detection sensor (not shown). For example, the motion detection sensor (not shown) may detect movement of the display device (3100) and receive the detected movement as user input. Additionally, the voice recognition device (not shown) and the motion detection sensor (not shown) described above may not be included within the user interface (350), but may be included within the display device (400) as modules independent of the user interface (350).

In addition, the user interface (350) may receive user input through a remote control device, etc. In this case, the user interface (350) may include a communication module for receiving a signal corresponding to the user input from the remote control device (not shown). For example, if the remote control device (not shown) transmits an IR signal corresponding to the user input, the user interface (350) may include a communication module (not shown) capable of receiving the IR signal.

The audio input/output unit (365) can output audio that can be audibly recognized by a user, or can detect and/or receive audio. Specifically, the audio input/output unit (365) can include an audio output unit (360) and an audio input unit (367).

The audio output unit (360) outputs audio under the control of the processor (310).

Specifically, the audio output unit (360) can output audio (e.g., voice, sound) input through the communication unit (330).

In the disclosed embodiment, the processor (310) can control audio received from the other party device (102) of the video call during the video call mode to be output through the audio output unit (360).

In addition, the audio output unit (360) can output audio stored in the memory (370) under the control of the processor (310). The audio output unit (360) can include at least one of a speaker (not shown), a headphone output terminal (not shown), or a S/PDIF (Sony/Philips Digital Interface) output terminal (not shown). The audio output unit (360) can include a combination of a speaker (not shown), a headphone output terminal (not shown), and a S/PDIF (Sony/Philips Digital Interface) output terminal (not shown).

The audio input unit (367) receives audio. Specifically, the audio input unit (367) may include a microphone (specifically, a microphone) (not shown) that receives audio, which is an external audio signal, and processes it into electrical voice data. For example, the microphone (not shown) included in the audio input unit (367) may receive an audio signal from an external device or a speaker, for example, a user during a video call. In addition, the microphone (not shown) included in the audio input unit (367) may utilize various noise removal algorithms to remove noise generated in the process of receiving an external audio signal.

In the disclosed embodiment, under the control of the processor (310), the audio input unit (367) can receive an audio signal including the user's voice during a video call mode, and process the received audio signal to remove noise so that the user's voice can be recognized.

FIG. 5 is a diagram illustrating a plurality of electronic devices used to perform a video call service according to the disclosed embodiment. In the configurations illustrated in FIG. 5, identical configurations as those illustrated in FIGS. 1 to 4 are depicted using the same drawing symbols. Furthermore, the display device (400) illustrated in FIG. 5 may correspond to the display device (103, or 300) illustrated in FIGS. 1 to 3.

Referring to FIG. 5, the server (500) can communicate with a mobile device (101), a counterpart device (102), and a display device (400) through a wireless communication network (301).

For example, the server (500) may be a server that provides video calls or video call services, or supports communications required for video call services. For example, at least two devices among a mobile device (101), a counterpart device (102), and a display device (400) are connected via a mobile communication network such as 3G, 4G, or 5G, and a video call can be performed between the mobile device (101) or the display device (400) and the counterpart device (102). In this case, the server (500) may be a communication server that supports mobile communications according to communication standards such as 3G, 4G, or 5G.

In addition, the mobile device (101), the counterpart device (102), and the display device (103) may be connected to an Internet communication network. Furthermore, at least two of the mobile device (101), the counterpart device (102), and the display device (400) may each perform a video call through an application that provides a video call service within the devices. Then, the mobile device (101), the counterpart device (102), and the display device (103) may be able to perform a video call and/or a video call by switching devices through the application. In this case, the server (500) may be a server that supports the application. Specifically, the server (500) may be a server of a user who creates and/or distributes the application.

As described above, the operations of the video call execution method described above can be performed using the communication relay of the server (500). Although the communication relay operation of the server (500) is not illustrated in FIG. 3, the communication relay operation of the server (500) described above may be included in transmitting and receiving data or signals between two different devices. Specifically, the server (500) may be responsible for transmitting data or signals to support the execution (or switching execution) of a video call service.

For example, the server (500) may perform a relay operation to transmit a second image generated by the display device (400) to the counterpart device. In addition, the server (500) may transmit or transfer various data and signals used to perform a video call service to at least one of the mobile device (101), the counterpart device (102), and the display device (103).

In the following description, even if it is described that predetermined data or a signal is transmitted from one device (e.g., a display device) to another device (e.g., a counterpart device), this includes not only cases where predetermined data or a signal is directly transmitted from one device (e.g., a display device) to another device (e.g., a counterpart device), but also cases where predetermined data or a signal is transmitted from one device (e.g., a display device) to another device (e.g., a counterpart device) through a communication relay of a server (500).

Hereinafter, operations performed through a display device (103, 300 or 400) according to the disclosed embodiment will be described in detail with reference to FIGS. 6 to 16.

FIG. 6 is a flowchart illustrating a method for performing a video call according to the disclosed embodiment. Referring to FIG. 6, the method for performing a video call (600) illustrates a method for performing a video call performed through a display device according to the disclosed embodiment, for example, a display device (103, 300, or 400). In addition, the method may be a flowchart illustrating operations performed in the display device (103, 300, or 400) according to the disclosed embodiment. Therefore, in the operations included in the method for performing a video call (600) according to the disclosed embodiment, descriptions that overlap with the operations of the display device (103, 300, or 400) described above will be omitted.

Hereinafter, a method for performing a video call (600) will be described as an example in which the method is performed through the display device (400) described in FIG. 4.

Referring to FIG. 6, the video call performing method (600) recognizes at least one object included in a first image acquired for performing a video call (S610). Step S610 may be performed by the processor (310). The object recognition operation is described in detail below with reference to FIG. 7.

In addition, the video call performing method (600) receives a user input selecting at least one of the at least one object recognized in step S610 (S620). Step S620 may be performed by receiving the user input from the user interface (350). Alternatively, step S620 may be performed by receiving the user input from the communication unit (330).

Here, the user input may be a signal reflecting the user's intent to select at least one object recognized in step S610. Specifically, the user input may be received in response to a user interface screen. Examples of user input received in response to the user interface screen are described in detail below with reference to FIGS. 8 to 10.

In addition, the user input may be received in response to the recognition of the gesture of each of the users displayed in the first image. For example, the display device (400) may perform gesture recognition based on the images captured by the camera (340). For example, when the camera (340) captures user A1 (116), user A2 (117), and user A3 (118), and user A1 (116) indicates an ‘X’ shape with his arm, and user A2 (117) and user A3 (118) indicate an ‘O’ shape with his arm, the processor (310) may perform gesture recognition based on the images captured by the camera (340). Then, based on the gesture recognition result, the processor (310) may determine that user A1 (116) is selected, and user A2 (117) and user A3 (118) are not selected.

Additionally, user input may be received in response to voice recognition of each user displayed within the first image. For example, the processor (310) may perform voice recognition based on a voice signal received from the audio input unit (367) of the display device (400). Furthermore, based on the voice recognition results, selection or non-selection of each user displayed within the first image may be determined.

Continuing, the video call performing method (600) acquires a second image including an object selected from at least one object included in the first image based on the user input received in step S620 (S630). Step S630 may be performed by the processor (310). The operation of acquiring the second image will be described in detail below with reference to FIGS. 11 to 13.

In addition, the video call execution method (600) controls the second image acquired in step S630 to be transmitted to the other party's device (102) (S640). Step S640 may be performed in the communication unit (330) under the control of the processor (310).

The method (600) for performing a video call according to the disclosed embodiment will be described in more detail below with reference to FIGS. 7 to 16. Furthermore, in describing FIGS. 7 to 16, the display device (400) illustrated in FIG. 4 will be described. Furthermore, in FIGS. 7 to 16, the same components as those illustrated in FIGS. 1 to 6 are indicated by the same drawing symbols. Therefore, any overlapping descriptions will be omitted.

FIG. 7 is a diagram illustrating an object recognition operation according to the disclosed embodiment. Specifically, FIG. 7 illustrates an example of a first image (700).

Referring to FIG. 7, the camera (310) activated under the control of the processor (310) can capture the space in front of the display device (400) to obtain a first image (700).

The first image (700) can image the entire space in front of the display device (400), including user A1 (116), user A2 (117), and user A3 (118).

The camera (310) generally has a wider angle of view than the mobile device (101) so that it can capture the entire space in front of the display device (400). In addition, if the camera (310) has a zoom function, the lens of the camera (310) can capture the subject by enlarging it or capturing the subject by reducing it. For example, if the lens of the camera (310) is set to reduce the subject, the first image acquired by the camera (310) can image a wider space. In FIG. 7, an example is illustrated in which the camera (310) captures the upper bodies of users A1 (116), A2 (117), and A3 (118) to capture the first image (700).

In step S610, the processor (310) can recognize an object included in the first image (700). Specifically, the processor (310) can recognize at least one object included in the first image (700). The object recognized in the first image (700) may be at least one person, at least one object, and a background included in the first image (700). Here, the person may be an individually identifiable face of a person. In addition, the object may be furniture, an object, an animal, etc. that are identifiable in the first image (700). In addition, the background may represent an environment representing a space represented by the first image (700).

Specifically, object recognition can be performed by various techniques that can individually distinguish and recognize objects, such as face detection techniques, pedestrian detection, and face recognition.

In addition, object recognition techniques based on deep learning can be utilized for object recognition. Specifically, methods for performing object recognition, object tracking, and object discrimination using artificial intelligence (AI) technology that performs operations through neural networks are being developed and utilized. For convenience of explanation, the operations of analyzing images to perform object recognition, object tracking, and object discrimination will be collectively referred to as “object recognition.”

Specifically, artificial intelligence technology analyzes input images or data by performing computations through a neural network to obtain a desired result. Here, the neural network can be trained on learning data (e.g., multiple different images) to optimize and set internal weight values. Then, through the neural network with optimized weight values, the input data is learned on its own, thereby outputting the desired result.

Specifically, the neural network can be a deep neural network (DNN). In addition, deep neural network (DNN) operations can include convolutional neural network (CNN) operations, etc. Specifically, a data recognition model for object recognition can be implemented through the exemplified neural network, and the implemented recognition model can be trained using training data. Then, input data, for example, images captured by a camera, can be analyzed or classified using the trained data recognition model, and objects can be recognized in each of the input images and the recognized objects can be output as output data. In addition, a convolutional neural network (CNN) refers to all neural networks that perform an algorithm to analyze images and find patterns, and there can be various types of neural networks.

That is, in the disclosed embodiment, the neural network may be a neural network trained to input an image and extract objects within the image.

In the disclosed embodiment, a neural network for object recognition may be implemented within a processor (310). In this case, when the processor (310) inputs a first image (700) into a neural network (not shown), the neural network (not shown) analyzes the first image (700) to separate and extract each of the objects contained therein, namely, user A1 (116), user A2 (117), and user A3 (118).

Objects recognized in the first image (700), which is a result output from a neural network (not shown), can be displayed as boxes 710, 720, and 730. That is, the neural network (not shown) can recognize different faces recognized in the first image (700) and output the recognized different faces by distinguishing them as boxes 710, 720, and 730.

Additionally, the neural network (not shown) may be implemented in a server or external device outside of the display device (400). In this case, the display device (400) may transmit the first image (700) to the neural network (not shown) implemented in the external server or external device, and receive the result output from the neural network (not shown) through the communication unit (330).

Figure 8 is another flowchart illustrating a method for conducting a video call according to the disclosed embodiment. In Figure 8, operations identical to those illustrated in Figure 6 are depicted using the same drawing symbols, and thus a detailed description thereof will be omitted. Furthermore, the video call conducting method (800) will be described as being performed via the display device (400) described in Figure 4, as an example.

Referring to FIG. 8, the video call performing method (800) may further include step S615 compared to the video call performing method (600).

The video call performing method (800) may further include, following step S610, a step (S615) of outputting a user interface screen for selecting at least one object included in the first image (700). Here, step S615 may be performed by having the display (320) display the user interface screen under the control of the processor (310).

The user interface screen output at step S615 is described in detail below with reference to FIGS. 9 and 10.

FIG. 9 is a drawing showing an example of a user interface screen output in the disclosed embodiment.

Referring to FIG. 9, an example of a user interface screen (900) output at step S615 is illustrated. For example, the user interface screen (900) includes the first image (700) illustrated in FIG. 7, and may include a user interface menu for individually selecting at least one object recognized in the first image (700).

For example, the user interface screen (900) may include an indication (e.g., box indications 710, 720, 730, etc.) presenting at least one object recognized in the first image (700) and a pointer (920) for individually selecting at least one object.

A user of a display device (400) can use a pointer (920) to select or deselect at least one object recognized in the first image (700), namely, user A1 (116), user A2 (117), and user A3 (118). Here, deselection means not selecting. Specifically, the user can deselect (or deselect) an object that he or she does not want to include in the video call screen.

In addition, the user of the display device (400) may be a person who requested a video call, or a person with legal use rights of the display device (400) (e.g., the owner of the display device (400)), and may be at least one of the objects, user A1 (116), user A2 (117), and user A3 (118).

Referring to FIG. 9, a user of a display device (400) can select or deselect each of the objects, user A1 (116), user A2 (117), and user A3 (118), by manipulating a pointer (920). In addition, the selection or deselection operation can be performed in a set manner.

For example, when a user double-clicks a point within a box (710) indicating an object to be selected by manipulating the pointer (920) or points to it for a predetermined period of time, the processor (310) can recognize that the object (e.g., user A1 (116) corresponding to the box (710)) has been selected.

As another example, when a user manipulates the pointer (920) to move the pointer (920) along the path (910) from a point within the box (710) indicating an object to be selected to a point outside the box (710), the object (e.g., user A1 (116) corresponding to the box (710)) may be recognized as being released (or not selected).

As another example, when a user manipulates a pointer (920) and clicks once on a point within a box (710), the processor (310) may recognize that the object (e.g., user A1 (116) corresponding to the box (710)) is selected, and when a user manipulates a pointer (920) and clicks once on a point within the box (710), the processor (310) may recognize that the object (e.g., user A1 (116) corresponding to the box (710)) is not selected.

As another example, a user may click only the box corresponding to the object to be selected by manipulating the pointer (920) with the pointer (920). In this case, the processor (310) may recognize the object for the box clicked by the pointer (920) (e.g., user A1 (116) corresponding to the box (710)) as selected, and recognize the object for the box not clicked by the pointer (920) as not selected.

As another example, if no user input for object selection is received for a predetermined period of time (e.g., 10 seconds) after the user interface screen (900) is output, the processor (310) may determine that all objects presented within the user interface screen (900) (specifically, at least one object recognized in step 610) are selected.

Additionally, the user may want the other party to not know where the video call is taking place. In such a case, the user may manipulate the pointer (920) to deselect the background portion. For example, if the background is not selected by the user (i.e., deselected), the processor (310) may process the background captured in the first image (700) by blurring, mosaic-processing, or erasing the background entirely, thereby obscuring the background, thereby generating a second image.

FIG. 10 is a drawing showing another example of a user interface screen output in the disclosed embodiment. Referring to FIG. 10, another example of a user interface screen (1000) output in step S615 is illustrated.

For example, the user interface screen (1000) may include a first image (700) as illustrated in FIG. 7, and may include at least one of a selection menu (e.g., 1011) and a release menu (e.g., 1012) for individually selecting at least one object recognized in the first image (700).

A user of the display device (400) can select or deselect at least one object recognized in the first image (700), namely user A1 (116), user A2 (117), and user A3 (118), by clicking on a selection menu (e.g., 1011) and a deselection menu (e.g., 1012) using a pointer (920).

In addition to the examples shown in FIGS. 9 and 10, the user interface screen for individually selecting at least one recognized object may be modified or generated in various forms.

In the disclosed embodiment, the processor (310) can acquire a second image that includes at least one object selected by a user input and does not include at least one object not selected by the user input. The acquisition operation of the second image is described in detail below with reference to FIGS. 11 to 13.

FIG. 11 is a drawing showing an example of an image generated by a display device according to the disclosed embodiment.

Referring to FIG. 11, when all of the objects recognized in the first image (700), namely user A1 (116), user A2 (117), and user A3 (118), are selected (S1110), the processor (310) can acquire the first image (700) as a second image (1130).

Alternatively, if only some of the objects recognized in the first image (700), such as user A1 (116), user A2 (117), and user A3 (118), are selected (S1120), the first image (700) may be edited to include only the selected objects to generate the second image (1140).

For example, if user A2 (117) is not selected by user input, the processor (310) may edit the unselected user A2 (117) in the first image (700) by removing, blurring, mosaicing, displaying it as an image representing a virtual object, etc., to generate a second image (1140). In FIG. 11, an example is shown in which the processor (310) generates the second image (1140) by removing or deleting the unselected user A2 (117) in the first image (700). In addition, the processor (3100) may perform background restoration on the area where A2 (117) is removed or deleted, so that a natural second image (1140) is generated.

FIG. 12 is a diagram illustrating another example of an image generated by a display device according to the disclosed embodiment. Referring to FIG. 12, a second image (1200) generated when user A2 (117) is not selected in the first image (700) is illustrated.

Referring to FIG. 12, the processor (310) can generate a second image (1140) by displaying a replacement image (1220) in the area (1210) where the user A2 (117) who was not selected in the first image (700) was imaged. Here, the replacement image is an image representing a virtual object and can be an image generated by the processor (310) itself. Alternatively, the replacement image can be an image set or selected in advance by the user.

FIG. 13 is a diagram illustrating another example of an image generated by a display device according to the disclosed embodiment. Referring to FIG. 13, a second image (1300) generated when user A2 (117) is not selected in the first image (700) is illustrated.

Referring to FIG. 13, the processor (310) can delete the area (1210) where the user A2 (117) who was not selected in the first image (700) was imaged, and change at least one of the position, size, and magnification of the remaining objects, user A1 (116) and user A3 (118), to generate the second image (1300).

Specifically, if the area (1210) where the user A2 (117) who was not selected in the first image (700) was imaged is deleted, the space utilization efficiency of the first image (700) may decrease and the user A1 (116) and the user A3 (118) may appear awkward because they are too far apart. Therefore, in order to increase the space utilization efficiency of the objects included in the second image (1300) and prevent the objects from appearing awkward, the processor (310) may change at least one of the position, size, and magnification of the selected objects, user A1 (116) and user A3 (118), to generate the second image (1300). The second image (1300) may include the changed position of the user A1 (116a) and the changed position of the user A3 (118a).

Additionally, an image generation technology based on artificial intelligence technology may be used to generate a second image (1140) by changing at least one of the position, size, and magnification of the selected objects, user A1 (116) and user A3 (118).

Referring back to FIG. 6, the second image (e.g., 1300) generated in step S630 is transmitted to the counterpart device (102) (S640). Step S640 may be performed in the communication unit (330) under the control of the processor (310). Specifically, the second image may be directly transmitted from the communication unit (330) of the display device (400) to the counterpart device (102).

Alternatively, the second image may be transmitted to the counterpart device (102) using a communication relay of a server (e.g., 500 of FIG. 5). The communication unit (330) of the display device (400) may transmit the second image to the server (e.g., 500 of FIG. 5), and the server (e.g., 500 of FIG. 5) may then transmit the second image back to the counterpart device (102).

Specifically, the processor (310) may control the communication unit (330) to initiate transmission of the second image after a user input for selecting or deselecting a recognized object is received. Alternatively, the processor (310) may stop transmitting an image to be included in the video call screen until a user input for selecting or deselecting a recognized object is received. Here, the image to be included in the video call screen may be the first image when all objects identified in the first image are selected, and may be the second image when only some of the objects identified in the first image are selected.

Additionally, in the disclosed embodiment, object recognition may be performed repeatedly while a video call is being performed. For example, if the camera (340) operates at a predetermined frame per second (FPS) and acquires multiple frames per second, object recognition may be performed for each frame. Alternatively, object recognition may be performed at a fixed frame interval. For example, when 1 to 100 frames are sequentially acquired, object recognition may be performed every 10 frames. Alternatively, object recognition may be performed at a predetermined interval, such as every second.

Specifically, when a video call is initiated, each of the recognized multiple objects can be assigned an identification number, and the recognized objects can be tracked based on the assigned identification number. Accordingly, it will be possible to periodically determine whether the identified object continues to be maintained during the video call or whether a new, unidentified object is recognized. Here, in tracking the recognized object, at least one of the visual features of the recognized object and the location information corresponding to the recognized object can be utilized.

When tracking a recognized object, if the image data values of the entire recognized object are used to determine whether the recognized object is maintained or a new object appears for each frame, the amount of computation may be large. Therefore, to speed up the computation, visual features within the image, such as brightness, shading information, edge information, and/or vertex information of specific pixels extracted from the image, can be extracted, and based on the extracted visual features, it can be determined whether the recognized object is maintained or a new object appears. Alternatively, based on the position where the object was recognized, it can be determined whether the recognized object continues to exist within a certain offset range from that position, thereby determining whether the recognized object is maintained or a new object appears.

As described above, performing object tracking based on visual features and final location information of recognized objects can reduce the amount of computation required to determine whether a recognized object is maintained or a new object has appeared.

Additionally, in the disclosed embodiment, selection and deselection of objects recognized in the first image acquired by the camera (340) may be re-performed at predetermined time intervals or upon user request. For example, user A1 (116) may want his/her image to be included in a video call screen, and then may want his/her image to be excluded from the video call screen again. In this case, based on the request of user A1 (116), the processor (310) may de-select user A1 (116) recognized in the first image.

Fig. 14 is another flowchart illustrating a method for conducting a video call according to the disclosed embodiment. In Fig. 14, operations identical to those illustrated in Fig. 8 are depicted using the same drawing symbols, and thus detailed descriptions are omitted. Furthermore, the video call conducting method (1400) will be described as an example in which the method is performed via the display device (400) described in Fig. 4.

Referring to FIG. 14, the video call performing method (1400) may further include steps S650, S655, S660, S670, and S680 compared to the video call performing method (800).

Specifically, the video call performing method (1400) can determine whether a new object is recognized in the updated first image when the camera (340) acquires the updated first image (S650). Here, step S650 can be performed by the processor (310).

Specifically, considering the frame rate of the camera (340), it is possible to determine whether a new object is recognized in the updated first image at a predetermined frame interval or a predetermined time interval, or whether a previously recognized object is excluded. That is, the ‘new object recognition’ illustrated in step S650 of FIG. 14 may have a meaning that includes both cases where a previously unrecognized object is additionally recognized and/or cases where a previously recognized object disappears.

As a result of the judgment of S650, if a new object is recognized, a user interface screen for selecting a new object can be output to the display (320) (S660). In addition, as a result of the judgment of S650, if there is no change in the recognized object, the process can return to step S650 according to step S655. Here, steps S655 and S660 can be performed under the control of the processor (310). Since the output operation of the user interface screen for selecting an object corresponds to the aforementioned step S615, a detailed description thereof will be omitted.

Continuing, the video call performing method (1400) may update the second image to include the new object based on a user input selecting the new object (S670). Here, step S670 may be performed by the processor (310).

In addition, the video call execution method (1400) can transmit the updated second image generated in S670 to the other party's device (102) (S680). Here, step S680 can be performed in the communication unit (330) under the control of the processor (310).

Figure 15 is a drawing for explaining the operations shown in Figure 14.

FIG. 16 is a drawing showing another example of an image generated by a display device according to the disclosed embodiment.

In FIGS. 15 and 16, the same configurations as those illustrated in FIGS. 1 and 4 are depicted using the same drawing symbols. Therefore, redundant descriptions are omitted.

Referring to FIG. 15, a first image (1550) acquired for a video call from a camera (340) at time t1 includes user A1 (116), user A2 (117), and user A3 (118). That is, user A1 (116), user A2 (117), and user A3 (118) were present in the space in front of the display device (400). If user A1 (116), user A2 (117), and user A3 (118), which are objects recognized in the first image, are all selected, the second image (1522) may be identical to the first image (1550), and is an image including user A1 (116), user A2 (117), and user A3 (118).

At time t2 following time t1, the camera (340) can capture a new first image (1560) for a video call. For example, a new user, user A4 (119), is additionally positioned in the space in front of the display device (400) at time t2 following time t1. Therefore, the first image (1560) acquired at time t2 includes a new object, user A4 (119), compared to the first image (1550) acquired at time t1, and the processor (310) can recognize the new object, user A4 (119), at step S650.

Referring to FIG. 16, at step S660, the processor (310) can control a user interface screen (1600) for selection or deselection of a newly recognized object, for example, user A4 (119), to be output on the display (320).

Accordingly, the user can input a user input indicating selection or non-selection of a newly recognized object into the display device (400). Then, the processor (310) can generate an updated second image based on the user input (S670).

For example, when a newly recognized object, for example, user A4 (119), is selected (S1621), the processor (310) can generate a second image (1630) so that it includes user A4 (119) and previously selected objects, such as user A1 (116), user A2 (117), and user A3 (118).

Additionally, if a newly recognized object, for example, user A4 (119), is not selected (S1622), the processor (310) may generate a second image (1650) so that user A4 (119) is excluded and only the previously selected objects, user A1 (116), user A2 (117), and user A3 (118) are included.

In addition, the disclosed embodiment can output a user interface screen (e.g., 900 of FIG. 9 or 1000 of FIG. 10) for selecting an object to be included in a video call screen in response to a user input for changing whether objects are included in a video call screen. For example, if at least one of a plurality of users included in a video call screen no longer wants his or her image to be included in the video call screen, a user input requesting this can be transmitted to the display device. Then, the display device can output a user interface screen (e.g., 900 of FIG. 9 or 1000 of FIG. 10) for selecting an object to be included in the video call screen in response to receiving the user input.

Alternatively, the disclosed embodiment may output a user interface screen (e.g., 900 of FIG. 9 or 1000 of FIG. 10) for selecting an object to be included in the video call screen at predetermined time intervals after the video call screen is displayed. Accordingly, if a user does not wish to be included in the video call screen after the video call has been initiated, the second image may be updated to reflect the user's intention.

As described above, the disclosed embodiments described with reference to FIGS. 1 to 16 can provide an image (specifically, the second image described above) constituting the video call screen in a display device that performs video calls using a large screen, so that the privacy of at least one user can be protected according to the intention of each user during the video call. Accordingly, by flexibly protecting the privacy of the user during the video call, user satisfaction can be increased.

A method for performing a video call according to one embodiment of the present disclosure may be implemented in the form of program commands that can be executed via various computer means and recorded on a computer-readable medium. Furthermore, an embodiment of the present disclosure may be a computer-readable recording medium having recorded thereon one or more programs containing commands for executing the method for performing a video call.

The above computer-readable medium may include program commands, data files, data structures, etc., either singly or in combination. The program commands recorded on the medium may be those specially designed and configured for the present invention, or may be those known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands, such as ROMs, RAMs, and flash memories. Examples of program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc.

Here, the device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, ‘non-transitory’ only means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and this term does not distinguish between cases where data is stored semi-permanently in the storage medium and cases where data is stored temporarily. For example, a ‘non-transitory storage medium’ may include a buffer in which data is temporarily stored.

According to one embodiment, the method for performing a video call according to various embodiments disclosed in the present document may be provided as a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

Specifically, the method for performing a video call according to the disclosed embodiment can be implemented as a computer program product including a recording medium storing a program.

Although the embodiments have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements made by those skilled in the art using the basic concept of the present invention defined in the following claims also fall within the scope of the present invention.

300, 400: Display devices
301: Communications Network
310: Processor
320: Display
330: Communications Department
340: Camera
350: User Interface
360: Audio output
365: Audio input/output section
367: Audio input
370: Memory

Claims (20)

display;
A communication unit that performs communication with external devices;
a user interface that receives user input; and
comprising a processor that executes at least one instruction,
The above processor
Recognize multiple objects included in the first image acquired for performing a video call,
Controlling a user interface screen for selecting each of the plurality of objects included in the first image to be output through the display,
Based on a user input selecting at least one of the plurality of objects received corresponding to the output of the user interface screen, a second image including the selected object among the plurality of objects included in the first image is obtained,
Controlling the communication unit so that the second image is transmitted to the other party's device;
The above processor
A display device that obtains the second image by changing at least one of the position, size, and magnification of the selected object among the plurality of objects included in the first image. delete In the first paragraph, the processor
A display device that obtains the second image, which includes at least one object selected by the user input and does not include at least one object not selected by the user input. In the first paragraph, the processor
A display device that initiates transmission of the second image after the user input is received. In the first paragraph, the processor
A display device that stops transmitting images to be included in a video call screen until the above user input is received. In the first paragraph,
Includes more cameras,
The above processor
Obtaining the updated first image through the above camera,
When the camera acquires the updated first image, it determines whether a new object is recognized in the updated first image,
When the new object is recognized, a user interface screen for selecting the new object is controlled to be output through the display.
A display device that updates the second image to include the new object based on a user input selecting the new object. In the first paragraph, the processor
A display device that generates the second image by deleting the unselected object from the first image if there is an object among at least one object included in the second image that is not selected by the user input. In the first paragraph, the processor
A display device that generates the second image by processing the image so that a virtual object corresponding to the unselected object is included in the first image when there is an object among the plurality of objects included in the first image that is not selected by the user input. In the first paragraph,
Includes more cameras,
The above processor
A display device that activates the camera in response to a video call request and acquires the first image through the activated camera. In the first paragraph, the processor
A display device that recognizes at least one person, a background of a space, and an object included in the space included in the first image, and controls a user interface screen for selecting or deselecting each of the recognized at least one person, background of a space, and object included in the space to be output through the display. In the first paragraph, the processor
A display device that performs object recognition by obtaining the plurality of objects output from the neural network when the neural network, which has received the first image as input, performs an operation for object recognition to extract and output a plurality of objects included in the first image. In a method for performing a video call through a display device,
A step of recognizing a plurality of objects included in a first image acquired for performing a video call;
A step of outputting a user interface screen for selecting each of the plurality of objects included in the first image;
A step of receiving a user input for selecting at least one of the plurality of objects corresponding to the output of the user interface screen;
A step of obtaining a second image including a selected object among the plurality of objects included in the first image based on the user input; and
Including a step of transmitting the second image to the other party's device,
The step of obtaining the above second image is
A step of changing at least one of the position, size, and magnification of the selected object among the plurality of objects included in the first image; and
A method for performing a video call, comprising the step of generating a second image including a selected object having at least one of the position, size, and magnification changed. delete In the 12th paragraph, the step of obtaining the second image
A method for performing a video call, comprising the step of obtaining the second image including at least one object selected by the user input and not including at least one object not selected by the user input. In the 12th paragraph, the transmitting step
A method for performing a video call, comprising the step of starting to transmit the second image to the other party's device after the user input is received. In paragraph 12,
When the first image is updated, a step of determining whether a new object is recognized in the updated first image;
When the new object is recognized, a step of outputting a user interface screen for selecting the new object; and
A method for performing a video call, further comprising the step of updating the second image to include the new object based on a user input selecting the new object. In the 12th paragraph, the step of obtaining the second image
A method for performing a video call, further comprising the step of generating the second image by deleting the unselected object from the first image if there is an object among at least one object included in the second image that is not selected by the user input. In the 12th paragraph, the step of obtaining the second image
A method for performing a video call, further comprising the step of generating the second image by performing image processing so that a virtual object corresponding to the unselected object is included in the first image, if there is an object among the plurality of objects included in the first image that is not selected by the user input. In the 12th paragraph, the step of recognizing the object
A step of receiving an output of the neural network when the neural network that has received the first image performs an operation for object recognition to extract and output a plurality of objects included in the first image; and
A method for performing a video call, comprising the step of obtaining the output of the above nerves as the plurality of objects. A non-transitory computer-readable storage medium having recorded thereon a program comprising at least one instruction for performing a method of performing a video call through a display device,
How to make the above video call
A step of recognizing a plurality of objects included in a first image acquired for performing a video call;
A step of outputting a user interface screen for selecting each of the plurality of objects included in the first image;
A step of receiving a user input for selecting at least one of the plurality of objects corresponding to the output of the user interface screen;
A step of obtaining a second image including an object selected from among the plurality of objects included in the first image based on the user input; and
Including a step of transmitting the second image to the other party's device,
The step of obtaining the above second image is
A step of changing at least one of the position, size, and magnification of the selected object among the plurality of objects included in the first image; and
A storage medium comprising a step of generating the second image including the selected object having at least one of the changed position, size, and magnification.