CN115762108B

CN115762108B - Remote control method, remote control device, and controlled device

Info

Publication number: CN115762108B
Application number: CN202210248607.6A
Authority: CN
Inventors: 牛洋; 任哲坡; 汤琼; 丁洪霞
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2021-08-30
Filing date: 2022-03-14
Publication date: 2026-01-23
Anticipated expiration: 2042-03-14
Also published as: WO2023030067A1; CN115762108A

Abstract

The embodiment of the invention provides a remote control method, remote control equipment and controlled equipment. In the technical scheme provided by the embodiment of the invention, the controlled equipment comprises a screen, and the method comprises the steps of receiving first information of remote control equipment, determining a target position of the remote control equipment, which is opposite to the screen, according to the first information, and executing first operation on the screen according to the target position. According to the embodiment of the invention, the remote operation of the controlled equipment can be realized on the premise that the user does not need to perform mechanical frequent operation on the five-way navigation key, and the experience of the user in the use process is improved.

Description

Remote control method, remote control device, and controlled device

[ Field of technology ]

The present invention relates to the field of remote control technologies, and in particular, to a remote control method, a remote control device, and a controlled device.

[ Background Art ]

The current remote control method of the controlled device mainly comprises two types, one is remote control of the remote controller, and the other is remote control of the controlled device by using a mobile phone.

The existing remote controllers, whether infrared remote controllers or Bluetooth remote controllers, comprise five-way navigation keys. The user is easy to get on hand when using the remote controller to remotely control the controlled device, but the operation experience of the user is poor compared with a mobile phone and a tablet which are similar interactive products. Under the trend of large screens, more and more contents can be displayed on the screen, and when a user uses a basic five-way guide key, the user wants to quickly locate the desired contents, and the user needs to perform mechanical frequent operation to realize the quick positioning. The mobile phone is used as a large-screen remote controller, the mobile phone is required to have an infrared function, and corresponding software is required to be installed on the mobile phone for supporting. In addition, the mobile phone is used as a remote controller of the controlled equipment, can be limited by the distance and the infrared emission angle, and the problem of interaction completion through the back and forth switching of the five-way navigation key is not fundamentally solved, so that experience is poor in the use process.

Therefore, the remote control method of the controlled device at present requires the user to perform mechanical frequent operation on the five-way navigation key to realize remote operation on the large screen, so that experience is poor in the use process.

[ Invention ]

In view of the above, the embodiments of the present invention provide a remote control method, a remote control device, and a controlled device, which can enable a user to implement remote operation on the controlled device without performing mechanical frequent operation on five-way navigation keys, thereby improving experience of the user in the use process.

In a first aspect, an embodiment of the present invention provides a remote control method applied to a controlled device, where the controlled device includes a screen, the method including:

Receiving first information of remote control equipment;

determining a target position of the remote control equipment, which is opposite to the screen, according to the first information;

And executing a first operation on the screen according to the target position.

In one possible implementation, the first information includes position coordinates of the target position of the remote control device facing the screen;

determining a target position of the remote control device facing the screen according to the first information, wherein the target position comprises:

And determining the target position corresponding to the position coordinate on the screen according to the position coordinate.

In one possible implementation, the position coordinates of the target position are derived from an image containing the outline of the screen taken by a binocular camera of the remote control device.

In one possible implementation, the first information includes an image containing a contour of the screen taken by a binocular camera of the remote control device;

the determining the target position of the remote control device facing the screen according to the first information comprises determining the target position corresponding to the position coordinate on the screen according to the image.

In one possible implementation manner, the determining, according to the image, a target position on the screen corresponding to the position coordinate includes:

Obtaining position coordinates of the target position, which is opposite to the screen, of the remote control equipment according to the image;

In one possible implementation, the first operation includes displaying a pointer at the target location, or highlighting or shading a control corresponding to the target location.

In another aspect, an embodiment of the present invention provides a remote control method, applied to a remote control device, where the method includes:

Acquiring first information, wherein the first information is used for determining a target position of a screen of the remote control equipment, which is opposite to the controlled equipment;

And sending the first information to the controlled device so that the controlled device determines the target position according to the first information and performs a first operation on the screen according to the target position.

In one possible implementation, the remote control device includes a binocular camera, and the first information includes position coordinates of the target position of the remote control device facing the screen;

Acquiring first information, including:

When the remote control device points to the screen, shooting an image containing the outline of the screen through the binocular camera;

and obtaining the position coordinates of the target position, which is opposite to the screen, of the remote control equipment according to the image.

In one possible implementation, the remote control device includes a binocular camera, and the first information includes an image including a contour of the screen captured by the binocular camera;

Acquiring first information, including:

and when the remote control device points to the screen, shooting the image through the binocular camera.

In another aspect, an embodiment of the present invention provides a remote control apparatus applied to a controlled device, where the controlled device includes a screen, the apparatus including:

the receiving module is used for receiving the first information of the remote control equipment;

the determining module is used for determining the target position of the remote control equipment, which is opposite to the screen, according to the first information;

And the execution module is used for executing a first operation on the screen according to the target position.

the determining module is specifically configured to determine, according to the position coordinate, the target position on the screen corresponding to the position coordinate.

the determining module is specifically configured to determine, according to the image, the target position on the screen corresponding to the position coordinate.

In one possible implementation manner, the determining module is specifically configured to obtain, according to the image, a position coordinate of the target position where the remote control device is opposite to the screen, and determine, according to the position coordinate, the target position on the screen corresponding to the position coordinate.

In another aspect, an embodiment of the present invention provides a remote control apparatus, applied to a remote control device, including:

the remote control device comprises an acquisition module, a control module and a control module, wherein the acquisition module is used for acquiring first information, and the first information is used for determining a target position of a screen of the remote control device, which is opposite to the controlled device;

and the sending module is used for sending the first information to the controlled equipment so that the controlled equipment can determine the target position according to the first information and execute a first operation on the screen according to the target position.

The acquisition module is specifically configured to, when the remote control device points to the screen, capture an image including a contour of the screen through the binocular camera, and obtain, according to the image, a position coordinate of the target position of the remote control device facing the screen.

The acquisition module is specifically used for shooting the image through the binocular camera when the remote control equipment points to the screen.

In another aspect, an embodiment of the present invention provides a controlled device, including a screen, a processor, and a memory, where the memory is configured to store a computer program, the computer program including program instructions that when executed by the processor cause the controlled device to perform the steps of:

Receiving first information of remote control equipment;

And executing a first operation on the screen according to the target position.

In one possible implementation, the program instructions, when executed by the processor, cause the controlled device to perform the steps of:

the first information comprises position coordinates of the target position of the remote control device facing the screen;

the first information comprises an image containing the outline of the screen, which is shot by a binocular camera of the remote control device;

The determining, according to the image, a target position on the screen corresponding to the position coordinate includes:

In another aspect, an embodiment of the present invention provides a remote control device, including a processor and a memory, where the memory is configured to store a computer program, the computer program including program instructions that, when executed by the processor, cause the remote control device to perform the steps of:

In one possible implementation, the remote control device includes a binocular camera, the first information includes position coordinates of the target position of the remote control device facing the screen, and when the processor executes the program instructions, the remote control device is caused to perform the steps of:

Acquiring first information, including:

In a possible implementation, the remote control device comprises a binocular camera, the first information comprises an image taken by the binocular camera containing an outline of the screen, and when the processor executes the program instructions, the remote control device is caused to perform the steps of:

Acquiring first information, including:

In another aspect, embodiments of the present invention provide a computer readable storage medium storing a computer program comprising program instructions which, when executed by a computer, cause the computer to perform a method as described above.

The remote control method, the remote control device and the controlled device provided by the embodiment of the invention have the technical scheme that the controlled device comprises a screen, the method comprises the steps of receiving first information of the remote control device, determining a target position of the remote control device, which is opposite to the screen, according to the first information, and executing first operation on the screen according to the target position. According to the embodiment of the invention, the remote operation of the controlled equipment can be realized on the premise that the user does not need to perform mechanical frequent operation on the five-way navigation key, and the experience of the user in the use process is improved.

[ Description of the drawings ]

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed in the embodiments will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a block diagram of a remote control system according to an embodiment of the present invention;

FIG. 2 is a schematic diagram of a remote control system according to another embodiment of the present invention;

FIG. 3 is a schematic diagram of a pointing screen of a remote control device;

FIG. 4 is a schematic diagram of the outline of the screen itself and outline information of the screen;

FIG. 5 is a schematic diagram of the workflow of an algorithm module;

FIG. 6 is a flow chart of a remote control method according to an embodiment of the present invention;

Fig. 7 is a specific flowchart of the first operation corresponding to the first information performed on the screen according to the first information in fig. 6;

FIG. 8 is a flow chart of a remote control method according to another embodiment of the present invention;

FIG. 9 is a flowchart showing the first information obtained in FIG. 8 for determining a target position of a screen of a remote control device facing a controlled device;

Fig. 10 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

[ Detailed description ] of the invention

For a better understanding of the technical solution of the present invention, the following detailed description of the embodiments of the present invention refers to the accompanying drawings.

It should be understood that the described embodiments are merely some, but not all, embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

The terminology used in the embodiments of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in this application and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

It should be understood that the term "and/or" as used herein is merely an association relationship describing the associated object, and means that three relationships may exist, for example, a and/or b, and may mean that a single first exists while a single first and a single second exist. In addition, the character "/" herein generally indicates that the front and rear associated objects are an "or" relationship.

Because the controlled equipment is provided with an open operating system and a chip and is provided with an open application platform, the bidirectional man-machine interaction function can be realized, and people can freely select to watch any favorite program in free time. In addition, with the development of artificial intelligence technology, the controlled equipment can recommend programs conforming to the watching habit and preference of users, so that entertainment is greatly improved, and the controlled equipment gradually becomes an interaction center of the most important living room entertainment. But the interaction mode of the controlled device is realized by a remote control method.

In order to solve the technical problems, the embodiment of the invention provides a remote control method, remote control equipment and controlled equipment.

Referring to fig. 1, fig. 1 is a schematic diagram of a remote control system according to an embodiment of the present invention. The remote control system of the embodiment of the invention comprises two electronic devices, wherein the two electronic devices can comprise a controlled device and a remote control device, and the communication mode between the controlled device and the remote control device comprises wireless transmission. As shown in fig. 1, the remote control system includes a remote control device 100 and a controlled device 200. The remote control device 100 and the controlled device 200 may perform wireless transmissions, including bluetooth, infrared, mobile network, or WLAN transmissions, for example. Wherein the remote control device 100 comprises a binocular camera and a transmitting module. The controlled device 200 includes a receiving module, an algorithm module, and a screen.

In fig. 1, a binocular camera of a remote control device 100 is used for capturing an image of a screen of a controlled device 200, the image includes an outline of the screen and transmits the image to a transmitting module, and the transmitting module is used for transmitting first information to the controlled device 200 through a wireless transmission mode. Wherein the first information comprises an image containing the outline of the screen taken by the binocular camera of the remote control device 100. The receiving module of the controlled device 200 is configured to receive the first information and send the first information to the algorithm module. The algorithm module determines the position coordinates of the target position of the remote control device 100 facing the screen based on the first information. Specifically, the algorithm module obtains the position coordinates of the target position of the remote control device 100 facing the screen according to the image, and transmits the position coordinates to the screen. The screen is used for determining a target position corresponding to the position coordinates on the screen according to the position coordinates, and executing a first operation on the screen according to the target position.

Referring to fig. 2, fig. 2 is a schematic diagram of a remote control system according to another embodiment of the present invention. The remote control system of the embodiment of the invention comprises two electronic devices, wherein the two electronic devices can comprise a controlled device and a remote control device, and the communication mode between the controlled device and the remote control device comprises wireless transmission. As shown in fig. 1, the remote control system includes a remote control device 100 and a controlled device 200. The remote control device 100 and the controlled device 200 may perform wireless transmissions, including bluetooth, infrared, mobile network, or WLAN transmissions, for example. The remote control device 100 comprises a binocular camera, an algorithm module and a sending module. The controlled device 200 includes a receiving module and a screen.

In fig. 2, the binocular camera of the remote control apparatus 100 is used to capture an image of the screen of the controlled apparatus 200, the image containing the outline of the screen, and transmit the image containing the outline of the screen to the algorithm module. The algorithm module obtains the position coordinates of the target position of the remote control device 100 facing the screen according to the image, and sends the first information to the sending module. Wherein the first information includes position coordinates of a target position where the remote control device 100 is facing the screen. The sending module is configured to send the first information to the controlled device 200 through a wireless transmission manner. The receiving module of the controlled device 200 is configured to receive the first information and transmit the first information to the screen. The screen is used for determining a target position of the remote control device facing the screen according to the first information, and executing a first operation on the screen according to the target position. Specifically, the screen is used for determining a target position corresponding to the position coordinates on the screen according to the position coordinates, and executing a first operation on the screen according to the target position.

In fig. 1 and 2, the remote control device 100 includes a binocular camera. Specifically, as shown in fig. 3, a binocular camera is located at the front end of the remote control device 100, and can recognize the outline of the screen. Two images containing the outline of the screen are obtained by shooting the screen through the binocular camera, as shown in fig. 4, the left image containing the outline of the screen is shot by the left-eye camera, and the right image containing the outline of the screen is shot by the right-eye camera.

For example, binocular cameras include binocular depth cameras.

It should be noted that, as shown in fig. 3, when the remote control device 100 is facing the screen, the line between the center point of the remote control device 100 and the target position on the screen is parallel to the longer two sides of the remote control device 100.

The algorithm module is specifically configured to calibrate the binocular camera in an offline state, correct two images of the screen to remove distortion caused by the binocular camera, calculate a matching point between the two corrected images through binocular matching to obtain a parallax map, and calculate a position coordinate of the remote control device 100, which is opposite to the screen, through 3D coordinates according to the parallax map, as shown in fig. 5.

The purpose of calibrating the binocular cameras in an offline state is to align the two cameras by acquiring internal parameters and external parameters of the two cameras. Firstly, calibrating a left-eye camera to obtain internal parameters and external parameters of the left-eye camera, secondly, calibrating a right-eye camera to obtain internal parameters and external parameters of the right-eye camera, and finally, calibrating a binocular camera to obtain a translation rotation relationship between the left-eye camera and the right-eye camera. The internal parameters comprise focal length, image center, distortion coefficient and the like, and the external parameters comprise a rotation matrix and a translation matrix.

The difference of the two images in the X direction can be obtained by correcting the two images of the screen, so that the parallax calculation accuracy can be improved. Specifically, correcting the two images of the screen includes distortion correction and conversion to a standard form.

The binocular matching is a core part of binocular depth estimation, and the main purpose is to calculate the relative matching relation of pixels between two images. Specifically, the binocular matching includes 5 steps of matching error calculation, error integration, disparity map calculation, disparity map optimization and disparity map correction.

Wherein, reconstructing three-dimensional information of a screen in the picture according to the matching information in the parallax map and the triangulation principle, and obtaining the position coordinate of the remote control device 100 facing the screen through 3D coordinate calculation.

The first operation comprises the steps of displaying a pointer at a target position, or highlighting or shading a control corresponding to the target position.

It should be noted that, when the user moves the remote control device 100, the image including the outline of the screen acquired by the remote control device 100 may change, and the position coordinate of the remote control device 100 facing the screen needs to be recalculated, the controlled device 200 refreshes the pointer position according to the new position coordinate, and the pointer moves according to the pointing direction of the remote control device, or the controlled device 200 refreshes the highlighted or shaded control according to the new position coordinate.

Based on the architecture diagrams provided in fig. 1 and 2, a remote control method is provided in an embodiment of the present invention, and is applied to a controlled device 200, where the controlled device 200 includes a screen. Fig. 6 is a flowchart of a remote control method according to an embodiment of the present invention. As shown in fig. 6, the method includes:

Step 102, first information of a remote control device is received.

The first information comprises position coordinates of a target position of the remote control device facing the screen when the remote control device points to the screen, or comprises an image containing the outline of the screen, which is shot by a binocular camera of the remote control device.

In the embodiment of the present invention, as shown in fig. 1, the first information includes an image including a contour of a screen captured by a binocular camera of the remote control device, and the receiving module of the controlled device 200 receives the first information of the remote control device 100 and then transmits the first information to the algorithm module.

In the embodiment of the present invention, as shown in fig. 2, the first information includes the position coordinates of the target position where the remote control device 100 is facing the screen, and the receiving module of the controlled device 200 receives the first information transmitted by the remote control device 100 and then transmits the first information to the screen.

The remote control device comprises a binocular camera, and the position coordinates of the target position are obtained according to an image containing the outline of the screen, which is shot by the binocular camera of the remote control device. Specifically, the binocular camera is located at the front end of the remote control device, and can identify the outline of the screen, and two images containing the outline of the screen are obtained by shooting the screen through the binocular camera.

For example, binocular cameras include binocular depth cameras.

And 104, determining the target position of the remote control device facing the screen according to the first information.

Alternatively, the first information includes location coordinates of a target location of the remote control device facing the screen, and step 104 includes determining a target location on the screen corresponding to the location coordinates based on the location coordinates.

As shown in fig. 2, the receiving module of the controlled apparatus 200 transmits the first information to the screen, and the screen determines the target position corresponding to the position coordinates from the position coordinates in the first information.

As another alternative, the first information comprises an image comprising a contour of the screen captured by a binocular camera of the remote control device, and step 104 comprises determining a target location on the screen corresponding to the location coordinates from the image.

Specifically, as shown in fig. 7, step 104 includes:

Step 1042, obtaining the position coordinates of the target position of the remote control device facing the screen according to the image.

As shown in fig. 1, the algorithm module of the controlled apparatus 200 obtains the position coordinates of the target position of the remote control apparatus 100 facing the screen from the image, and then transmits the position coordinates to the screen.

Specifically, the algorithm module calibrates the binocular camera in an offline state, corrects two images of the screen to remove distortion caused by the binocular camera, calculates matching points between the corrected two images through binocular matching to obtain a parallax map, and calculates the position coordinate of the remote control device 100 facing the screen through 3D coordinates according to the parallax map.

Step 1044, determining a target position corresponding to the position coordinate on the screen according to the position coordinate.

As shown in fig. 1, the algorithm module of the controlled apparatus 200 transmits the position coordinates to the screen, and the screen determines the target position on the screen corresponding to the position coordinates according to the position coordinates in the first information.

And 106, executing a first operation on the screen according to the target position.

As shown in fig. 1 and 2, the screen of the controlled apparatus 200 performs a first operation according to the position coordinates at a target position corresponding to the position coordinates on the screen.

The first operation comprises the steps of displaying a pointer at the target position, or highlighting or shadow displaying a control corresponding to the target position.

When the user moves the remote control device, the image collected by the remote control device changes, the position coordinates need to be recalculated, the controlled device refreshes the pointer position according to the new position coordinates, the pointer moves according to the pointing direction of the remote control device, or the controlled device refreshes the highlighted or shadow display control according to the new position coordinates.

According to the technical scheme of the remote control method, first information of the remote control device is received, the target position of the remote control device, which is opposite to the screen, is determined according to the first information, and first operation is executed on the screen according to the target position. According to the embodiment of the invention, the remote operation of the controlled equipment can be realized on the premise that the user does not need to perform mechanical frequent operation on the five-way navigation key, and the experience of the user in the use process is improved.

Based on the architecture diagrams provided in fig. 1 and fig. 2, an embodiment of the present invention provides a remote control method applied to a remote control device 100. Fig. 8 is a flowchart of a remote control method according to another embodiment of the present invention. As shown in fig. 8, the method includes:

Step 202, acquiring first information, wherein the first information is used for determining a target position of a screen of a controlled device, which is opposite to the remote control device.

As an alternative, the remote control device comprises a binocular camera, the first information comprises position coordinates of a target position of the remote control device facing the screen, as shown in fig. 9, and step 202 comprises:

Step 2022, when the remote control device points to the screen, capturing an image containing the outline of the screen by the binocular camera.

The binocular camera is positioned at the front end of the remote control equipment, the outline of the screen can be identified, and the screen is shot by the binocular camera to obtain two images containing the outline of the screen.

For example, binocular cameras include binocular depth cameras.

In the embodiment of the present invention, as shown in fig. 2, the remote control device 100 photographs an image including the outline of the screen through the binocular camera and transmits the image to the algorithm module.

Step 2024, obtaining position coordinates of a target position of the remote control device facing the screen according to the image.

In the embodiment of the present invention, as shown in fig. 2, the remote control device 100 photographs an image including the outline of the screen through the binocular camera and transmits the image to the algorithm module. The algorithm module determines the position coordinates of the target position of the remote control device facing the screen according to the image, and sends the position coordinates to the sending module. Specifically, the algorithm module calibrates the binocular camera in an offline state, corrects two images of the screen to remove distortion caused by the binocular camera, calculates matching points between the corrected two images through binocular matching to obtain a parallax map, and calculates the position coordinate of the remote control device 100 facing the screen through 3D coordinates according to the parallax map. The sending module generates first information according to the position coordinates.

As another alternative, the remote control device comprises a binocular camera, the first information comprising capturing an image of the screen by the binocular camera to obtain two images comprising the outline of the screen, step 202 comprising capturing the image by the binocular camera when the remote control device is pointed at the screen.

In the embodiment of the present invention, as shown in fig. 1, the remote control device 100 photographs an image including the outline of a screen through a binocular camera, and sends the image to a sending module, and the sending module generates first information according to the image.

Step 204, the first information is sent to the controlled device, so that the controlled device determines a target position according to the first information, and performs a first operation on the screen according to the target position.

As shown in fig. 1, the first information includes an image including a contour of a screen captured by a binocular camera of the remote control apparatus 100. The transmitting module of the remote control device 100 transmits the first information to the receiving module of the controlled device 200. The receiving module receives the first information and sends the first information to the algorithm module. The algorithm module determines the position coordinates of the target position of the remote control device 100 facing the screen according to the first information, and transmits the position coordinates to the screen. Specifically, the algorithm module calibrates the binocular camera in an offline state, corrects two images of the screen to remove distortion caused by the binocular camera, calculates matching points between the corrected two images through binocular matching to obtain a parallax map, and calculates the position coordinate of the remote control device 100 facing the screen through 3D coordinates according to the parallax map. The screen determines a target position on the screen corresponding to the position coordinates according to the position coordinates, and performs a first operation on the screen according to the target position.

As shown in fig. 2, the first information includes position coordinates of a target position where the remote control device 100 is facing the screen. The transmitting module of the remote control device 100 transmits the first information to the receiving module of the controlled device 200. The receiving module receives the first information and sends the first information to the screen. The screen determines a target position on the screen corresponding to the position coordinates according to the position coordinates, and performs a first operation on the screen according to the target position.

The first operation comprises displaying a pointer at the target position, or highlighting or shading a control corresponding to the target position.

When the user moves the remote control device, the image of the screen acquired by the remote control device changes, the position coordinates need to be recalculated, the controlled device refreshes the pointer position according to the new position coordinates, the pointer moves according to the pointing direction of the remote control device, or the controlled device refreshes the highlighted or shadow-displayed control according to the new position coordinates.

In the technical scheme of the remote control method provided by the embodiment of the invention, the remote control equipment sends the first information to the controlled equipment, so that the controlled equipment determines the target position of the remote control equipment, which is opposite to the screen, according to the first information, and executes the first operation on the screen according to the target position. According to the embodiment of the invention, the remote operation of the controlled equipment can be realized on the premise that the user does not need to perform mechanical frequent operation on the five-way navigation key, and the experience of the user in the use process is improved.

The remote control method provided by the embodiment of the present invention is described in detail above with reference to fig. 1 to 9, and the device embodiment of the present invention will be described in detail below with reference to fig. 10. It should be understood that, the electronic device in the embodiment of the present invention may perform the methods of the foregoing embodiments of the present invention, that is, specific working processes of the following various products may refer to corresponding processes in the foregoing method embodiments.

The embodiment of the invention provides electronic equipment which can be terminal equipment or circuit equipment built in the terminal equipment. The electronic device may be adapted to perform the functions/steps of the method embodiments described above.

Fig. 10 is a schematic structural diagram of an electronic device 300 according to an embodiment of the present invention. The electronic device 300 may include a processor 310, an external memory interface 320, an internal memory 321, a universal serial bus (universal serial bus, USB) interface 330, a charge management module 340, a power management module 341, a battery 342, an antenna 1, an antenna 2, a mobile communication module 350, a wireless communication module 360, an audio module 370, a speaker 370A, a receiver 370B, a microphone 370C, an ear-piece interface 370D, a sensor module 380, keys 390, a motor 391, an indicator 392, a camera 393, a display 394, and a user identification module (subscriber identification module, SIM) card interface 395, among others. The sensor module 380 may include a pressure sensor 380A, a gyroscope sensor 380B, an air pressure sensor 380C, a magnetic sensor 380D, an acceleration sensor 380E, a distance sensor 380F, a proximity sensor 380G, a fingerprint sensor 380H, a temperature sensor 380J, a touch sensor 380K, an ambient light sensor 380L, a bone conduction sensor 380M, and the like.

It should be understood that the illustrated structure of the embodiment of the present invention does not constitute a specific limitation on the electronic device 300. In other embodiments of the invention, electronic device 300 may include more or less components than illustrated, or certain components may be combined, or certain components may be split, or different arrangements of components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

The processor 310 may include one or more processing units, for example, the processor 310 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (IMAGE SIGNAL processor, ISP), a controller, a video codec, a digital signal processor (DIGITAL SIGNAL processor, DSP), a baseband processor, and/or a neural network processor (neural-network processing unit, NPU), etc. Wherein the different processing units may be separate devices or may be integrated in one or more processors.

The controller can generate operation control signals according to the instruction operation codes and the time sequence signals to finish the control of instruction fetching and instruction execution.

A memory may also be provided in the processor 310 for storing instructions and data. In some embodiments, the memory in the processor 310 is a cache memory. The memory may hold instructions or data that the processor 310 has just used or recycled. If the processor 310 needs to reuse the instruction or data, it may be called directly from the memory. Repeated accesses are avoided and the latency of the processor 310 is reduced, thereby improving the efficiency of the system.

In some embodiments, processor 310 may include one or more interfaces. The interfaces may include an integrated circuit (inter-INTEGRATED CIRCUIT, I2C) interface, an integrated circuit built-in audio (inter-INTEGRATED CIRCUIT SOUND, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, a universal asynchronous receiver transmitter (universal asynchronous receiver/transmitter, UART) interface, a mobile industry processor interface (mobile industry processor interface, MIPI), a general-purpose input/output (GPIO) interface, a subscriber identity module (subscriber identity module, SIM) interface, and/or a universal serial bus (universal serial bus, USB) interface, among others.

The I2C interface is a bi-directional synchronous serial bus comprising a serial data line (SERIAL DATA LINE, SDA) and a serial clock line (derail clock line, SCL). In some embodiments, the processor 310 may contain multiple sets of I2C buses. The processor 310 may be coupled to the touch sensor 380K, charger, flash, camera 393, etc., respectively, via different I2C bus interfaces. For example, the processor 310 may couple the touch sensor 380K through an I2C interface, causing the processor 310 to communicate with the touch sensor 380K through an I2C bus interface, implementing the touch functionality of the electronic device 300.

The I2S interface may be used for audio communication. In some embodiments, the processor 310 may contain multiple sets of I2S buses. The processor 310 may be coupled to the audio module 370 via an I2S bus to enable communication between the processor 310 and the audio module 370. In some embodiments, the audio module 370 may communicate audio signals to the wireless communication module 360 via the I2S interface to enable answering calls via the bluetooth headset.

PCM interfaces may also be used for audio communication to sample, quantize and encode analog signals. In some embodiments, the audio module 370 and the wireless communication module 360 may be coupled by a PCM bus interface. In some embodiments, the audio module 370 may also transmit audio signals to the wireless communication module 360 via the PCM interface to enable phone answering via the bluetooth headset. Both the I2S interface and the PCM interface may be used for audio communication.

The UART interface is a universal serial data bus for asynchronous communications. The bus may be a bi-directional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, a UART interface is typically used to connect the processor 310 with the wireless communication module 360. For example, the processor 310 communicates with a bluetooth module in the wireless communication module 360 through a UART interface to implement bluetooth functions. In some embodiments, the audio module 370 may transmit audio signals to the wireless communication module 360 through a UART interface to implement a function of playing music through a bluetooth headset.

The MIPI interface may be used to connect the processor 310 to peripheral devices such as the display screen 394, the camera 393, and the like. The MIPI interfaces include camera serial interfaces (CAMERA SERIAL INTERFACE, CSI), display serial interfaces (DISPLAY SERIAL INTERFACE, DSI), and the like. In some embodiments, processor 310 and camera 393 communicate through a CSI interface, implementing the photographing function of electronic device 300. The processor 310 and the display screen 394 communicate via a DSI interface to implement the display functions of the electronic device 300.

The GPIO interface may be configured by software. The GPIO interface may be configured as a control signal or as a data signal. In some embodiments, a GPIO interface may be used to connect processor 310 with camera 393, display 394, wireless communication module 360, audio module 370, sensor module 380, and the like. The GPIO interface may also be configured as an I2C interface, an I2S interface, a UART interface, an MIPI interface, etc.

The USB interface 330 is an interface conforming to the USB standard specification, and may specifically be a Mini USB interface, a Micro USB interface, a USB Type C interface, or the like. The USB interface 330 may be used to connect a charger to charge the electronic device 300, or may be used to transfer data between the electronic device 300 and a peripheral device. And can also be used for connecting with a headset, and playing audio through the headset. The interface may also be used to connect other electronic devices, such as AR devices, etc.

It should be understood that the interfacing relationship between the modules illustrated in the embodiments of the present invention is only illustrative, and is not meant to limit the structure of the electronic device 300. In other embodiments of the present invention, the electronic device 300 may also employ different interfacing manners in the above embodiments, or a combination of multiple interfacing manners.

The charge management module 340 is configured to receive a charge input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charge management module 340 may receive a charging input of a wired charger through the USB interface 330. In some wireless charging embodiments, the charge management module 340 may receive wireless charging input through a wireless charging coil of the electronic device 300. The battery 342 is charged by the charge management module 340, and the electronic device may be powered by the power management module 341.

The power management module 341 is configured to connect the battery 342, the charge management module 340 and the processor 310. The power management module 341 receives input from the battery 342 and/or the charge management module 340 to power the processor 310, the internal memory 321, the display screen 394, the camera 393, the wireless communication module 360, and the like. The power management module 341 may also be configured to monitor battery capacity, battery cycle number, battery health (leakage, impedance), and other parameters. In other embodiments, the power management module 341 may also be disposed in the processor 310. In other embodiments, the power management module 341 and the charging management module 340 may also be disposed in the same device.

The wireless communication function of the electronic device 300 may be implemented by the antenna 1, the antenna 2, the mobile communication module 350, the wireless communication module 360, a modem processor, a baseband processor, and the like.

The antennas 1 and 2 are used for transmitting and receiving electromagnetic wave signals. Each antenna in the electronic device 300 may be used to cover a single or multiple communication bands. Different antennas may also be multiplexed to improve the utilization of the antennas. For example, the antenna 1 may be multiplexed into a diversity antenna of a wireless local area network. In other embodiments, the antenna may be used in conjunction with a tuning switch.

The mobile communication module 350 may provide a solution for wireless communication, including 2G/3G/4G/5G, etc., applied on the electronic device 300. The mobile communication module 350 may include at least one filter, switch, power amplifier, low noise amplifier (low noise amplifier, LNA), or the like. The mobile communication module 350 may receive electromagnetic waves from the antenna 1, perform processes such as filtering, amplifying, and the like on the received electromagnetic waves, and transmit the processed electromagnetic waves to the modem processor for demodulation. The mobile communication module 350 may amplify the signal modulated by the modem processor, and convert the signal into electromagnetic waves through the antenna 1 to radiate the electromagnetic waves. In some embodiments, at least some of the functional modules of the mobile communication module 350 may be disposed in the processor 310. In some embodiments, at least some of the functional modules of the mobile communication module 350 may be provided in the same device as at least some of the modules of the processor 310.

The modem processor may include a modulator and a demodulator. The modulator is used for modulating the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used for demodulating the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low frequency baseband signal to the baseband processor for processing. The low frequency baseband signal is processed by the baseband processor and then transferred to the application processor. The application processor outputs sound signals through an audio device (not limited to speaker 370A, receiver 370B, etc.), or displays images or video through display screen 394. In some embodiments, the modem processor may be a stand-alone device. In other embodiments, the modem processor may be provided in the same device as the mobile communication module 350 or other functional module, independent of the processor 310.

The wireless communication module 360 may provide solutions for wireless communication including wireless local area network (wireless local area networks, WLAN) (e.g., wireless fidelity (WIRELESS FIDELITY, wi-Fi) network), bluetooth (BT), global navigation satellite system (global navigation SATELLITE SYSTEM, GNSS), frequency modulation (frequency modulation, FM), near field communication (NEAR FIELD communication, NFC), infrared (IR), etc., as applied to the electronic device 300.

The wireless communication module 360 may be one or more devices that integrate at least one communication processing module. The wireless communication module 360 receives electromagnetic waves via the antenna 2, modulates the electromagnetic wave signals, filters the electromagnetic wave signals, and transmits the processed signals to the processor 310.

The wireless communication module 360 may also receive a signal to be transmitted from the processor 310, frequency modulate it, amplify it, and convert it to electromagnetic waves for radiation via the antenna 2.

In some embodiments, antenna 1 and mobile communication module 350 of electronic device 300 are coupled, and antenna 2 and wireless communication module 360 are coupled, such that electronic device 300 may communicate with a network and other devices via wireless communication techniques. The wireless communication techniques can include the Global System for Mobile communications (global system for mobile communications, GSM), general packet radio service (GENERAL PACKET radio service, GPRS), code division multiple access (code division multiple access, CDMA), wideband code division multiple access (wideband code division multiple access, WCDMA), time division code division multiple access (time-division code division multiple access, TD-SCDMA), long term evolution (long term evolution, LTE), BT, GNSS, WLAN, NFC, FM, and/or IR techniques, among others. The GNSS may include a global satellite positioning system (global positioning system, GPS), a global navigation satellite system (global navigation SATELLITE SYSTEM, GLONASS), a beidou satellite navigation system (beidou navigation SATELLITE SYSTEM, BDS), a quasi zenith satellite system (quasi-zenith SATELLITE SYSTEM, QZSS) and/or a satellite based augmentation system (SATELLITE BASED AUGMENTATION SYSTEMS, SBAS).

The electronic device 300 implements display functions through a GPU, a display screen 394, an application processor, and the like. The GPU is a microprocessor for image processing, connected to the display screen 394 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 310 may include one or more GPUs that execute program instructions to generate or change display information.

The display screen 394 is used for displaying images, videos, and the like. The display screen 394 includes a display panel. The display panel may employ a Liquid Crystal Display (LCD) CRYSTAL DISPLAY, an organic light-emitting diode (OLED), an active-matrix organic LIGHT EMITTING diode (AMOLED), a flexible light-emitting diode (FLED), miniled, microLed, micro-oLed, a quantum dot LIGHT EMITTING diode (QLED), or the like. In some embodiments, the electronic device 300 may include 1 or N display screens 394, N being a positive integer greater than 1.

Electronic device 300 may implement capture functionality through an ISP, camera 393, video codec, GPU, display 394, and application processor, among others.

The ISP is used to process the data fed back by camera 393. For example, when photographing, the shutter is opened, light is transmitted to the camera photosensitive element through the lens, the optical signal is converted into an electric signal, and the camera photosensitive element transmits the electric signal to the ISP for processing and is converted into an image visible to naked eyes. ISP can also optimize the noise, brightness and skin color of the image. The ISP can also optimize parameters such as exposure, color temperature and the like of a shooting scene. In some embodiments, the ISP may be provided in the camera 393.

Camera 393 is used to capture still images or video. The object generates an optical image through the lens and projects the optical image onto the photosensitive element. The photosensitive element may be a charge coupled device (charge coupled device, CCD) or a Complementary Metal Oxide Semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, which is then transferred to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV, or the like format. In some embodiments, electronic device 300 may include 1 or N cameras 393, N being a positive integer greater than 1.

The digital signal processor is used for processing digital signals, and can process other digital signals besides digital image signals. For example, when the electronic device 300 is selecting a frequency bin, the digital signal processor is used to fourier transform the frequency bin energy, or the like.

Video codecs are used to compress or decompress digital video. The electronic device 300 may support one or more video codecs. Thus, the electronic device 300 may play or record video in a variety of encoding formats, such as moving picture experts group (moving picture experts group, MPEG) 1, MPEG2, MPEG3, MPEG4, and the like.

The NPU is a neural-network (NN) computing processor, and can rapidly process input information by referencing a biological neural network structure, for example, referencing a transmission mode between human brain neurons, and can also continuously perform self-learning. Applications such as intelligent recognition of the electronic device 300, for example, image recognition, face recognition, voice recognition, text understanding, etc., can be realized through the NPU.

The external memory interface 320 may be used to connect an external memory card, such as a Micro SD card, to enable expansion of the memory capabilities of the electronic device 300. The external memory card communicates with the processor 310 through an external memory interface 320 to implement data storage functions. For example, files such as music, video, etc. are stored in an external memory card.

The internal memory 321 may be used to store computer executable program code comprising instructions. The internal memory 321 may include a storage program area and a storage data area. The storage program area may store an application program (such as a sound playing function, an image playing function, etc.) required for at least one function of the operating system, etc. The storage data area may store data created during use of the electronic device 300 (e.g., audio data, phonebook, etc.), and so on. In addition, the internal memory 321 may include a high-speed random access memory, and may also include a nonvolatile memory, such as at least one magnetic disk storage device, a flash memory device, a universal flash memory (universal flash storage, UFS), and the like. The processor 310 performs various functional applications of the electronic device 300 and data processing by executing instructions stored in the internal memory 321, and/or instructions stored in a memory provided in the processor.

The electronic device 300 may implement audio functionality through an audio module 370, a speaker 370A, a receiver 370B, a microphone 370C, an ear-headphone interface 370D, and an application processor, among others. Such as music playing, recording, etc.

The audio module 370 is used to convert digital audio information into an analog audio signal output and also to convert an analog audio input into a digital audio signal. The audio module 370 may also be used to encode and decode audio signals. In some embodiments, the audio module 370 may be disposed in the processor 310, or some of the functional modules of the audio module 370 may be disposed in the processor 310.

Speaker 370A, also known as a "horn," is used to convert audio electrical signals into sound signals. The electronic device 300 may listen to music, or to hands-free conversations, through the speaker 370A.

A receiver 370B, also referred to as a "earpiece", is used to convert the audio electrical signal into a sound signal. When electronic device 300 is answering a telephone call or voice message, voice may be received by placing receiver 370B close to the human ear.

Microphone 370C, also referred to as a "microphone," is used to convert sound signals into electrical signals. When making a call or transmitting voice information, the user can sound near the microphone 370C through the mouth, inputting a sound signal to the microphone 370C. The electronic device 300 may be provided with at least one microphone 370C. In other embodiments, the electronic device 300 may be provided with two microphones 370C, and may implement a noise reduction function in addition to collecting sound signals. In other embodiments, the electronic device 300 may also be provided with three, four, or more microphones 370C to enable collection of sound signals, noise reduction, identification of sound sources, directional recording functions, etc.

The earphone interface 370D is for connecting a wired earphone. The headset interface 370D may be a USB interface 330 or a 3.5mm open mobile electronic device platform (open mobile terminal platform, OMTP) standard interface, a american cellular telecommunications industry association (cellular telecommunications industry association of the USA, CTIA) standard interface.

The pressure sensor 380A is configured to sense a pressure signal and convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 380A may be disposed on the display screen 394.

The pressure sensor 380A is of various types, such as a resistive pressure sensor, an inductive pressure sensor, a capacitive pressure sensor, and the like. The capacitive pressure sensor may be a capacitive pressure sensor comprising at least two parallel plates with conductive material. When a force is applied to the pressure sensor 380A, the capacitance between the electrodes changes. The electronic device 300 determines the strength of the pressure from the change in capacitance. When a touch operation is applied to the display screen 394, the electronic apparatus 300 detects the touch operation intensity according to the pressure sensor 380A. The electronic device 300 may also calculate the location of the touch based on the detection signal of the pressure sensor 380A. In some embodiments, touch operations that act on the same touch location, but at different touch operation strengths, may correspond to different operation instructions. For example, when a touch operation with a touch operation intensity smaller than a first pressure threshold is applied to the short message application icon, an instruction to view the short message is executed. And executing an instruction for newly creating the short message when the touch operation with the touch operation intensity being greater than or equal to the first pressure threshold acts on the short message application icon.

The gyro sensor 380B may be used to determine a motion gesture of the electronic device 300. In some embodiments, the angular velocity of electronic device 300 about three axes (i.e., x, y, and z axes) may be determined by gyro sensor 380B. The gyro sensor 380B may be used for photographing anti-shake. For example, when the shutter is pressed, the gyro sensor 380B detects the shake angle of the electronic device 300, calculates the distance to be compensated by the lens module according to the angle, and makes the lens counteract the shake of the electronic device 300 through the reverse motion, so as to realize anti-shake. The gyro sensor 380B may also be used for navigating, somatosensory game scenes.

The air pressure sensor 380C is used to measure air pressure. In some embodiments, the electronic device 300 calculates altitude from barometric pressure values measured by the barometric pressure sensor 380C, aiding in positioning and navigation.

The magnetic sensor 380D includes a hall sensor. The electronic device 300 may detect the opening and closing of the flip holster using the magnetic sensor 380D. In some embodiments, when the electronic device 300 is a flip machine, the electronic device 300 may detect the opening and closing of the flip according to the magnetic sensor 380D. And then according to the detected opening and closing state of the leather sheath or the opening and closing state of the flip, the characteristics of automatic unlocking of the flip and the like are set.

The acceleration sensor 380E may detect the magnitude of acceleration of the electronic device 300 in various directions (typically three axes). The magnitude and direction of gravity may be detected when the electronic device 300 is stationary. The electronic equipment gesture recognition method can also be used for recognizing the gesture of the electronic equipment, and is applied to horizontal and vertical screen switching, pedometers and other applications.

A distance sensor 380F for measuring distance. The electronic device 300 may measure the distance by infrared or laser. In some embodiments, the electronic device 300 may range using the distance sensor 380F to achieve fast focus.

The proximity light sensor 380G may include, for example, a Light Emitting Diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The electronic device 300 emits infrared light outward through the light emitting diode. The electronic device 300 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it may be determined that an object is in the vicinity of the electronic device 300. When insufficient reflected light is detected, the electronic device 300 may determine that there is no object in the vicinity of the electronic device 300. The electronic device 300 can detect that the user holds the electronic device 300 close to the ear by using the proximity light sensor 380G, so as to automatically extinguish the screen to achieve the purpose of saving power. The proximity light sensor 380G may also be used in holster mode, pocket mode to automatically unlock and lock the screen.

The ambient light sensor 380L is used to sense ambient light level. The electronic device 300 may adaptively adjust the brightness of the display screen 394 based on the perceived ambient light level. The ambient light sensor 380L may also be used to automatically adjust white balance during photographing. The ambient light sensor 380L may also cooperate with the proximity light sensor 380G to detect if the electronic device 300 is in a pocket to prevent false touches.

The fingerprint sensor 380H is used to collect a fingerprint. The electronic device 300 can utilize the collected fingerprint characteristics to realize fingerprint unlocking, access an application lock, fingerprint photographing, fingerprint incoming call answering and the like.

The temperature sensor 380J is used to detect temperature. In some embodiments, the electronic device 300 performs a temperature processing strategy using the temperature detected by the temperature sensor 380J. For example, when the temperature reported by temperature sensor 380J exceeds a threshold, electronic device 300 performs a reduction in performance of a processor located in the vicinity of temperature sensor 380J in order to reduce power consumption to implement thermal protection. In other embodiments, when the temperature is below another threshold, the electronic device 300 heats the battery 342 to avoid the low temperature causing the electronic device 300 to shut down abnormally. In other embodiments, when the temperature is below a further threshold, the electronic device 300 performs boosting of the output voltage of the battery 342 to avoid abnormal shutdown caused by low temperatures.

Touch sensor 380K, also known as a "touch device". The touch sensor 380K may be disposed on the display screen 394, and the touch sensor 380K and the display screen 394 form a touch screen, which is also referred to as a "touch screen". The touch sensor 380K is for detecting a touch operation acting thereon or thereabout. The touch sensor may communicate the detected touch operation to the application processor to determine the touch event type. Visual output related to touch operations may be provided through the display screen 394. In other embodiments, touch sensor 380K may also be located on a surface of electronic device 300 other than at display 394.

The bone conduction sensor 380M may acquire a vibration signal. In some embodiments, bone conduction sensor 380M may acquire a vibration signal of a human vocal tract vibrating bone pieces. The bone conduction sensor 380M may also contact the pulse of the human body to receive the blood pressure pulsation signal. In some embodiments, bone conduction sensor 380M may also be provided in the headset, in combination with an osteoinductive headset. The audio module 370 may analyze the voice signal based on the vibration signal of the sound portion vibration bone block obtained by the bone conduction sensor 380M, so as to implement a voice function. The application processor can analyze heart rate information based on the blood pressure beat signals acquired by the bone conduction sensor 380M, so as to realize a heart rate detection function.

The keys 390 include a power on key, a volume key, etc. Key 390 may be a mechanical key. Or may be a touch key. The electronic device 300 may receive key inputs, generating key signal inputs related to user settings and function controls of the electronic device 300.

The motor 391 may generate a vibration alert. The motor 391 may be used for incoming call vibration alerting as well as for touch vibration feedback. For example, touch operations acting on different applications (e.g., photographing, audio playing, etc.) may correspond to different vibration feedback effects. The motor 391 may also correspond to different vibration feedback effects by touch operations applied to different areas of the display screen 394. Different application scenarios (such as time reminding, receiving information, alarm clock, game, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect may also support customization.

The indicator 392 may be an indicator light, which may be used to indicate a state of charge, a change in charge, a message indicating a missed call, a notification, etc.

The SIM card interface 395 is for interfacing with a SIM card. The SIM card may be inserted into the SIM card interface 395 or removed from the SIM card interface 395 to enable contact and separation with the electronic device 300. The electronic device 300 may support 1 or N SIM card interfaces, N being a positive integer greater than 1. The SIM card interface 395 may support Nano SIM cards, micro SIM cards, and the like. The same SIM card interface 395 can be used to insert multiple cards simultaneously. The types of the plurality of cards may be the same or different. The SIM card interface 395 may also be compatible with different types of SIM cards. The SIM card interface 395 may also be compatible with external memory cards. The electronic device 300 interacts with the network through the SIM card to realize functions such as communication and data communication. In some embodiments, the electronic device 300 employs an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 300 and cannot be separated from the electronic device 300.

Embodiments of the present invention provide a computer readable storage medium having instructions stored therein which, when executed on a terminal device, cause the terminal device to perform the functions/steps as in the method embodiments described above.

Embodiments of the present invention also provide a computer program product comprising instructions which, when run on a computer or any of the at least one processor, cause the computer to perform the functions/steps as in the method embodiments described above.

In the embodiments of the present invention, "at least one" means one or more, and "a plurality" means two or more. "and/or", describes an association relation of association objects, and indicates that there may be three kinds of relations, for example, a and/or B, and may indicate that a alone exists, a and B together, and B alone exists. Wherein A, B may be singular or plural. The character "/" generally indicates that the context-dependent object is an "or" relationship. "at least one of the following" and the like means any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c may represent a, b, c, a-b, a-c, b-c, or a-b-c, wherein a, b, c may be single or plural.

Those of ordinary skill in the art will appreciate that the various elements and algorithm steps described in the embodiments disclosed herein can be implemented as a combination of electronic hardware, computer software, and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.

It will be clear to those skilled in the art that, for convenience and brevity of description, specific working procedures of the above-described systems, apparatuses and units may refer to corresponding procedures in the foregoing method embodiments, and are not repeated herein.

In several embodiments provided by the present invention, any of the functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution in the form of a software product stored in a storage medium, comprising several instructions for causing an electronic device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. The storage medium includes a U disk, a removable hard disk, a read-only memory (ROM), a random access memory (random access memory, RAM), a magnetic disk, an optical disk, or other various media capable of storing program codes.

The foregoing is merely exemplary embodiments of the present invention, and any person skilled in the art may easily conceive of changes or substitutions within the technical scope of the present invention, which should be covered by the present invention. The protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A remote control method, characterized by being applied to a controlled device including a screen, the method comprising:

The method comprises the steps of receiving first information of remote control equipment, wherein the first information comprises position coordinates of a target position, opposite to a screen, of the remote control equipment, the position coordinates are coordinates obtained by the remote control equipment according to a first contour image and a second contour image of the screen, the first contour image and the second contour image are contour images of the screen shot by a binocular camera on the remote control equipment, and the coordinates obtained by the remote control equipment according to the first contour image and the second contour image of the screen comprise the steps that the remote control equipment corrects the binocular camera in an off-line state and then removes distortion caused by the binocular camera through correcting the first contour image and the second contour image, obtains a parallax map through matching points between the first contour image and the second contour image after binocular matching calculation and obtains the position coordinates through 3D coordinate calculation according to the parallax map;

determining the target position of the remote control equipment, which is opposite to the screen, according to the first information;

And executing a first operation on the screen according to the target position.

2. The method of claim 1, wherein determining from the first information that the remote control device is facing the target location of the screen comprises:

3. The method of claim 1 or 2, wherein the first operation comprises displaying a pointer at the target location, or highlighting or shading a control corresponding to the target location.

4. A remote control method, characterized by being applied to a remote control device, the method comprising:

calibrating a binocular camera on the remote control equipment in an off-line state;

when the remote control device points to a screen of the controlled device, a first contour image and a second contour image containing the screen are shot through the binocular camera;

Removing distortion caused by the binocular camera by correcting the first contour image and the second contour image;

obtaining a parallax image through binocular matching calculation of matching points between the corrected first contour image and the corrected second contour image;

calculating a position coordinate of a target position of the remote control equipment, which is opposite to the screen, through a 3D coordinate according to the parallax map;

And sending first information to the controlled device, wherein the first information comprises the position coordinates, so that the controlled device determines the target position according to the first information and performs a first operation on the screen according to the target position.

5. The method of claim 4, wherein the first operation comprises displaying a pointer at the target location or highlighting or shading a control corresponding to the target location.

6. A remote control apparatus, characterized by being applied to a controlled device including a screen, the apparatus comprising:

The system comprises a receiving module, a remote control device, a receiving module and a display module, wherein the first information comprises position coordinates of a target position of the remote control device, the target position is opposite to the screen, the position coordinates are coordinates obtained by the remote control device according to a first contour image and a second contour image of the screen, the first contour image and the second contour image are contour images of the screen shot by a binocular camera on the remote control device, the remote control device obtains coordinates according to the first contour image and the second contour image of the screen, the coordinates obtained by the remote control device according to the first contour image and the second contour image of the screen comprise that the remote control device corrects the binocular camera in an off-line state to remove distortion caused by the binocular camera and obtains a parallax map through matching points between the first contour image and the second contour image after the correction through binocular matching calculation, and obtains the position coordinates through 3D coordinate calculation according to the parallax map;

7. The apparatus according to claim 6, wherein the determining module is specifically configured to determine the target position on the screen corresponding to the position coordinate according to the position coordinate.

8. The apparatus of claim 6 or 7, wherein the first operation comprises displaying a pointer at the target location or highlighting or shading a control corresponding to the target location.

9. A remote control apparatus for use with a remote control device, the apparatus comprising:

The system comprises an acquisition module, a control module and a control module, wherein the acquisition module is used for acquiring first information, the first information is used for determining a target position of a screen of a controlled device, the remote control device is opposite to the controlled device, the first information comprises position coordinates of the target position, the position coordinates are obtained by calibrating a binocular camera on the remote control device in an offline state, shooting a first contour image and a second contour image which comprise the screen through the binocular camera when the remote control device points to the screen, correcting the first contour image and the second contour image to remove distortion caused by the binocular camera, calculating a matching point between the corrected first contour image and the corrected second contour image through binocular matching, and calculating a parallax image through 3D coordinates according to the parallax image;

10. The apparatus of claim 9, wherein the first operation comprises displaying a pointer at the target location or highlighting or shading a control corresponding to the target location.

11. A controlled device comprising a screen, a processor and a memory, wherein the memory is for storing a computer program comprising program instructions which, when executed by the processor, cause the controlled device to perform the steps of:

And executing a first operation on the screen according to the target position.

12. The apparatus of claim 11, wherein the program instructions, when executed by the processor, cause the controlled apparatus to perform the steps of:

Determining the target position of the remote control device facing the screen according to the first information comprises the following steps:

13. The apparatus of claim 11 or 12, wherein the first operation comprises displaying a pointer at the target location or highlighting or shading a control corresponding to the target location.

14. A remote control device comprising a processor and a memory, wherein the memory is for storing a computer program comprising program instructions which, when executed by the processor, cause the remote control device to perform the steps of:

15. The apparatus of claim 14, wherein the first operation comprises displaying a pointer at the target location or highlighting or shading a control corresponding to the target location.