CN114237787B - A cloud desktop image transmission method and device - Google Patents
A cloud desktop image transmission method and device Download PDFInfo
- Publication number
- CN114237787B CN114237787B CN202111372689.7A CN202111372689A CN114237787B CN 114237787 B CN114237787 B CN 114237787B CN 202111372689 A CN202111372689 A CN 202111372689A CN 114237787 B CN114237787 B CN 114237787B
- Authority
- CN
- China
- Prior art keywords
- block
- level
- cache
- primary
- desktop
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/451—Execution arrangements for user interfaces
- G06F9/452—Remote windowing, e.g. X-Window System, desktop virtualisation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention provides a cloud desktop image transmission method and device. The method is used for solving the technical problem of low transmission efficiency of the text cloud desktop image, the text desktop image is subjected to two-stage block division, the hit judgment of two-stage blocks is carried out on the desktop image to be transmitted, and the image data is replaced by the block cache identification of the hit block. The segmentation of the two-level blocks adopts the transverse block segmentation to distinguish the text blocks from the non-text blocks, so that the character interface characteristics are fully utilized, the bandwidth occupation of text desktop scenes in a cloud desktop transmission protocol is reduced, and the smoothness of the cloud desktop in a weak network environment is improved.
Description
Technical Field
The invention relates to the technical field of communication and cloud computing, in particular to a cloud desktop image transmission method and device.
Background
In actual deployment, a cloud desktop transmission protocol client (abbreviated as a cloud desktop client) is generally installed on a thin terminal in the current virtual desktop infrastructure (Virtual Desktop Infrastructure, VDI) protocol, the cloud desktop client and a cloud desktop server interact desktop data through a remote desktop protocol, the cloud desktop server compresses and transmits cloud desktop images to the cloud desktop client, and the cloud desktop client is decompressed and then presented to a user through a local display system.
The cloud desktop data transmission efficiency is always the focus of the attention of developers and users, on one hand, because the bandwidth is a resource of the users, the cost investment is related, the occupied bandwidth is high when a single cloud desktop is connected, the total bandwidth is too high when all people use the cloud desktop at the same time, and the cost of network equipment investment and operator bandwidth lease when the users build the cloud desktop environment is affected. Another reason is that because bandwidth is high, it can affect the user experience in situations where link conditions are not good, such as wide area network remote connections.
At present, in the VDI infrastructure, when the desktop is refreshed, the virtual machine of the server side can reduce the transmitted data volume by adopting an interface cache de-duplication mode, and only the image of the desktop change area is transmitted to the cloud desktop client side. However, in a scenario that the cloud desktop mainly displays a text interface, a lossless compression mode is generally required to be adopted by a server side, and desktop data transmission efficiency of the cloud desktop is not high.
Disclosure of Invention
In view of the above, the invention provides a method, a device and a storage medium for transmitting text images on a cloud desktop, which are used for solving the technical problem of low text cloud desktop image data transmission efficiency.
Based on an aspect of the embodiment of the invention, a cloud desktop image transmission method is provided, and the method is applied to a cloud desktop server and comprises the following steps:
judging whether the desktop image to be transmitted currently is a text interface or not;
when the interface is judged to be a text interface, acquiring all primary blocks intersected with a desktop refreshing area to form a primary refreshing block set, wherein the primary blocks are screen display areas obtained by segmenting a cloud desktop based on a preset primary block size;
scanning and cutting the secondary blocks are respectively carried out on each primary block in the primary refreshing block set, and the hash value of each secondary block is calculated;
calculating hash values of all primary blocks of the desktop image to be transmitted currently;
the method comprises the steps of carrying out block cache identification replacement of blocks of hit first-level blocks for first-level blocks in a first-level non-refreshing block set;
and packaging and compressing the desktop image data to be transmitted currently after finishing the block cache identification replacement processing, and then sending the desktop image data to the cloud desktop client.
Further, the method for judging whether the desktop image to be transmitted is a text interface comprises the following steps:
Judging whether the desktop image is a literal interface by comparing the change rate of the color change of the desktop image, judging the desktop image as the literal interface when the change rate is larger than a preset change rate threshold value, otherwise judging the desktop image as a non-literal interface, or,
And judging whether the input desktop image is a text interface or not by adopting the trained deep learning neural network model.
Further, the method for scanning and splitting the secondary block for each primary block in the primary refresh block set respectively comprises the following steps:
And carrying out pixel-level line scanning on the image data of the first-level block, and dividing the area containing the characters and the blank area into a plurality of second-level blocks.
Further, the method for calculating the hash value of the first-level block is as follows:
and combining the hash values of all the secondary blocks of the primary block, calculating the hash value of the combined data, and taking the hash value of the combined data as the hash value of the image data of the primary block.
Further, the method for replacing the block cache identifier of the block of the hit second-level block for each first-level block in the first-level refresh block set comprises the following steps:
matching in a second-level block cache list in a local cache pool by using the hash value of the second-level block, and judging whether hit occurs or not;
if the second-level block cache data structure in the second-level block cache list is hit, replacing the image data of the second-level block by the block cache identifier of the second-level block;
if the second-level block cache data structure in the second-level block cache list is not hit, recording the cache data structure of the second-level block in a local cache pool;
the secondary block cache data structure includes, but is not limited to, block start coordinates, block length and width, block cache identification, hash value of the block.
Further, the method further includes an aging cleaning step of the cache data structure including a primary block and a secondary block, including:
adding an aging time field for each primary block and each secondary block to record the duration of the cache;
Refreshing a cache time field of a block when the block is hit each time;
and periodically judging whether the buffer time length of each block exceeds a preset threshold through the aging cleaning timer, and releasing the buffer space of the corresponding block if the buffer time length exceeds the preset threshold.
Based on another aspect of the embodiment of the present invention, there is also provided a cloud desktop image transmission device, where the device is applied to a cloud desktop server, and includes:
the text interface judging module is used for judging whether the desktop image to be transmitted currently is a text interface or not;
The refreshing block acquisition module is used for acquiring all primary blocks intersected with the desktop refreshing area to form a primary refreshing block set when the text interface is judged, wherein the primary blocks are screen display areas obtained by dividing the cloud desktop based on the preset primary block size;
The second-level block segmentation module is used for respectively scanning and segmenting the second-level blocks for each first-level block in the first-level refreshing block set and calculating the hash value of each second-level block;
the primary block hash module is used for calculating hash values of all primary blocks of the desktop image to be transmitted currently;
The block cache identification replacing module is used for carrying out block cache identification replacement of the blocks of the hit first-level blocks for the first-level blocks in the first-level non-refreshed block set;
and the compression transmission module is used for packing and compressing the desktop image data to be transmitted currently after the block cache identification replacement processing is completed and then sending the data to the cloud desktop client.
Further, the text interface judging module judges whether the desktop image is a text interface or not by comparing the change rate of the color change of the desktop image, and judges the desktop image as a text interface when the change rate is larger than a preset change rate threshold value, or judges the desktop image as a non-text interface if not, or judges whether the input desktop image is a text interface by adopting a trained deep learning neural network model.
Further, the second-level block segmentation module performs pixel-level line scanning on the image data of the first-level block, and segments the text-containing region and the blank region into a plurality of second-level blocks.
Further, the apparatus further comprises:
The system comprises a first-level block, a second-level block, an aging processing module, a buffer time field and an aging cleaning timer, wherein the first-level block and the second-level block are used for storing the buffer time, the aging processing module is used for adding the aging time field for recording the buffer time of each first-level block and each second-level block, refreshing the buffer time field of each block when each block is hit, periodically judging whether the buffer time of each block exceeds a preset threshold or not through the aging cleaning timer, and releasing the buffer space of the corresponding block if the buffer time exceeds the preset threshold.
Based on another aspect of the present invention, there is also provided a cloud desktop image transmission device, which is applied to a cloud desktop client, including:
the decompression module is used for receiving the compressed data packet of the text interface sent by the cloud desktop server and analyzing the compressed data packet;
The first-level block processing module is used for directly reading the image data of the corresponding first-level block from the local cache through the block cache identifier carried in the data packet when the hit type identifier of the block in the data packet indicates that the first-level block is hit, and reading the image data of the first-level block from the data packet and caching the image data of the corresponding first-level block in the local cache pool when the hit type identifier of the block in the data packet indicates that the first-level block and the second-level block are not hit;
The system comprises a second-level block processing module, a client and a client, wherein the second-level block processing module is used for analyzing the data structure of a second-level block included in a data packet and sequentially reading the data structure of the second-level block when the hit type identification of the block in the data packet indicates that the second-level block is hit;
And the block combination module is used for splicing the image data of all the first-level blocks read from the compressed data packet to obtain complete desktop image data and outputting the complete desktop image data to a display driver of the client.
According to the invention, two-stage block division is carried out on the text desktop image, two-stage block hit judgment is carried out on the desktop image to be transmitted, and the block cache identification of the hit block is used for replacing image data. The segmentation of the two-level blocks adopts the transverse block segmentation to distinguish the text blocks from the non-text blocks, so that the character interface characteristics are fully utilized, the bandwidth occupation of text desktop scenes in a cloud desktop transmission protocol is reduced, and the smoothness of the cloud desktop in a weak network environment is improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following description will briefly describe the drawings required to be used in the embodiments of the present invention or the description in the prior art, and it is obvious that the drawings in the following description are only some embodiments described in the present invention, and other drawings may be obtained according to these drawings of the embodiments of the present invention for a person having ordinary skill in the art.
FIG. 1 is a schematic diagram of a text interface and desktop image block division for cloud desktop display according to an embodiment of the present invention;
FIG. 2 is a diagram illustrating a secondary block division based on a primary block of a text interface according to an embodiment of the present invention;
FIG. 3 is a flowchart illustrating steps of a method for transmitting a cloud desktop image according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of a server cloud desktop image transmission device according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of a client cloud desktop image transmission device according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the invention.
Detailed Description
The terminology used in the embodiments of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of embodiments of the invention. As used in this embodiment of the invention, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. The term "and/or" as used in this disclosure refers to any or all possible combinations comprising one or more of the associated listed items.
It should be understood that although the terms first, second, third, etc. may be used in embodiments of the present invention to describe various information, these information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, the first information may also be referred to as second information, and similarly, the second information may also be referred to as first information, without departing from the scope of embodiments of the present invention. Depending on the context, furthermore, the word "if" used may be interpreted as "at..once" or "when..once" or "in response to a determination".
In an office scenario, the cloud desktop may display a text interface in most of the time, for example, when a user views an edited document or mail using a word, excel, notepad, outlook, PDF, or other software, and when the user reads the article using a web page or reading software, most of the area presented in the desktop may be a line of text, a table, or the like.
In order to improve the data transmission efficiency of the cloud desktop, a cloud desktop block caching mechanism can be used, namely, a cloud desktop image is segmented according to a preset size, for example, the cloud desktop is segmented according to the size of pixel block areas such as 16×16,64×64, 128×128 or 256×256, the cloud desktop server packages the segmented cloud desktop image data to a cloud desktop client running on the terminal, the cloud desktop server and the client both Cache the sent desktop block image data, and then if a certain cached block image data is hit, the server only sends a block Cache identifier (Cache ID) of the block image data to the cloud desktop client, and the client directly obtains data from a local Cache through the block Cache identifier for filling and displaying.
The inventor finds through analysis that, although the cloud desktop image block data cache transmission mode improves the transmission efficiency of cloud desktop data to a certain extent, for a text interface, the block data cache mode does not fully utilize the characteristics of the text interface, so that the hit probability of block image data is not particularly high, and the transmission efficiency is not high.
Fig. 1 is a schematic diagram of a text interface and a desktop image partition displayed on a cloud desktop according to an embodiment of the present invention, in which a plurality of rows of text are displayed in a desktop image of the cloud desktop, the rows of text are left blank between each other, the whole desktop image is divided into N blocks, each block has a size of l×w, L is a length, W is a width, and a unit may be a pixel. For example, a256×256 pixel resolution desktop image may be divided into n=16, l=64 pixels, and w=64 pixels. As can be seen from the figure, there is a lot of white space between lines of text and around the document, and these white space areas are usually unchanged. If the large block is used for caching, as long as any character symbol or line position in the block is changed, the change of the block data is caused, the server needs to transmit the whole data to the cloud desktop client in a compressed mode, in fact, many blank areas are unchanged, the percentage of the character change area to the whole block area is possibly small, and if the whole block data is transmitted in a compressed mode, bandwidth waste is caused.
In order to improve desktop image transmission efficiency in a text desktop scene in a cloud desktop transmission protocol and increase smoothness when a cloud desktop displays a text interface in an environment with insufficient network bandwidth resources, the invention provides a cloud desktop image transmission method, which relates to processing steps of a cloud desktop server and two ends of a cloud desktop client, and the two ends are required to be coordinated and matched with each other to achieve the aim of the invention.
The basic idea of the invention is to identify the interface type aiming at the cloud desktop image and perform two-stage block division on the text interface. Firstly, carrying out first-level block division on the whole desktop image, carrying out second-level block division in the first-level block, adopting a non-fixed transverse or longitudinal (determined according to a character typesetting format) block division mode which accords with character typesetting characteristics, respectively locally establishing a two-level block cache list by a cloud desktop server and a client, directly transmitting a block cache identifier of the first-level block to the cloud desktop client when the first-level block does not change and needs repeated transmission, continuing judging the second-level block when the first-level block cache is missed, transmitting the block cache identifier of the second-level block to the cloud desktop client when the second-level block cache is hit, and directly reading a cache block corresponding to the block cache identifier from the local for display when the cloud desktop client receives the block cache identifier of the first-level block or the second-level block.
FIG. 2 is a diagram illustrating a secondary block division based on a primary block of a text interface according to an embodiment of the present invention. The size of the secondary block may be variable, for example, in fig. 2, the i-th primary block is divided into 4 secondary blocks according to the positions of the text and the blank area.
Fig. 3 is a flowchart illustrating steps of a cloud desktop image transmission method according to an embodiment of the present invention, where the illustrated example is a specific implementation process of transmitting a cloud desktop image by a server, including:
step 301, the cloud desktop server judges whether a desktop image to be transmitted currently is a text interface or not;
In the cloud desktop infrastructure, a server virtual display card driver can capture cloud desktop images, the cloud desktop images can be processed and sent frame by frame, and the cloud desktop server can select different transmission modes according to scene configuration, for example, in a streaming mode transmission in a very fast mode, and the cloud desktop images are processed by using a micro-frame algorithm. In the high-definition mode, a series of processing procedures such as de-duplication, text interface separation and compression are adopted, and then the processing procedures are sent to the cloud desktop client. In the high-definition mode, the desktop image can be divided into a text interface image and a non-text interface image for processing respectively, and different algorithms are adopted for processing to improve the transmission efficiency. Finally, after the processed desktop image is packaged, a preset compression algorithm is adopted to compress the desktop image, and the compressed desktop image is sent to the client. The processing of de-duplication refers to removing the same parts of the front and back frame images, and the refresh area of the desktop can be obtained through the de-duplication step, and the repeated parts are not transmitted, so that the transmission data volume can be reduced.
Step 311. For the processing manner of the non-text interface, the invention is not described in detail, for example, nebula algorithm may be adopted.
In an embodiment of the present invention, whether the desktop image is a text interface is determined by comparing the change rate of the color change of the desktop image, the color change rate of the natural image is usually mild, and the change rate of the text interface image is usually steep, so that whether the desktop image is a text interface can be determined by setting a change rate threshold, and when the change rate is greater than a preset change rate threshold, the desktop image is determined to be a text interface, otherwise, the desktop image is determined to be a non-text interface. For example, the rate of change is determined by calculating the hash value of the sum of all pixels for each pixel row, and then comparing the differences in hash values from row to row.
In another embodiment of the present invention, a deep learning neural network model is used to more accurately determine whether the current desktop image is a text interface, for example, vgg, googlenet, inceptionv4 neural network algorithm models, training the neural network model through a sample to obtain a neural network model that can be applied finally, inputting the desktop image into the trained neural network model, and the neural network model can output the determination result.
Step 302, when judging that the character interface is a character interface, acquiring all first-level blocks intersected with a desktop refreshing area to form a first-level refreshing block set, and forming a first-level non-refreshing block set without intersection, wherein the first-level blocks are screen display areas obtained by segmenting a cloud desktop based on a preset first-level block size;
The purpose of the step is to determine which primary blocks are involved in the desktop refreshing area, and for the primary blocks which are not refreshed, the cache data structure of the primary block image data in the local cache pool can be hit, and the image data can be directly replaced by the block cache identification of the primary block, and for the refreshed primary blocks, the hit judgment and processing of the secondary blocks are required to be continued.
When a text interface is transmitted for the first time, the whole desktop is a refreshing area, and a server side and a client side can locally establish all first-level block cache data structures. After the text interface is refreshed, the cloud desktop server can obtain the position and the size of a desktop refreshing area through the virtual display card driving interface, and can obtain all the primary blocks intersected with the desktop refreshing area based on the distribution position of each primary block on the screen.
In the present invention, the primary blocks may be divided into fixed size, i.e. the size of each primary block is consistent and fixed, as shown in fig. 1. Of course, according to typesetting characteristics of actual literal interfaces of different languages, a non-division fixed-size primary block dividing mode can be adopted, so long as two sides of the cloud desktop server and two sides of the client are synchronous and consistent, the invention is not particularly limited.
Step 303, scanning and cutting the secondary blocks for each primary block in the primary refresh block set, and calculating the hash value of each secondary block.
In an embodiment of the present invention, a method for scanning and splitting a first-level block includes:
Step 3031, performing pixel-level line scanning on the image data of the first-level block, and dividing the area containing the characters and the blank area into a plurality of second-level blocks;
Referring to the example of fig. 2, the ith primary block may be transversely segmented into 4 secondary blocks by pixel line scanning, and the specific segmentation may be that, starting from the first line, the line-by-line scanning is performed, if the current line includes pixels that are not white (assuming that the color of the white area between the text lines is white), the line is illustrated to be located in the text area, the downward scanning is continued until the line with white entire line or the last line of the primary block is scanned, and the plurality of consecutive lines located in the text area in front are divided into a group to form a secondary block. In the same manner, if a line (white-reserving area) with the whole line being white is scanned, the line is positioned in the white-reserving area, and then the downward scanning is continued until the line including non-white pixel points in the whole line or the last line of the first-level block is scanned, and a plurality of continuous lines positioned in the white-reserving area are divided into a group to form a second-level block.
In this way, the primary block can be divided into a plurality of secondary blocks. Considering that the background of some text interface documents is not necessarily pure white, the desktop image can be processed before the secondary blocks are divided, the desktop image is converted into a black-white image (the text is emphasized and the white is left) and then the secondary blocks are divided.
Step 3032, calculating a hash value of each secondary block, generating a secondary block data structure of each secondary block, and forming a secondary block list of the current primary block.
The secondary block data structure can comprise the initial coordinates of the secondary block, the length and width of the block, the hash value of the block, the block cache identification of the block and the like, the secondary block data structure forms a secondary block list of the current primary block, the secondary block list can be temporarily stored in a memory, and the secondary block data structure and the secondary block list are called a secondary block cache data structure and a secondary block cache list after being cached in a local cache pool.
Step 304, calculating hash values of all primary blocks of the desktop image to be transmitted currently;
This step computes the hash values of all primary blocks of the desktop image currently to be transmitted, including primary refresh blocks and primary non-refresh blocks. In an embodiment of the present invention, for a first-level block in a first-level refresh block set, the method for calculating the hash value of the first-level block may have the following two methods:
in a first mode, directly performing hash value calculation on the image data of the first-level block to obtain a hash value of the first-level block;
in a second mode, the hash values of all the secondary blocks of the primary block are combined, the hash value of the combined data is calculated, and the hash value of the combined data is used as the hash value of the image data of the primary block.
Step 305. For each primary block in the set of primary refresh blocks, a block cache identification replacement of the blocks of the hit secondary block is performed.
The first-level blocks in the first-level refreshing block set are first-level blocks which are changed relative to the prior transmission desktop image, so that the first-level blocks in the first-level refreshing block set cannot hit the first-level block cache data structure in the local cache pool, and therefore, the second-level block cache compression transmission is required for the first-level blocks. The basic idea of the block cache compression transmission of the second-level block is that for the second-level block hitting the second-level block cache data structure in the local cache pool, the block image data is replaced by the block cache identifier of the second-level block.
In this embodiment, the local buffer pool is used to buffer the primary block buffer data structure and the secondary block buffer data structure that have been successfully transmitted before, and both the secondary block buffer data structures can be managed by a list. For example, all the first-level block cache data structures form a first-level block cache list, and each first-level block cache data structure is connected to a second-level block cache list formed by all the second-level block cache data structures belonging to the first-level block through pointers. A secondary block cache data structure may include, but is not limited to, block start coordinates, block length and width, block cache identification (or referred to as a block number), hash value of the block. The block cache identifier of a secondary block may be a hash value of the block, or a unique identifier generated by the server and the client according to a preset rule, and used for uniquely identifying a block at both ends. The invention is not particularly limited to the organization and management form of a two-level block cache data structure.
In this step, the following processing procedure 3051-3053 needs to be invoked in a circulating manner, and all the secondary blocks in the primary block are processed one by one until all the secondary blocks in the current primary block are processed.
Step 3051, matching in a second-level block cache list in a local cache pool by using the hash value of the second-level block, and judging whether hit occurs;
Step 3052, if the second-level block cache data structure in the second-level block cache list is hit, the block cache identifier of the second-level block is used to replace the image data of the block.
The image data of the corresponding secondary block and the cache data structure of the corresponding secondary block are cached in the cloud desktop client, and when the cloud desktop client receives the block cache identifier of the secondary block sent by the server, the corresponding block image data can be read from the local cache for display.
Step 3053, if the second-level block cache data structure in the second-level block cache list is not hit, recording the second-level block cache data structure in the local cache pool;
In the case of a miss to a secondary tile, the cache data structure of the secondary tile needs to be recorded in the secondary tile cache list of the current primary tile in the local cache pool.
Step 306, performing block cache identification replacement of the blocks of the hit first-level blocks for the first-level blocks in the first-level non-refreshed block set;
The block cache compression transmission process of the first-level block is similar to the block cache compression transmission process of the second-level block. For a first-level block hitting a first-level block cache data structure in the local cache pool, the block image data is identified by a block cache of the first-level block.
Step 307, the image data of the current desktop image with transmission is packed and compressed and then sent to the cloud desktop client.
In the embodiment of the invention, the cloud desktop server can package and compress the image data to be transmitted by taking the desktop image frame as a unit, the packaged data of the desktop image to be transmitted comprises the block cache identification of the hit block, the image data of the miss block and the like, and the packaged data packet organizes the image data to be transmitted by a certain data structure. In the packed data packet data structure, a hit type identification indicating whether a secondary block is hit or not may be carried for each block.
In the embodiment of the present invention, although the description is made in a sequential step manner, the sequence of performing the block cache compression transmission of the first-stage refresh block and the first-stage non-refresh block is not limited, and the refresh block and the non-refresh block may be arranged in a possible sequence or may be arranged at intervals, for example, the first-stage block of the dashed frame in fig. 1 is a refresh block, the non-dashed frame is a non-refreshed first-stage block, and the server may process the image data transmission of the first-stage non-refresh block after processing the image data transmission of the first-stage refresh block, and specifically, the sequence is determined according to the arrangement relation of the blocks.
In an embodiment of the present invention, in order to avoid the transitional consumption of the local cache and timely release the cache block data that has failed after the refresh, the method further includes an aging cleaning step of the cache data structure of the first-level block and the second-level block, including:
(1) Adding an aging time field for each primary block and each secondary block to record the duration of the cache;
(2) Refreshing a cache time field of a block when the block is hit each time;
(3) And periodically judging whether the buffer time length of each block exceeds a preset threshold through the aging cleaning timer, and releasing the buffer space of the corresponding block if the buffer time length exceeds the preset threshold.
In an embodiment of the present invention, a hit count field may be added to the cache data structure of a second-level block, the hit count is added after each hit, and when the cache data of the block with the low hit rate is cleaned by aging, the cache data of the block with the low hit rate is cleaned preferentially.
In the embodiment of the invention, the cloud desktop client divides the desktop image into blocks by adopting the same primary block division method as the server, and decompresses the data packet after receiving the compressed data packet of the text interface sent by the cloud desktop server by the client, and performs the following processing:
If the hit type identification of the block in the compressed data packet indicates that the first-level block is hit, the client directly reads the image data of the corresponding first-level block from the local cache through the block cache identification carried in the compressed data packet;
if the hit type identification of the block in the compressed data packet indicates that the first-level block and the second-level block are not hit, the client reads the image data of the first-level block from the compressed packet and caches the image data of the corresponding first-level block in a local cache pool;
If the hit type identification of the block in the compressed data packet indicates that the second-level block is hit, the client further analyzes the data structure of the second-level block included in the data packet, and sequentially reads the data structure of the second-level block, if the hit type field in the data structure of the second-level block indicates that the current second-level block is hit, the client directly searches the cache data structure of the second-level block in the local cache pool according to the block cache identification in the data structure of the second-level block, and then reads the image data of the corresponding second-level block, and if the hit type field in the data structure of the second-level block indicates that the current second-level block is not hit, the client reads the image data of the second-level block from the data structure of the current second-level block and caches (or covers the image data of the corresponding region in the current first-level block) in the local cache pool.
The client reads the image data of all the first-level blocks from the compressed data packet, assembles the complete desktop image data, and outputs the complete desktop image data to the display driver of the client so as to display the desktop image.
Fig. 4 is a schematic structural diagram of a server-side cloud desktop image transmission device according to an embodiment of the present invention, where each functional module in the device 400 may be implemented by software, hardware, or a combination of software and hardware. The apparatus 400 includes:
the text interface judging module 401 is configured to judge whether a desktop image to be currently transmitted is a text interface;
The refreshing block acquisition module 402 is used for acquiring all primary blocks intersected with the desktop refreshing area to form a primary refreshing block set when the text interface is judged, wherein the primary blocks are screen display areas obtained by segmenting the cloud desktop based on a preset primary block size;
The second-level block segmentation module 403 is configured to perform scan segmentation of the second-level blocks on each first-level block in the first-level refresh block set, and calculate a hash value of each second-level block;
The primary block hash module 404 is configured to calculate hash values of all primary blocks of the desktop image to be currently transmitted;
The block cache identification replacing module 405 is used for carrying out block cache identification replacement of the blocks of the hit first-level blocks for the first-level blocks in the first-level non-refreshed block set;
The compression transmission module 406 is configured to package and compress the desktop image data to be transmitted currently after completing the block cache identifier replacement processing, and send the compressed desktop image data to the cloud desktop client
Further, the text interface judging module 401 judges whether the desktop image is a text interface by comparing the picture change rate, and judges that the desktop image is a text interface when the change rate is larger than a preset change rate threshold value, or judges that the desktop image is a non-text interface, or judges whether the input desktop image is a text interface by adopting a trained deep learning neural network model.
Further, the second-level block segmentation module 403 performs pixel-level line scanning on the image data of the first-level block, and segments the text-containing region and the blank region into a plurality of second-level blocks.
Further, the apparatus 400 further includes an aging processing module, configured to increase an aging time field for each of the first-level block and the second-level block to record a time length of the buffer, refresh the buffer time field of the block when each hit occurs, periodically determine, by using an aging cleaning timer, whether the buffer time length of each block exceeds a preset threshold, and if so, release the buffer space of the corresponding block.
Fig. 5 is a schematic structural diagram of a client cloud desktop image transmission device according to an embodiment of the present invention, where each functional module in the device 500 may be implemented by software, hardware, or a combination of software and hardware. The apparatus 500 includes:
The decompression module 501 is configured to receive a compressed data packet of a text interface sent by a cloud desktop server, and parse the compressed data packet;
The first-level block processing module 502 is configured to directly read image data of a corresponding first-level block from the local cache through a block cache identifier carried in the data packet when the hit type identifier of the block in the data packet indicates that the first-level block is hit, and read image data of the first-level block from the data packet and cache image data of the corresponding first-level block in the local cache pool when the hit type identifier of the block in the data packet indicates that the first-level block and the second-level block are not hit;
The second-level block processing module 503 is configured to parse the data structure of the second-level block included in the data packet and sequentially read the data structure of the second-level block when the hit type identifier of the block in the data packet indicates that the second-level block is hit, if the hit type field in the data structure of the second-level block indicates that the current second-level block is hit, the client directly searches and reads the image data of the corresponding second-level block in the local cache pool according to the block cache identifier in the data structure of the second-level block, and if the hit type field in the data structure of the second-level block indicates that the current second-level block is not hit, the client reads the image data of the second-level block from the data structure of the current second-level block and caches the image data in the local cache pool.
The block combining module 504 is configured to assemble the image data of all the primary blocks read from the compressed data packet into complete desktop image data, and output the complete desktop image data to a display driver of the client.
Fig. 6 is a schematic diagram of an electronic device according to an embodiment of the present invention, where the device 600 includes a processor 610, such as a Central Processing Unit (CPU), a communication bus 620, a communication interface 640, and a storage medium 630. Wherein the processor 610 and the storage medium 630 may communicate with each other via a communication bus 620. The storage medium 630 stores a computer program for implementing the cloud desktop image transmission method provided by the present invention, or the storage medium 630 includes a software functional module capable of implementing the functions of the apparatus 400 or the apparatus 500, where the computer program is executed by the processor 610 or the software functional module is loaded and executed to implement the functions of each step of the method provided by the present invention.
The storage medium may include a random access Memory (Random Access Memory, RAM) or may include a Non-Volatile Memory (NVM), such as at least one magnetic disk Memory. In addition, the storage medium may be at least one storage device located remotely from the processor. The processor may be a general purpose processor including a central Processing unit (Central Processing Unit, CPU), a network processor (Network Processor, NP), etc., or may be a digital signal processor (DIGITAL SIGNAL Processing, DSP), application SPECIFIC INTEGRATED Circuit (ASIC), field-Programmable gate array (Field-Programmable GATE ARRAY, FPGA) or other Programmable logic device, discrete gate or transistor logic device, discrete hardware components.
It should be appreciated that embodiments of the invention may be implemented or realized in computer hardware, a combination of hardware and software, or by computer instructions stored in non-transitory memory. The method may be implemented in a computer program using standard programming techniques, including a non-transitory storage medium configured with a computer program, where the storage medium so configured causes a computer to operate in a specific and predefined manner. Each program may be implemented in a high level procedural or object oriented programming language to communicate with a computer system. However, the program(s) can be implemented in assembly or machine language, if desired. In any case, the language may be a compiled or interpreted language. Furthermore, the program can be run on a programmed application specific integrated circuit for this purpose. Furthermore, the operations of the processes described in the present invention may be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The processes (or variations and/or combinations thereof) described herein may be performed under control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications), by hardware, or combinations thereof, collectively executing on one or more processors. The computer program includes a plurality of instructions executable by one or more processors.
Further, the method may be implemented in any type of computing platform operatively connected to a suitable computing platform, including, but not limited to, a personal computer, mini-computer, mainframe, workstation, network or distributed computing environment, separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, and so forth. Aspects of the invention may be implemented in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, optical read and/or write storage medium, RAM, ROM, etc., such that it is readable by a programmable computer, which when read by a computer, is operable to configure and operate the computer to perform the processes described herein. Further, the machine readable code, or portions thereof, may be transmitted over a wired or wireless network. When such media includes instructions or programs that, in conjunction with a microprocessor or other data processor, implement the steps described above, the invention described herein includes these and other different types of non-transitory computer-readable storage media. The invention also includes the computer itself when programmed according to the methods and techniques of the present invention.
The foregoing is merely exemplary of the present invention and is not intended to limit the present invention. Various modifications and variations of the present invention will be apparent to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111372689.7A CN114237787B (en) | 2021-11-18 | 2021-11-18 | A cloud desktop image transmission method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111372689.7A CN114237787B (en) | 2021-11-18 | 2021-11-18 | A cloud desktop image transmission method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114237787A CN114237787A (en) | 2022-03-25 |
| CN114237787B true CN114237787B (en) | 2025-02-11 |
Family
ID=80750040
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111372689.7A Active CN114237787B (en) | 2021-11-18 | 2021-11-18 | A cloud desktop image transmission method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114237787B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114422807B (en) * | 2022-03-28 | 2022-10-21 | 麒麟软件有限公司 | Transmission optimization method based on Spice protocol |
| CN115278376B (en) * | 2022-05-25 | 2024-03-22 | 西安万像电子科技有限公司 | Audio and video data transmission method and device |
| CN117640627B (en) * | 2024-01-25 | 2024-04-09 | 微网优联科技(成都)有限公司 | Collaborative work method and system realized through cloud computer |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102541624A (en) * | 2011-12-29 | 2012-07-04 | 国云科技股份有限公司 | Virtual desktop fusion method between desktop virtualization client and virtual desktop |
| CN107145340A (en) * | 2017-03-22 | 2017-09-08 | 深信服科技股份有限公司 | Virtual desktop image transfer method and device |
Family Cites Families (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6571245B2 (en) * | 1998-12-07 | 2003-05-27 | Magically, Inc. | Virtual desktop in a computer network |
| CN107368269B (en) * | 2016-05-11 | 2019-03-26 | 北京京东尚科信息技术有限公司 | Methods, devices and systems of transmission screen images |
| CN110545446B (en) * | 2018-05-28 | 2022-01-18 | 深信服科技股份有限公司 | Desktop image encoding and decoding methods, related devices and storage medium |
| CN109901907B (en) * | 2019-03-15 | 2022-05-20 | 北京星网锐捷网络技术有限公司 | Method and server for transmitting virtual desktop |
| CN111918065B (en) * | 2019-05-08 | 2025-03-25 | 中兴通讯股份有限公司 | Information compression/decompression method and device |
| CN110392262B (en) * | 2019-07-03 | 2022-04-26 | 锐捷网络股份有限公司 | Method and device for compressing virtual desktop image |
| CN111951275A (en) * | 2020-07-27 | 2020-11-17 | 新华三大数据技术有限公司 | Image transmission method and device |
| CN112365317A (en) * | 2020-11-12 | 2021-02-12 | 东方明珠新媒体股份有限公司 | Ordering method and device based on scene virtual dining table |
-
2021
- 2021-11-18 CN CN202111372689.7A patent/CN114237787B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102541624A (en) * | 2011-12-29 | 2012-07-04 | 国云科技股份有限公司 | Virtual desktop fusion method between desktop virtualization client and virtual desktop |
| CN107145340A (en) * | 2017-03-22 | 2017-09-08 | 深信服科技股份有限公司 | Virtual desktop image transfer method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114237787A (en) | 2022-03-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114237787B (en) | A cloud desktop image transmission method and device | |
| US11373272B2 (en) | Digital gradient signal processing system and method for signals comprising at least three dimensions | |
| EP4009231A1 (en) | Video frame information labeling method, device and apparatus, and storage medium | |
| US20140225928A1 (en) | Manipulation of textual content data for layered presentation | |
| US10986380B2 (en) | Method for generating video mask information, method for preventing occlusion from barrage, server and client | |
| CN112215171B (en) | Target detection method, device, equipment and computer readable storage medium | |
| US8782617B2 (en) | Apparatus and method for translating open vector graphic application program interface | |
| CN110235176A (en) | Processing method and processing device, data transmission method and device, the storage medium of image | |
| JP2011138533A5 (en) | ||
| CN102834805B (en) | Screen recording method, screen recording control method and device | |
| CN113516666A (en) | Image cropping method, device, computer equipment and storage medium | |
| US20120005564A1 (en) | Content distribution system and method | |
| CN106920270B (en) | Animation processing method and device | |
| JPWO2011077550A1 (en) | Screen relay device | |
| US8947717B2 (en) | Print data processing apparatus | |
| US11461909B2 (en) | Method, medium, and apparatus for specifying object included in image utilizing inverted index | |
| CN112051974B (en) | Electronic price tag information transmission method and system, electronic price tag, server | |
| CN111954082B (en) | Mask file structure, mask file reading method, computer device and readable storage medium | |
| CN108804652B (en) | Method and device for generating cover picture, storage medium and electronic device | |
| CN114942854B (en) | Screenshot method of GIF (graphics in field) diagram and related equipment | |
| CN114693918B (en) | Image recognition method and device and computer readable storage medium | |
| CN113438444B (en) | Picture transmission storage method and system | |
| US7903119B2 (en) | Compression of image regions according to graphics command type | |
| CN116405724A (en) | Image generation method, system, electronic device and storage medium | |
| CN110189388B (en) | Animation detection method, readable storage medium, and computer device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |