WO2021136068A1 - Resource dispatching method and apparatus, electronic device, and computer readable medium - Google Patents

Abstract

The present disclosure provides a resource dispatching method. Said method comprises: determining, in response to a new task request, whether actuators are all in a working state, if so, adding a new task to a level-1 cache queue to wait, and if not, adding the new task to a level-2 cache queue to wait or executing the new task by means of the actuators. The present disclosure further provides a resource dispatching apparatus, an electronic device and a computer readable medium.

Description

Resource scheduling method, device, electronic equipment and computer readable medium

Cross-references to related applications

This application is based on a Chinese patent application with application number 201911395401.0 and an application date of December 30, 2019, and claims the priority of the Chinese patent application. The entire content of the Chinese patent application is hereby incorporated into this application by way of introduction.

Technical field

The embodiments of the present disclosure relate to the field of deep learning technology, and in particular, to a resource scheduling method, device, electronic device, and computer-readable medium.

Background technique

After the training is completed, the deep learning model is often deployed on a server based on cloud computing resources, so that the client can use the model according to its own push-off task. In practical applications, the server usually serves the inference task requests of different clients at the same time, and the inference task requests of these clients often use different models or different versions of the model. The server must not only meet the client's requirements for latency, but also consider its own performance (such as throughput) and resource utilization efficiency.

Due to the strong computing power of the server, the time it takes to complete one inference task request and multiple inference task requests is not much different. In order to improve performance and resource efficiency, the server usually combines multiple small reasoning task requests (with less resource occupancy) into a large task set. For example, when a reasoning task request contains a small amount of data, the reasoning Task requests are placed in the waiting queue, waiting for subsequent inference task requests to be merged into a task set with a large amount of data, and then calculations are performed. This scheduling method undoubtedly increases the delay of the client. Moreover, when the server has available computing resources (such as idle threads, CPUs, GPUs, etc.), since it needs to form a task set with sufficient data volume before computing starts, the server is "forced" to be in an idle state, resulting in a waste of server resources . In addition, when the server starts computing, it also takes a certain amount of time to copy the task set from the cache to the computing resources.

Summary of the invention

The embodiments of the present disclosure provide a resource scheduling method, device, electronic device, and computer readable medium to solve the problems of long time delay and poor resource usage efficiency due to the defects of the server scheduling method in the prior art.

In the first aspect, embodiments of the present disclosure provide a resource scheduling method, which includes: responding to a new task request to determine whether the executors are all in a working state; if so, adding the new task to the first-level cache queue to wait; if not, then The new task is added to the second-level cache queue to wait or be executed by the executor.

In a second aspect, embodiments of the present disclosure provide an electronic device, which includes: one or more processors; a storage device, on which one or more programs are stored, when the one or more programs are used by the one or more Execution by multiple processors, so that the one or more processors implement the resource scheduling method provided in the embodiments of the present disclosure.

In a third aspect, an embodiment of the present disclosure provides a computer-readable medium on which a computer program is stored, and when the program is executed by a processor, the resource scheduling method provided by the embodiment of the present disclosure is implemented.

Description of the drawings

The accompanying drawings are used to provide a further understanding of the embodiments of the present disclosure, and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure, and do not constitute a limitation to the present disclosure. The above and other features and advantages will become more apparent to those skilled in the art by describing detailed example embodiments with reference to the accompanying drawings. In the accompanying drawings:

FIG. 1 is a schematic block diagram of a server provided by an embodiment of the disclosure;

FIG. 2 is a flowchart of another resource scheduling method provided by an embodiment of the disclosure;

FIG. 3 is a flowchart of another resource scheduling method provided by an embodiment of the disclosure;

FIG. 4 is a flowchart after the executor completes the task in the resource scheduling method of the embodiment of the disclosure;

Fig. 5 is an exemplary block diagram of a resource scheduling device provided by an embodiment of the disclosure.

Detailed ways

In order to enable those skilled in the art to better understand the technical solutions of the present application, the resource scheduling method, device, electronic equipment, and computer-readable medium provided in the present application will be described in detail below with reference to the accompanying drawings.

Hereinafter, example embodiments will be described more fully with reference to the accompanying drawings, but the example embodiments may be embodied in different forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, the purpose of providing these embodiments is to make the present disclosure thorough and complete, and to enable those skilled in the art to fully understand the scope of the present disclosure.

As used herein, the term "and/or" includes any and all combinations of one or more related listed items.

The terms used herein are only used to describe specific embodiments and are not intended to limit the present disclosure. As used herein, the singular forms "a" and "the" are also intended to include the plural forms, unless the context clearly dictates otherwise. It will also be understood that when the terms "comprising" and/or "made of" are used in this specification, it specifies the presence of the described features, wholes, steps, operations, elements and/or components, but does not exclude the presence or Add one or more other features, wholes, steps, operations, elements, components, and/or groups thereof.

The embodiments described herein can be described with reference to plan views and/or cross-sectional views with the help of ideal schematic diagrams of the present disclosure. Therefore, the example illustrations may be modified according to manufacturing technology and/or tolerances. Therefore, the embodiment is not limited to the embodiment shown in the drawings, but includes a modification of the configuration formed based on the manufacturing process. Therefore, the regions illustrated in the drawings have schematic properties, and the shapes of the regions shown in the figures exemplify the specific shapes of the regions of the elements, but are not intended to be limiting.

Unless otherwise defined, the meanings of all terms (including technical and scientific terms) used herein are the same as those commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having meanings consistent with their meanings in the context of the related technology and the present disclosure, and will not be interpreted as having idealized or excessive formal meanings, Unless this article specifically defines it as such.

The resource scheduling method provided by the embodiments of the present disclosure is mainly for the deep learning training module to be deployed on a cloud resource-based server after the deep learning training module completes the training, so as to be called by the client. In order to reduce the delay of the server, while taking into account the throughput, the resource utilization rate of the server is improved.

The server provided by the embodiment of the present disclosure adopts a multi-level caching mechanism, and according to the usage of the executor in the server, the tasks of the client are cached in different cache queues, so that the server can complete the tasks in time.

FIG. 1 is a schematic block diagram of a server provided by an embodiment of the disclosure. As shown in FIG. 1, the server includes a cache module 100 and an execution module 200. The cache module 100 includes at least one first-level task set 101 for storing new tasks in the first-level cache queue. Each first-level task set 101 can at least Store a task. The execution module 200 includes at least one executor 201, and the executor 201 is used to store new tasks in the second-level cache queue and execute tasks, and each second-level task set 201 can store at least one task. Performing tasks can be a process of calculating the characteristics of the graph based on the model.

In the first aspect, an embodiment of the present disclosure provides a resource scheduling method. FIG. 2 is a flowchart of the resource scheduling method of an embodiment of the disclosure.

As shown in FIG. 2, the resource scheduling method provided by the embodiment of the present disclosure includes:

201: Determine whether the actuators are all in working state.

Respond to the new task request to determine whether the actuators in the execution module are all in working state.

It should be noted that the actuator usually has three states, namely, idle state, waiting state and working state. When the actuator is in working condition, it cannot take over the task. When the actuator is in the idle state and waiting state, it can take over tasks. Tasks include but are not limited to reasoning tasks.

202: When the executors are all in a working state, add a new task to the first-level cache queue to wait.

When all the actuators are in working state, it means that there are no available resources, and the new task is added to the first-level cache queue to wait.

203. When there is an executor in a non-working state, the new task is added to the second-level cache queue to wait or be executed by the executor.

When there is a non-working executor, it indicates that there are available resources, and the new task is added to the second-level cache queue. New tasks added to the second-level cache queue may be executed immediately, or they may need to wait in the second-level cache queue and be executed when the conditions are met.

In some embodiments, when it is judged that at least one actuator is in a non-working state, it can be further judged whether there is an actuator in a waiting state, and if not, it is judged whether there is an actuator in an idle state. In other words, when it is determined that at least one executor is in a non-working state, priority is given to the executors in the waiting state, and then the executors in the idle state are allocated tasks. When there is no executor in the waiting state and the executor in the idle state, the task is added to the first-level cache queue to wait.

According to the resource scheduling method provided by the embodiment of the present disclosure, when a new task request is received, if there are available resources, the new task will enter the second-level cache queue, and start calculation immediately when the conditions are met, without additional waiting and copying time If there are no available resources, the new task will enter the first-level cache queue. When the executor is free, the new task will be scheduled as soon as possible, which not only reduces the delay and improves the user experience, but also takes into account the throughput of the server. Increase the efficiency of the server.

FIG. 3 is a flowchart of a resource scheduling method provided by an embodiment of the disclosure.

3, the resource scheduling method provided by the embodiment of the present disclosure includes:

301: Determine whether there is an actuator in a waiting state.

When the server receives the task request from the client, it determines whether there is an executor in the waiting state in the server. If the server has an executor in the waiting state, it means that the server has available resources, then step 302 is executed; if the server does not have an executor in the waiting state, then go to step 304.

302. Obtain the total data volume of the second-level task set according to the current data volume of the current second-level task set and the data volume of the new task, and determine whether the total data volume is greater than the preset maximum second-level task set data volume; if so, Go to step 303; if not, go to step 305.

Among them, the secondary task set corresponding to the executor in the waiting state usually already has tasks, and the data amount of the existing tasks in the current secondary task set and the data amount of the new task are added to obtain the total number of the current secondary task set. The maximum data volume of the second-level task set is preset and can be set according to actual needs, or can be set according to the storage capacity of the second-level task set.

It should be noted that the second-level task set includes an open state and a closed state. When the second-level task set is in the open state, a new task can be added to the cache queue of the second-level task set. When the secondary task set is in the closed state, no new tasks can be added, and when the secondary task set is in the closed state, the executor will execute the tasks in the secondary task set.

303. The current secondary task set is closed, and the executor enters the working state.

When the total data volume is greater than the preset maximum data volume of the second-level task set, the current second-level task set is closed, that is, the status of the current second-level task set is changed to the closed state, and the executor corresponding to the current second-level task set enters work Status, start to calculate the tasks in the current secondary task set. It should be noted that because the total data volume is greater than the preset maximum data volume of the second-level task set, the new task cannot be arranged in the cache queue of the second-level task set, and therefore, the new task is not executed by the executor that enters the working state.

304: Determine whether there is an actuator in an idle state.

When the total data volume is greater than the preset maximum data volume of the second-level task set, it indicates that the new task cannot be arranged in the cache queue of the current second-level task set, and a new executor needs to be found. Therefore, it is necessary to determine whether there is an idle actuator, if yes, go to step 306; if not, go to step 309.

305. Add the new task to the current secondary task set.

When the total data volume is not greater than the preset maximum data volume of the second-level task set, the new task is added to the current second-level task set, and then go to step 307.

306. Add the new task to the newly created new secondary task set.

When there is an executor in an idle state in the execution module, a new secondary task set is created, the new task is added to the new secondary task set, and then step 307 is performed.

307. Determine whether the data amount of the secondary task set is equal to the maximum data amount of the secondary task set.

In step 307, whether it is the current secondary task set or the new secondary task set, it is determined whether the data amount of the secondary task set is equal to the maximum secondary task set data amount.

When the data amount of the secondary task set is equal to the maximum data amount of the secondary task set, go to step 303. When the data amount of the secondary task set is less than the maximum data amount of the secondary task set, step 308 is executed.

308. Modify the actuator to a waiting state.

When the data amount of the second-level task set is less than the maximum data amount of the second-level task set, the executor corresponding to the new second-level task set enters the waiting state. For the current secondary task set, since it is already in the waiting state, there is no step to modify the state.

309. Determine whether there is a current first-level task set in an open state in the first-level cache queue.

When there is no idle executor, it means that the execution module has no available resources, the new task needs to be cached in the buffer module, and it is necessary to find a suitable first-level task set for the new task.

Therefore, by judging whether there is an open current level 1 task set in the level 1 cache queue, new tasks are preferentially added to the buffer queue of the current level 1 task set that is open. When there is an open current level one task set in the first level cache queue, step 310 is executed; when there is no current level one task set in an open state in the first level cache queue, step 314 is executed.

310. Determine whether the total data volume of the first-level cache is less than or equal to a preset maximum data volume of the first-level task set.

Among them, the total data volume of the first-level cache is obtained based on the current data volume of the current first-level task set and the data volume of the new task, that is, the current data volume of the current first-level task set and the data volume of the new task are added to obtain the first-level cache The total amount of data.

If the total data amount is equal to or less than the preset maximum level one task set data amount, go to step 311; if the total data amount is greater than the preset maximum level one task set data amount, go to step 313.

311. Add the new task to the current first-level task set.

When the total data volume is equal to or less than the preset maximum data volume of the first-level task set, it indicates that the first-level task set still has space for storing new tasks, and the new task is entered into the current first-level task set.

312. Determine whether the total data amount is equal to the maximum first-level task set data amount.

It is judged whether the total data volume is equal to the data volume of the maximum first-level task set. If so, the current first-level task set is closed, that is, the status of the current first-level task set is changed to the closed state. The first level cache queue no longer receives new tasks. If not, keep the open state of the current first-level task set to receive new tasks and join the first-level cache queue to wait.

313, close the current first-level task set.

If the total data volume is greater than the preset maximum data volume of the first-level task set, it indicates that the first-level task set no longer receives new tasks, and then step 314 is executed. Closing the current first-level task set, that is, modifying the state of the current first-level task set to a closed state, and the first-level cache queue of the current first-level task set no longer receives new tasks.

314. Determine whether the first-level cache queue is full.

When there is no first-level task set in an open state in the first-level cache queue, it is determined whether the first-level cache queue is full. Or, when the current level 1 task set is closed, it is determined whether the level 1 cache queue is full. If the level cache queue is full, a failure message will be returned, indicating that the current server is busy and cannot receive new tasks from the client. If the L1 cache queue is not full, step 315 is executed.

315, create a new first-level task set.

If the first-level cache queue is not full, it means that the server still has available resources, create a new first-level task set, and set the newly created first-level task set to an open state. Since other task sets in the server have been closed, the newly created first-level task is integrated into the current first-level task set.

In the foregoing embodiment, the secondary task set is closed when the total data volume meets the condition, so that the executor executes the tasks in the secondary task set.

In some embodiments, the secondary task set may also be closed based on the waiting time of the executor. Specifically, when the time that the actuator is in the waiting state reaches a preset waiting time period, the secondary task set corresponding to the actuator is closed and enters the working state.

It should be noted that when the executor is in the waiting state for the preset waiting time, regardless of whether the data volume of the second-level task set reaches the maximum data volume of the second-level task set, it is adjusted to the closed state and the corresponding execution The inverter enters the working state, so as to avoid excessive delay.

In some embodiments, after the executor completes the tasks in the secondary task set, it can also obtain a new task set from the cache device.

FIG. 4 is a flowchart after the executor completes the task in the resource scheduling method of the embodiment of the disclosure. As shown in Figure 4, after the executor completes the tasks in the secondary task set, it also includes:

401: Determine whether the first level cache queue is empty.

The first level cache queue is a task queue cached when the execution module is in a busy state. When the first level cache queue is empty, it means that the server has not received a new task. If the first-level cache queue is empty, step 402 is executed; if the first-level cache queue is not empty, step 403 is executed.

402. The actuator enters an idle state.

Since there is no new task, the state of the executor is changed to the idle state.

403. Extract the task set in the first-level cache queue.

When the first-level cache queue is not empty, it means that a client has sent new tasks, and these tasks need to be processed in time. The server extracts the first-level task set from the first-level cache queue. The extracted first-level task set preferentially selects the first-level task set formed in the cache module to avoid increasing the delay.

In some embodiments, as long as the first-level task set extracted by the executor, regardless of whether the first-level task set is in the closed state, all enter the execution module.

404: The executor enters the working state to execute the task set.

After the executor extracts the first-level task set, it immediately turns to the working state and completes the task.

In the second aspect, embodiments of the present disclosure provide a resource scheduling device. Fig. 5 is an exemplary block diagram of a resource scheduling device provided by an embodiment of the disclosure. As shown in FIG. 5, the resource scheduling device includes: a cache module 500 and an execution module 600, wherein the cache module 500 is used to store new tasks of the first-level cache queue; the execution module 600 is used to store the information of the second-level cache queue New task and execution of said new task.

In some embodiments, the cache module 500 includes at least one first-level task set 501, and the first-level task set 501 is used to store new tasks in the first-level cache queue when the executors are all in a working state;

The execution module 600 includes at least one executor 601, and each executor 601 includes a pair of a secondary task set 611 and an execution model 612. The secondary task set 611 is used when the executor is in an idle state or a waiting state. When the time, the new task of the second-level cache queue is stored; the executor 612 is used to execute the task.

In the resource scheduling device provided by the embodiment of the present disclosure, when the server receives a new task request, it determines whether the executors in the execution device are all in a working state, and if not, the new task is added to the first-level cache queue to wait; if so, then The new task is added to the second-level cache queue to wait or be executed by the executor, that is, if there are available resources, the new task will enter the second-level cache queue, and start calculation immediately when the conditions are met, without additional waiting and Copy time, if there are no available resources, the new task will enter the first level cache queue. When the executor is free, the new task will be scheduled as soon as possible, which not only reduces latency and improves user experience, but also takes into account the server The throughput improves the efficiency of the server.

In a third aspect, embodiments of the present disclosure provide an electronic device, which includes: one or more processors; a storage device, on which one or more programs are stored, when one or more programs are used by one or more processors Execution, so that one or more processors implement any one of the foregoing resource scheduling methods.

In a fourth aspect, embodiments of the present disclosure provide a computer-readable medium on which a computer program is stored, and when the program is executed by a processor, any one of the foregoing resource scheduling methods is implemented.

A person of ordinary skill in the art can understand that all or some of the steps, functional modules/units in the system, and apparatus in the methods disclosed above can be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware embodiments, the division between functional modules/units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may consist of several physical components. The components are executed cooperatively. Some physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit . Such software may be distributed on a computer-readable medium, and the computer-readable medium may include a computer storage medium (or a non-transitory medium) and a communication medium (or a transitory medium). As is well known by those of ordinary skill in the art, the term computer storage medium includes volatile and non-volatile memory implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Sexual, removable and non-removable media. Computer storage media include but are not limited to RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassette, tape, magnetic disk storage or other magnetic storage device, or Any other medium used to store desired information and that can be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that a communication medium usually contains computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and may include any information delivery medium. .

Example embodiments have been disclosed herein, and although specific terms are adopted, they are used and should only be interpreted as general descriptive meanings, and are not used for the purpose of limitation. In some instances, it is obvious to those skilled in the art that, unless expressly indicated otherwise, the features, characteristics, and/or elements described in combination with a specific embodiment may be used alone, or features, characteristics, and/or elements described in combination with other embodiments may be used. Combination of features and/or components. Therefore, those skilled in the art will understand that various changes in form and details can be made without departing from the scope of the present disclosure as set forth by the appended claims.

Claims (15)

A resource scheduling method, which includes: In response to a new task request, it is determined whether the executors are all in a working state, and if so, the new task is added to the first-level cache queue to wait; if not, the new task is added to the second-level cache queue to wait or be executed by the executor. The method according to claim 1, wherein the adding the new task to the first level cache queue to wait comprises: It is determined whether there is a current first-level task set in an open state in the first-level cache queue; if so, the new task enters the current first-level task set. The method according to claim 2, wherein after the new task enters the current first-level task set, the method further comprises: The total data volume is obtained according to the current data volume of the current first-level task set and the data volume of the new task; if the total data volume is equal to or less than the preset maximum first-level task set data volume, the new task Enter the current first-level task set; if the total data volume is greater than the preset maximum first-level task set data volume, close the current first-level task set, and create a new first-level task set, so that the new task Enter the new level task set and wait. The method according to claim 3, wherein after the new task enters the current first-level task set, the method further comprises: It is determined whether the total data amount is equal to the maximum first-level task set data amount, and if so, the current first-level task set is closed. The method according to claim 3, wherein after the total data amount is greater than the preset maximum data amount of the first-level task set and the current first-level task set is closed, the method further comprises: Determine whether the first-level cache queue is full; if not, create a new first-level task set and set the new first-level task set to an open state; if yes, return a failure message. The method according to claim 2, wherein when there is no current first-level task set in an open state in the first-level cache queue, Determine whether the first-level cache queue is full; if not, create a new first-level task set and set the new first-level task set to an open state; if yes, return a failure message. The method according to claim 1, wherein the judging whether the executors are all in a working state in response to a task request comprises: In response to the task request, determine whether there is an executor in the waiting state; if so, obtain the total data volume of the current secondary task set according to the current data volume of the current secondary task set and the data volume of the new task; if not, Then judge whether there is an actuator in an idle state. The method according to claim 7, wherein, after obtaining the total data amount of the current secondary task set according to the current data amount of the current secondary task set and the data amount of the new task, the method further comprises: Determine whether the total data volume of the current secondary task set is greater than the preset maximum data volume of the secondary task set, and if so, close the current secondary task set and enable the executor corresponding to the current secondary task set to enter Work status, and determine whether there is an executor in an idle state; if not, add the new task to the newly created new secondary task set. The method according to claim 8, wherein after adding the new task to the newly created new secondary task set, the method further comprises: Judging whether the data amount of the new secondary task set is equal to the maximum secondary task set data amount, and if so, close the new secondary task set, and make the executor corresponding to the new secondary task set enter a working state; If not, the executor corresponding to the new secondary task set enters a waiting state. The method according to claim 8, wherein after the determining whether there is an executor in an idle state, the method further comprises: If there are idle executors, create a new secondary task set; If there is no idle executor, it is judged whether there is an open task set in the second-level cache queue. The method according to claim 10, wherein after said creating a new secondary task set, it further comprises: Determine whether the total data volume of the new second-level task set is greater than the preset maximum data volume of the second-level task set, and if so, close the new second-level task set and enable the executor corresponding to the new second-level task set Enter the working state; if not, make the executor corresponding to the new secondary task set enter the waiting state. The method according to claim 9 or 11, wherein when the executor is in the waiting state for a preset waiting period of time, the secondary task set corresponding to the executor is closed, and the working state is entered. The method according to claim 1, wherein after the executor completes the task, it further comprises: Determine whether the first-level cache queue is empty, if so, the executor enters an idle state; if not, extract the first-level task set in the first-level cache queue; Perform the set of tasks. An electronic device, which includes: One or more processors; A storage device having one or more programs stored thereon, and when the one or more programs are executed by the one or more processors, the one or more processors implement any one of claims 1-13 The method described in the item. A computer-readable medium with a computer program stored thereon, which, when executed by a processor, implements the method according to any one of claims 1-13.

Images