CN114356643A - Automatic task discovery failure and recovery method in remote sensing satellite processing system - Google Patents

Abstract

The application provides a method for automatically discovering task failure and recovering in a remote sensing satellite processing system, which comprises the following steps: according to the established error capturing mechanism, under the condition that the execution failure of the task is monitored, the execution of the failed task is initiated again; if the failed task fails to be executed again, terminating the execution of the first task flow corresponding to the failed task; the first task process comprises at least one subtask; analyzing an influence domain of task execution failure on the execution process of the first task flow, and determining a target task with the minimum influence degree on restarting the execution process of the first task flow in the first task flow; creating a second task flow based on the target task, and taking the target task as an initial task of the second task flow; the second task process is a task process executed when the first task process is restarted after the first task process is terminated.

Description

A method for automatically discovering mission failure and recovery in a remote sensing satellite processing system

technical field

The present application relates to the field of remote sensing satellite data processing, in particular to a method for automatically discovering task failure and recovery in a remote sensing satellite processing system.

Background technique

In a remote sensing satellite processing system, there are usually dozens or even hundreds of processing nodes, and there are hundreds of thousands of processing tasks at the same time. In the process of task processing, due to occasional errors in hardware and system software, such as occasional read and write errors in the storage file system, occasional errors in cluster communication, occasional errors in the job scheduling system, etc., it is easy to fail to process the task, causing the system to fail to process normally. Remote sensing data, resulting in reduced system availability.

In related technologies, problems can be found and solved manually, but it requires meticulous work and energy, and the efficiency is low; or a fault-tolerant mechanism oriented to dedicated computing functions is adopted to increase the fault-tolerant capability of the system, but the time overhead is relatively large.

SUMMARY OF THE INVENTION

The purpose of this application is to provide a method for automatically discovering and recovering task failures in a remote sensing satellite processing system, which is used for automatic discovery and recovery of failed tasks in a sensing satellite processing system.

The present application provides a method for automatically discovering task failure and recovery in a remote sensing satellite processing system, including:

According to the established error capture mechanism, in the case of monitoring that the task execution fails, the execution of the failed task is re-initiated; if the failed task fails to be executed again, the execution of the first task flow corresponding to the failed task is terminated; The first task flow includes at least one subtask; analyze the impact domain of task execution failure on the execution process of the first task flow, and determine the execution process for restarting the first task flow in the first task flow Create a second task flow based on the target task, and use the target task as the starting task of the second task flow; wherein, the second task flow is the first task After the process is terminated, the task process executed when the first task process is restarted.

Optionally, according to the established error capture mechanism, in the case of monitoring that the task execution fails, the execution of the failed task is re-initiated; if the failed task fails to be executed again, the first corresponding to the failed task is terminated. Before the execution of the task flow, the method further includes: establishing an error capture mechanism, and during the execution of the task flow of the system, if the task execution fails, output a corresponding standard error output file.

Optionally, according to the established error capture mechanism, in the case of monitoring that the task execution fails, the execution of the failed task is re-initiated; if the failed task fails to be executed again, the first corresponding to the failed task is terminated. The execution of the task process includes: monitoring the generation of the standard error output file, and in the case of monitoring the generation of the standard error output file, according to the generated standard error output file, determining the first task corresponding to the first task node where the task execution failed. task information; re-execute the first task according to the task information of the first task.

Optionally, analyzing the impact domain of the task execution failure on the execution process of the first task process, and determining the first task process that has the least impact on restarting the execution process of the first task process. The target task includes: in the case of failure to re-execute the first task, determining the first task flow corresponding to the first task according to the task information of the first task; wherein the first task The task node is a task node corresponding to the first task in the first task flow.

Optionally, in the case where re-execution of the first task fails, after determining the first task flow corresponding to the first task according to the task information of the first task, the method further Including: traversing the first task node in the first task flow and the input data of the tasks corresponding to each task node in the previous task nodes in reverse order; , and the second task corresponding to the second task node that has input data required for executing the task is determined as the target task.

Optionally, the determining, according to the generated standard error output file, the task information of the first task corresponding to the first task node whose task execution fails includes: using a job scheduling system to obtain the standard error output file, and according to the The erroneous task name in the standard error output file, and/or the erroneous task number determines the task information of the first task; wherein, the job scheduling system includes any one of the following: Portable Batch Batch System PBS, Load Sharing Facility LSF , SLURM, a simple Linux utility for resource management.

In the remote sensing satellite processing system provided by the present application, the method for automatically discovering mission failure and recovery is to first establish an error capture mechanism, and according to the established error capture mechanism, in the case of monitoring the failure of mission execution, re-initiate the execution of the failed mission; If the failed task fails to be executed again, the execution of the first task process corresponding to the failed task is terminated. Afterwards, the impact domain of the task execution failure on the execution process of the first task process is analyzed, and the target task in the first task process that has the least impact on restarting the execution process of the first task process is determined. Finally, a second task flow is created based on the target task, and the target task is used as the starting task of the second task flow, which solves the problem of processing task failure caused by the accidental failure of the processing system, and replaces the traditional manual Sophisticated methods such as screening for failed tasks and manually initiating erroneous tasks increase the usability of the system.

Description of drawings

In order to illustrate the technical solutions in the present application or the prior art more clearly, the following briefly introduces the accompanying drawings that are needed in the description of the embodiments or the prior art. Obviously, the drawings in the following description are of the present application. For some embodiments of the present invention, for those of ordinary skill in the art, other drawings can also be obtained from these drawings without any creative effort.

Fig. 1 is one of the flow charts of automatic discovery task failure and recovery method in the remote sensing satellite processing system provided by the application;

Fig. 2 is the second schematic flow chart of automatic discovery task failure and recovery method in the remote sensing satellite processing system provided by the application;

Fig. 3 is the schematic flow chart of capturing task error provided by the application;

Fig. 4 is a schematic diagram of the analysis process flow of error task impact domain provided by the present application;

FIG. 5 is related information in the process description file provided by this application.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below with reference to the accompanying drawings in the present application. Obviously, the described embodiments are part of the embodiments of the present application. , not all examples. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

The terms "first", "second" and the like in the description and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It is to be understood that the data so used are interchangeable under appropriate circumstances so that the embodiments of the present application can be practiced in sequences other than those illustrated or described herein, and distinguish between "first", "second", etc. The objects are usually of one type, and the number of objects is not limited. For example, the first object may be one or more than one. In addition, "and/or" in the description and claims indicates at least one of the connected objects, and the character "/" generally indicates that the associated objects are in an "or" relationship.

In a remote sensing satellite processing system with dozens or hundreds of processing nodes and a petabyte (PB) level storage system, there are hundreds of thousands of processing tasks at the same time. This processing system is a typical computing-intensive and data-intensive processing system. In the process of task processing, due to occasional errors in hardware and system software, such as occasional read and write errors in the storage file system, occasional errors in cluster communication, occasional errors in the job scheduling system, etc., it is easy to fail to process the task, causing the system to fail to process normally. Remote sensing data, resulting in reduced system availability.

In order to solve such problems, there are currently two main solutions: 1) Humans find errors, determine the steps where the errors occur, and then start from a reasonable intermediate step or from the initial step, which requires meticulous work and energy, and is inefficient; 2) Fault tolerance and initiation measures for general big data processing jobs, such as Apache Spark, a fast and general computing engine designed for large-scale data processing, can perform redundant state management based on finer-grained data or tasks, but it is also limited Considering the versatility of the framework, it is difficult to further optimize the features such as fine-grained job tasks, complex dependencies between tasks, and diverse fault tolerance requirements; 3) Adopt fault tolerance oriented to dedicated computing functions, and increase tolerance and initiation mechanisms in the form of checkpoints , such as Apache Flink, which uses a "distributed snapshot" checkpoint mechanism to complete fault tolerance. When executing batch tasks, it adopts the method of executing tasks from scratch to achieve fault tolerance. Although this method is simple to implement, it brings a lot of time overhead.

In order to improve the efficiency of discovering failed tasks and recovering, as shown in FIG. 1 , the schematic diagram of the automatic discovery task failure and recovery method in the remote sensing satellite processing system provided by the embodiment of the present application, firstly establish an error capture mechanism when the task execution fails, The task program is developed based on the error capture mechanism, so that the failure information can be output to the standard error output file when the task execution fails. After discovering the failed task, immediately re-initiate the failed task and continue the target process; if it fails again, terminate the execution of the first task process corresponding to the failed task; perform an impact domain analysis on the task process corresponding to the failed task, and according to the analysis The result resumes the execution of the task.

The present application proposes a method for automatically discovering and recovering task failures in a remote sensing satellite processing system by capturing the task error log, analyzing the impact domain of the error task process, restarting the relevant process, etc., so as to solve the processing task caused by the accidental failure of the processing system. The problem of failure improves the availability of the system.

The following describes in detail the method for automatically discovering task failure and recovering in the remote sensing satellite processing system provided by the embodiments of the present application through specific embodiments and application scenarios with reference to the accompanying drawings.

As shown in FIG. 2 , a method for automatically discovering task failure and recovery in a remote sensing satellite processing system provided by an embodiment of the present application may include the following steps 201 to 203:

Step 201, according to the established error capture mechanism, in the case of monitoring that the task execution fails, re-initiate the execution of the failed task; if the failed task fails to be executed again, then terminate the first task process corresponding to the failed task. implement.

Wherein, the first task flow includes at least one subtask.

Exemplarily, the above-mentioned first task process is any one of multiple task processes processed by the remote sensing satellite processing system. Usually, the data processing of each orbit of the remote sensing satellite processing system includes multiple logically sequenced task processes, and each task process contains at least one sub-task; and concurrent task processes in the unit of ten thousand are processed at the same time every day.

In a possible implementation manner, in order to monitor the above-mentioned multiple task processes and find out a task process in which the task execution fails, the embodiment of the present application further provides a mechanism for capturing task errors.

Exemplarily, in the remote sensing satellite processing system provided by the embodiment of the present application, in the method for automatically discovering task failure and recovery, before step 201 to step 203, it is also necessary to establish a monitoring mechanism for task failure, that is, the above-mentioned error capture mechanism, the task failure detection mechanism. The mechanism may specifically include the following steps 204:

Step 204 , establishing an error capturing mechanism, and during the execution of the task flow of the system, if the task execution fails, output the corresponding standard error output file.

Exemplarily, taking the execution process of the first task flow as an example, during the execution of the first task flow, the scheduling system will capture the standard error output of the failed task and input it into the standard error output file.

Exemplarily, in the process of processing a task by the remote sensing satellite processing system, a task program development specification and a job script writing specification are formulated. When the processing system is running, the job script of the task is generated according to the set processing flow. The job script includes: the execution path of the task, the calling information such as input parameters, and the standard error output path when the task execution fails.

When the processing system executes the task, the job script of the task is submitted to the scheduling system, and the scheduling system calls the task program indicated by the execution path in the job script, and executes the task program according to the input parameters in the job script.

When an abnormal error occurs in the task program running and the task execution fails, the task program written in accordance with the specification will write the error into the standard error output file, and the scheduling system will output the output of the task program and the standard error at the operating system level to the above standard error. in the output path. The processing system determines that the task execution fails.

Exemplarily, by monitoring the above-mentioned standard error output file, the task process monitoring can be implemented, and the failure of task execution can be found in time.

For example, as shown in Figure 3, the running program corresponding to the task needs to be written in accordance with the specification. After the processing system generates the running script (that is, the above job script), it is submitted to the job scheduling system. The job scheduling system captures the standard error output and generates Error output file. The processing system acknowledges the failed task.

It is understandable that due to the certain dependencies between the various task processes and the various tasks of Zhang Hong in the task process during the processing, when the task execution of a task node fails, it will affect the task execution of subsequent task nodes. implement. Therefore, when a task execution failure is detected during the execution of the first task process, the following two processing methods are used for processing:

Way 1:

In processing mode 1, the task is re-executed immediately.

Exemplarily, when execution of the first task of the first task node in the first task flow fails, the first task may be determined as the above-mentioned target task, and the first task is re-executed. That is, an attempt is made to directly re-execute the first task. If the execution is successful, it means that the impact of the failure of the task execution is small, and the first task can be restored to normal by re-executing the first task.

Exemplarily, the above target task includes: a task corresponding to the current task node whose task execution fails.

Exemplarily, the above step 201 may include the following steps 201a and 201b:

Step 201a: Monitor the generation of the standard error output file, and determine the task information of the first task corresponding to the first task node whose task execution fails according to the generated standard error output file when the generation of the standard error output file is monitored.

Step 201b: Re-execute the first task according to the task information of the first task.

Exemplarily, according to the error capture mechanism established in step 204, it is possible to monitor in real time whether a new standard error output file is generated. When monitoring the generation of a new standard error output file, the generated standard error output file can be used. Identify tasks that failed to execute. All failed tasks in the system can be monitored through steps 201a and 201b.

Way 2:

If processing mode 1 fails, continue to open mode 2.

In processing mode 2, the analysis of the impact domain is performed, and the matched task flow is restarted.

Step 202: Analyze the impact domain of the task execution failure on the execution process of the first task process, and determine the target task in the first task process that has the least impact on restarting the execution process of the first task process.

Exemplarily, the above-mentioned influence domain is used to reflect the degree of influence of the failure of executing the task on the recovery of the task flow.

Exemplarily, after terminating the execution of the first task flow, the first task flow needs to be analyzed, and the function of the analysis is to determine which task flow starting node to execute from when restarting, and the first task flow needs to be analyzed. The execution of the process has minimal impact.

It can be understood that when the execution of the first task process is terminated due to a task execution failure, the execution of the first task process can be restarted directly from the initial task process of the first task process, but such processing This method will repeat and execute tasks that have been executed normally, resulting in a waste of system resources. Therefore, when the task execution fails, it needs to be analyzed to determine the target task in the task flow that has the least impact on restarting the execution process of the first task flow.

Specifically, the task of the superior node that needs to be re-executed can be determined according to whether the starting node of the first task node where the task execution fails still has input data required for executing the task.

Exemplarily, the relevant information of the upper-level task node of the first task node may be determined through the following steps, that is, the above step 202, which may include the following step 202a:

Step 202a: In the case of failure to re-execute the first task, determine the first task flow corresponding to the first task according to the task information of the first task.

The first task node is a task node corresponding to the first task in the first task process.

Exemplarily, the above task failure monitoring mechanism can monitor whether a new standard error output file is created, and after monitoring that the new standard error output file is created, determine the failure of executing the failed task according to the standard error output file. task information.

Exemplarily, the first task may be determined according to relevant information in the standard error output file output when the first task fails to execute. The standard error output file may include information such as the task ticket number of the first task, the task program path of the first task, and input data required for the execution of the first task. At the same time, the standard error output file may also include process information of the first task process. According to the process information, the task information of all upper-level task nodes in the task process where the first task node is located (ie, the above-mentioned first task process) can be determined.

For example, if the task program is written in C++ language, the error message should be output with cerr when writing the code; if the task program is written in java, the error message should be output with System.err.println. Taking the task program AUTOCHECKGEO as an example, it is automatically generated by the processing system. The job script file to be submitted to the job scheduling system must contain the following contents:

#PBS -e /public/VRS//work/vrs20220105011831/Log/pbs/

/public/VRS//main/modules/bin/AUTOCHECKGEO

/public/VRS//work/vrs20220105011831/Parameter/AUTOCHECKGEO.xml

Among them, vrs20220105011831 is the unique task order number; /public/VRS//work/vrs20220105011831/Log/pbs/ specifies the error file output path, and the processing system outputs the captured error information to the AUTOCHECKGEO.sh.e2230612 file in the directory. Go, 2230612 in the file name is the submitted job number; /public/VRS//main/modules/bin/AUTOCHECKGEO /public/VRS//work/vrs20220105011831/Parameter/ AUTOCHECKGEO.xml respectively specifies the program called by the job For AUTOCHECKGEO, the input parameters are contained in AUTOCHECKGEO.xml.

Specifically, the above step 201a may include the following step 202a1:

Step 201a1: Use a job scheduling system to acquire the standard error output file, and determine the task information of the first task according to the error task name and/or the error task number of the standard error output file.

Wherein, the job scheduling system includes any one of the following: Portable Batch System (PBS), Load Sharing Facility (LSF), Simple Linux Utility for Resource Management (Simple Linux Utility for Resource) Management, SLURM).

Exemplarily, when a task fails to be executed, task information such as the task name and task number of the task, and error information are input into the corresponding standard error output file name or file content. When the system detects that a new standard error output file is generated, it can determine the failed task according to the information in the standard error output file.

Illustratively, the above-mentioned Portable Batch System (PBS), Load Sharing Facility (LSF), Simple Linux Utility for Resource Management (SLURM), or self- Define a method to get the standard error output file.

Exemplarily, after the above step 202a, the above step 202 may further include the following steps 202b and 202c:

Step 202b, traverse the input data of the tasks corresponding to the first task nodes in the first task flow and the tasks corresponding to each task node in the previous task nodes in reverse order.

Step 202c: Determine the second task corresponding to the second task node that is closest to the first task node and has input data required for executing the task in the first task flow as the target task.

Exemplarily, after the re-execution of the first task fails, the execution of the first task process needs to be resumed from the upper-level task node of the first task node. Specifically, the task corresponding to the task node that is closest to the first task node and has input data required for executing the task may be determined as the target task according to the proximity principle.

It can be understood that if the upper-level node of the task process where the first task node is located does not save the input data for executing the task corresponding to the node, the execution of the first task process cannot be resumed from this node, and it is necessary to traverse the higher-level task process. Whether to save the input data required to execute the task until the task process start node with the input data required to execute the task is traversed, and the task corresponding to this node is determined as the target task.

It should be noted that, if neither the first task node nor the initial task node of the upper-level task process of the first task node has the input data required to execute the task corresponding to the task node, that is, the second task cannot be found. node, the re-execution of the first task process is terminated.

For example, as shown in FIG. 4 , in the schematic diagram of the analysis process of the error task impact domain provided by the embodiment of the present application, after the error output file is found, the execution of the error task is re-initiated, and if the execution is successful, the recovery process is ended; if the execution fails , then carry out an impact domain analysis, and determine the target task that has the least impact on the execution of the restoration of the first task process, and then create a second task process according to the target task, and then obtain the input parameters required to execute the task. This task provides data assurance.

Step 203: Create a second task flow based on the target task, and use the target task as a starting task of the second task flow.

The second task flow is a task flow executed when the first task flow is restarted after the first task flow is terminated.

Exemplarily, after determining the target task in the first task flow that has the least impact on restarting the execution process of the first task flow, the execution of the first task flow can be restarted and the above target task is re-executed.

It should be noted that, since the above-mentioned first task flow has been terminated in the case of re-executing the failed task and failing again, in order to complete the execution of the subsequent tasks in the first task flow, a new task flow needs to be created, that is, The above second task flow.

Exemplarily, after obtaining the task information of the first task through the above-mentioned standard error output file, the first task flow corresponding to the first task can be matched according to the task information, thereby realizing the control of the first task flow, And use the first task process as a recovery process. Specifically, the task process can be matched through the process description file.

As shown in Figure 5, it is the relevant information in the process description file, where L1A represents the level of the data, and workflowid=87 represents the task process number 87, and paraformat represents the task program and its sequence included in the task process. order. According to the data level, it can be found from the task process description table that its minimum impact domain is L1A, and the process where L1A is located is the recovery process. Then obtain the initial input data of the recovery process. If the initial input data does not exist, it will trace back to the upper-level data until the original data. If the data of a certain level in the traceback exists, specify the process of the data at this level as the task recovery process; if all levels of data do not exist, it does not match the recovery process. In the process description file shown in FIG. 5 , if the L1A data does not exist, the processing system sequentially matches the CAT level up to the R0 level.

Specifically, before restarting the execution of the first task flow, a job script of the tasks to be executed or re-executed in the first task flow may be sequentially generated according to the obtained input data, including: the target task and the target task Job script for tasks to be executed after. Afterwards, according to the execution sequence of the generated job script, the restoration of the first task flow is completed by creating and executing a new task flow (ie, the second task flow described above).

For example, as shown in FIG. 5, taking AUTOCHECKGEO as an example, if the current input data L1A data exists, the matching task flow is the task flow No. 87, and it is used as the recovery flow. The input parameter when the error occurs is: AUTOCHECKGEO.xml, and the execution program is AUTOCHECKGEO; when the task process is restarted, it is re-executed according to the recovery process numbered 87. Since the initial running program of the recovery process is CORRECT, the processing system first generates CORRECT.xml and CORRECT.sh, and then submits the CORRECT.sh script, and executes tasks such as CORRECT and AUTOCHECKGEO in sequence.

It should be noted that the task failure processing method provided in this application is not limited to the field of satellite data processing, and is also applicable to the field of large-scale concurrent operations such as satellite data quality evaluation and satellite data deep processing.

In the remote sensing satellite processing system provided by the embodiment of the present application, the method for automatically discovering task failure and recovery is to first establish an error capture mechanism, and according to the established error capture mechanism, in the case of monitoring that the task execution fails, the execution of the failed task is re-initiated; If the failed task fails to be executed again, the execution of the first task process corresponding to the failed task is terminated. Afterwards, the impact domain of the task execution failure on the execution process of the first task process is analyzed, and the target task in the first task process that has the least impact on restarting the execution process of the first task process is determined. Finally, a second task flow is created based on the target task, and the target task is used as the starting task of the second task flow, which solves the problem of processing task failure caused by the accidental failure of the processing system, and replaces the traditional manual Sophisticated methods such as screening for failed tasks and manually initiating erroneous tasks increase the usability of the system.

It should be noted that, in the embodiments of the present application, the above methods are shown in the accompanying drawings. The automatic discovery task failure and recovery methods in the remote sensing satellite processing system are all illustratively described with reference to a figure in the embodiment of the present application as an example. During specific implementation, the method for automatically discovering task failure and recovering in the remote sensing satellite processing system shown in the drawings of the above methods can also be implemented in combination with any other drawings shown in the above embodiments that can be combined, and will not be repeated here.

From the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and certainly can also be implemented by hardware. Based on this understanding, the above-mentioned technical solutions can be embodied in the form of software products in essence or the parts that make contributions to the prior art, and the computer software products can be stored in computer-readable storage media, such as ROM/RAM, magnetic A disc, an optical disc, etc., includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform the methods described in various embodiments or some parts of the embodiments.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: it can still be The technical solutions described in the foregoing embodiments are modified, or some technical features thereof are equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions in the embodiments of the present application.

Claims (6)

1. A method for automatically discovering task failure and recovering in a remote sensing satellite processing system is characterized by comprising the following steps:

according to the established error capturing mechanism, under the condition that the execution failure of the task is monitored, the execution of the failed task is initiated again; if the failed task fails to be executed again, terminating the execution of the first task flow corresponding to the failed task; the first task process comprises at least one subtask;

analyzing an influence domain of task execution failure on the execution process of the first task flow, and determining a target task with the minimum influence degree on restarting the execution process of the first task flow in the first task flow;

creating a second task flow based on the target task, and taking the target task as an initial task of the second task flow;

the second task process is a task process executed when the first task process is restarted after the first task process is terminated.

2. The method according to claim 1, characterized in that, in case of monitoring a task execution failure according to an established error trapping mechanism, the execution of the failed task is reinitiated; if the failed task fails to be executed again, before terminating the execution of the first task flow corresponding to the failed task, the method further includes:

and establishing an error capturing mechanism, and outputting a corresponding standard error output file if the task fails to be executed in the execution process of the task flow of the system.

3. The method according to claim 2, characterized in that, in case of monitoring a task execution failure according to the established error trapping mechanism, the execution of the failed task is reinitiated; if the failed task fails to execute again, terminating the execution of the first task flow corresponding to the failed task, including:

monitoring generation of a standard error output file, and determining task information of a first task corresponding to a first task node with task execution failure according to the generated standard error output file under the condition that the generation of the standard error output file is monitored;

and re-executing the first task according to the task information of the first task.

4. The method according to claim 3, wherein the analyzing the influence domain of the task execution failure on the execution process of the first task flow and determining the target task in the first task flow with the least influence on restarting the execution process of the first task flow comprises:

under the condition that the re-execution of the first task fails, determining the first task flow corresponding to the first task according to the task information of the first task;

the first task node is a task node corresponding to the first task in the first task flow.

5. The method according to claim 4, wherein after determining the first task flow corresponding to the first task according to task information of the first task in case of failure to re-execute the first task, the method further comprises:

sequentially traversing the input data of the tasks corresponding to the first task node and each task node in the previous task nodes in the first task process in a reverse order;

and determining a second task corresponding to a second task node which is closest to the first task node in the first task flow and has input data required by task execution as the target task.

6. The method according to claim 3, wherein the determining task information of the first task corresponding to the first task node with failed task execution according to the generated standard error output file comprises:

acquiring the standard error output file by adopting an operation scheduling system, and determining task information of the first task according to an error task name and/or an error task number in the standard error output file;

wherein the job scheduling system comprises any one of: a portable batch system PBS, a load sharing facility LSF, a simple Linux utility SLURM for resource management.

Images