Project

General

Profile

Task #15335

Add support to submit jobs for a workflow splitted in n DAGs

Added by Vito Di Benedetto over 3 years ago. Updated over 3 years ago.

Status:
Closed
Priority:
Normal
Category:
-
Target version:
Start date:
Due date:
% Done:

100%

Estimated time:
10.00 h
Spent time:
Duration:

Description

In the usual experiment workflow there are N jobs submitted for each exp stage.
The exp stages are sequential.
Then there is a special stage where the files of the last exp stage are merged and some code is execute to analyze data in the merged file to produce plots.
This kind of workflow can be represented by a DAG with independent branches, where the nodes in each branch are jobs for each exp stage.
Then another node depending on the jobs of the last exp stage can be added to the DAG.
In this configuration, if a job fails, the jobs in the other branches are not affected, but the job associated to the last node of the DAG will not be executed as there is a dependency that failed.

In a configuration where the exp workflow can be split in n DAGs where each DAG consist of independent branches, a failure in one of the job in one of the branches doesn't affect the subsequent DAG.

In this configuration, the first job that is executed for the last stage of a DAG, it will submit a monitor job that monitors the status of the jobs of the DAG and when all the jobs of the DAG are completed, the monitor job will submit the next DAG

History

#1 Updated by Vito Di Benedetto over 3 years ago

  • Status changed from Assigned to Work in progress
  • % Done changed from 0 to 80

The implementation of this feature is done, now it needs to be tested.

#2 Updated by Vito Di Benedetto over 3 years ago

  • Status changed from Work in progress to Resolved
  • % Done changed from 80 to 100

#3 Updated by Vito Di Benedetto over 3 years ago

  • Status changed from Resolved to Closed

#4 Updated by Vito Di Benedetto over 3 years ago

  • Start date deleted (01/26/2017)

#5 Updated by Vito Di Benedetto over 3 years ago

  • Parent task deleted (#13748)


Also available in: Atom PDF