Project

General

Profile

Feature #23247

Support workflows where dependencies are live-streamed

Added by Marc Mengel 3 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Target version:
Start date:
09/10/2019
Due date:
% Done:

0%

Estimated time:
Scope:
Internal
Experiment:
-
Stakeholders:
Duration:

Description

This from a discussion at the DUNE computing model workshop.

Our current model is that a stage runs and stores its output, and the next stage reads that stored output

Another approach is to have one stage generating a stream, and the next stage reading that live stream; this
means:
  • the two stages must be launched essentially in parallel (or the second stage first, and the first one when the second one is running...) (recursively if there is a N stage stream).
  • One may need extra submission flags to make sure the parts are able to communicate.

This would be flags on the dependencies.



Also available in: Atom PDF