Support workflows where dependencies are live-streamed
This from a discussion at the DUNE computing model workshop.
Our current model is that a stage runs and stores its output, and the next stage reads that stored outputAnother approach is to have one stage generating a stream, and the next stage reading that live stream; this
- the two stages must be launched essentially in parallel (or the second stage first, and the first one when the second one is running...) (recursively if there is a N stage stream).
- One may need extra submission flags to make sure the parts are able to communicate.
This would be flags on the dependencies.