Experiments need to have an easy way to design many workflows by changing one or more parameters.
We have found this situation at least in DUNE (for MCC9) and SBND with their starting MC production.
In these cases, the workflow is the usual:
where gen needs to be produced x times for x different fcl files and then proceed through all other stages.
POMS can handle this type of workflow by using the "dataset" field and the split type "list" to pass the different fcl files, i.e. list(fcl1,fcl2,fcl3)
An example is the campaign ID 1571 for SBND.
In order to proceed from gen to the other stages, dependent campaign stages need to be created and the it should be
job_output.add_to_dataset = _poms_task
Possible future development discussed with Marc:
Create a generic field "parameter" which can be an array (for an ntuple of parameters).
#4 Updated by Marc Mengel about 1 year ago
Okay, so one can use a split_type of:
- list(), with a list of values in the "dataset" field
- multiparam(), with a list of lists (in json) in the "dataset" field, and
it will permute one from column a with one from column b, etc.
One can then use either %(dataset)s or %(parameter)s to get use the resulting values in your launch commands.
Need to look at updating field descriptions that currently say "Dataset" to say "Dataset or Parameter Lists".
#5 Updated by Marc Mengel about 1 year ago
Okay so now it's referred to as "Dataset or Split Data" except on the
campaign_stage_info page where it's either "Dataset" or "Split Data"
depending if it has commas in it (which really ought to be a check on
whether its a "generator" type campaign stage, as well, but whatever.)
And you can access the split result as "dataset" or "parameter" now in %(dataset)s or %(parameter)s in your command expansions.