Project

General

Profile

Feature #20213

Multiparameter workflow

Added by Anna Mazzacane about 2 years ago. Updated over 1 year ago.

Status:
Closed
Priority:
Normal
Assignee:
Target version:
Start date:
06/22/2018
Due date:
% Done:

100%

Estimated time:
Scope:
Internal
Experiment:
-
Stakeholders:
Duration:

Description

Experiments need to have an easy way to design many workflows by changing one or more parameters.
We have found this situation at least in DUNE (for MCC9) and SBND with their starting MC production.
In these cases, the workflow is the usual:
gen->g4->detsim->reco->...
where gen needs to be produced x times for x different fcl files and then proceed through all other stages.
POMS can handle this type of workflow by using the "dataset" field and the split type "list" to pass the different fcl files, i.e. list(fcl1,fcl2,fcl3)
An example is the campaign ID 1571 for SBND.
In order to proceed from gen to the other stages, dependent campaign stages need to be created and the it should be
job_output.add_to_dataset = _poms_task

Possible future development discussed with Marc:
Create a generic field "parameter" which can be an array (for an ntuple of parameters).

History

#1 Updated by Stephen White about 2 years ago

  • Target version set to v4_1_0

#2 Updated by Stephen White almost 2 years ago

  • Assignee set to Marc Mengel

#3 Updated by Marc Mengel over 1 year ago

First part of this is multiparam splitter type...

#4 Updated by Marc Mengel over 1 year ago

Okay, so one can use a split_type of:

  • list(), with a list of values in the "dataset" field
  • multiparam(), with a list of lists (in json) in the "dataset" field, and
    it will permute one from column a with one from column b, etc.

One can then use either %(dataset)s or %(parameter)s to get use the resulting values in your launch commands.

Need to look at updating field descriptions that currently say "Dataset" to say "Dataset or Parameter Lists".

#5 Updated by Marc Mengel over 1 year ago

Okay so now it's referred to as "Dataset or Split Data" except on the
campaign_stage_info page where it's either "Dataset" or "Split Data"
depending if it has commas in it (which really ought to be a check on
whether its a "generator" type campaign stage, as well, but whatever.)

And you can access the split result as "dataset" or "parameter" now in %(dataset)s or %(parameter)s in your command expansions.

#6 Updated by Marc Mengel over 1 year ago

  • Status changed from New to Resolved

#7 Updated by Marc Mengel over 1 year ago

Came back and made a first stab at allowing a dataset_or_split_data custom editor in each split_type.

#8 Updated by Marc Mengel over 1 year ago

  • % Done changed from 0 to 100

#9 Updated by Stephen White over 1 year ago

  • Status changed from Resolved to Closed


Also available in: Atom PDF