Project

General

Profile

Design notes

This page was started by J. Wolcott as a place to consolidate thoughts about design considerations.
Hopefully it will eventually evolve into a design specification...

Major functions of production scripts

  • Prepare framework options files
  • Prepare non-framework wrapper scripts (e.g., for MINOS overlay/sim)
  • Prepare/execute jobsub commands
  • Provide interface for finding processed data/MC
  • Provide interface for managing catalog of processed files

Design patterns

  • Production happens in "stages": GENIE, Cal, Reco, Ana, ....
  • A thing you can do is specified by a (stage, run/subrun) pair, ie "Run this stage on this subrun"

Notes

  • Should probably work in a "plug-in" manner to make future additions more straightforward
    • (PAR) An example might be having a "plug-in" that takes a list of (stage, run/subrun) pairs and runs or submits them. Then you'd have one class for submitting with jobsub, one for submitting with jobsub_client, and one for running interactively.
  • Want to consolidate all the regular expressions into one place rather than have them littered everywhere
    • (PAR) Eg, via a class that knows how to write options in options files, like optionsFileWriter.setOption("EventSelector.EvtMax", 2)
  • Just one top-level ProcessX.py script that can do data, MC, and ana
  • (PAR) Maybe have each stage know what options it can take, so the top-level ProcessX.py only needs to know general options like run number and input/output dirs
  • (PAR) Need a more generic way of mangling file names (ie, if we're on grid, it'll be in $CONDOR_DIR_INPUT, but if we're interactive, we need to turn /pnfs paths into dcap URLs). Right now, this is done case-by-case, and have to remember it everywhere.
  • (PAR) Right now, some stages run SystemTestsApp.exe directly, some write their own wrapper script, some call GaudiRunProcessor.py. We should make them all do the same thing: more consistency should mean less code