Project

General

Profile

TaskRequirements

The system should be capable of managing various granularities of tasks:

  • High level tasks, or "campaigns"
    • examples:
      • process all experiment data with version 9 of reconstructionn
      • run all 6 phases of MonteCarlo with specified versions of phases for a billion events
    • May involve multiple dependant tasks/workflow
    • Generally must be broken down into reasonalby sized batches to actually run
    • Campaign level tasks have state:
      • new
      • pending approval
      • approved
      • running
      • succeeded/failed
  • Batches
    • small enough portion of a high-level task to be submitted at once
    • includes tracking:
      • sam project progress
      • job exit codes
      • output files handled via FTS
    • Batches have workflow matching campaign, but on their subset of the data
    • Some rule or mechanism for splitting campaign datasets into batch-level datasets must be provided
    • Batch-level tasks may also have completion criteria, such as:
      • output datasets reaching tape
      • output datasets reaching same count as input dataset
      • check script passing on output data and/or logs
      • experiment check-off saying it is finished
    • Batch level tasks can be considered to have state:
      • pending
      • submitted
      • project running (if SAM project involved)
      • jobs completed
      • batch completed (per criterea above)
  • Automatic submission
    • Periodical processing of certain type of data with particular software versions.
      • Easily configurable:
        • Define dataset pattern
        • Turn on/off
        • Define frequency.
        • Define time offset
        • Scripts/fcl files to use
      • Implement authentication policy approved by production team and CD.
    • Spans batch submissions automatically.