Project

General

Profile

TutorialOutline

Front page

  • Service status
  • Hold/allow jobs

First Time Setup

Before we start

  • make sure you can
    • log in somewhere
    • kdestroy (so you arent' relying on your krb5 creds)
    • setup things
    • submit a small job of this type
    • with above specifying on command line
      • software version,
      • SAM dataset
      • number of jobs
  • Have a log/detailed notes of same.

config overview

(yes it needs a setup wizard...)

Three components, in left menu, to setup:

  • Launch Template
    • "To prepare to launch jobs I ssh into xxx and source yyy and zzz"
  • Job Type
    • "To submit a monte carlo job, I run exp_launch.py --dataset fred"
  • Campaign Layer
    • "This is Fred's monte carlo run with version x and configs y"

Launch template

"To prepare to launch jobs I ssh into xxx and source yyy and zzz"

  • left menu
  • Compose Launch Template
  • pick experiment (if not there, file ticket)
  • Click [ + Add ]
  • Give it a name (i.e. exppro_on_expgpvm02), a hostname (i.e. expgpvm02) , and account (i.e. exppro)
  • Fill in a setup entry like "source /some/exp/setup; setup exp_sw %(version)s"
  • NOTE: use %(version)s if you're setting up some version, so POMS can fill it in with the correct version from a given campagin layer later.
  • hit Submit
  • Bonus points: ssh into that account, add
    poms/cd/pomsgpvm01.fnal.gov@FNAL.GOV
    to .k5login

Job Type

"To submit a monte carlo job, I run 'exp_launch.py --mc --dataset fred'"

  • left menu
  • Compose Job Type
  • pick experiment (if not there, file ticket)
  • Click [ + Add ]
  • Give it a name (i.e. exp_mc_type1)
  • list Output File Patterns (i.e. mc_out%.root) of outputs that will get declared to SAM
  • Launch script
    • give it a launch script to run, that will have parameters passed to it.
    • if your usual launch script doesn't accept command line parameters for, say, dataset definition, etc. you need to put in a shell function that does, then call that...
    • I recommend fife_launch :-)
  • Definition Parameters
    • click [/] (w/ pencil) to edit
    • this is a mechanism to define things that might get overridden further, so it's slightly tricky to do right the first time; We'll talk more about this later.
    • left hand "Key" column must be unique
    • use [+] button to add more pairs.
    • name value pairs that make parameters to script
    • If its like -O sect.var=value split it as [-O sect.var=] [value]
    • if option needs space after it, include in box [-g ] [group]
  • Recovery Launches
    • click [/] (w/ pencil) to edit
    • Pick recovery type for up to 5 recoveries.
    • recommend 1. consumed_status 2. pending files
    • hit [Accept]
  • hit [Submit]

Campaign Layer

"This is Fred's monte carlo run with version x and configs y"

  • left menu
  • Compose Campaign Layer
  • pick experiment (if not there, file ticket)
  • Click [ + Add ]
  • Give it a name i.e. "Spring16CosmicsMC"
  • List VO Role (i.e Production or Calibration...)
  • Software version -- exp software version that will be in output metadata
  • Dataset -- full input dataset (i.e. mc config files, raw files to process, etc.)
  • Dataset split type
    • If Dataset is large; and not defined as "draining"..
    • usually mod(k) to split into k pieces is good
    • new(window=2d,start_time=123456) for two days of data per submission
    • let us know if other split types are useful (i.e. by subrun?)
  • Completion type/percent -- (i.e. Located @ 95% (if using SAM))
  • Parameter Overrides
    • edit with [/] (w/pencil) click, as with Job Types, above
    • here is where picking good parameters in the Job Type pays off
    • If something needs to be different/added to the launch in the Job Type for this campaign, you can add it here
    • anything with matching Key here overrides one from Job Type
    • new Keys are added to options
    • hit [Accept] when done
  • Depends On
    • Will do more later, but:
    • edit with [/] (w/pencil) click
    • Can pick other exisitng Campaign layer for your experiment (but not yet, as we don't have any)
    • Can pick this campaign layer, means launch another batch as each one completes.
    • Specify particular output file pattern from that campaign's output you use as input.
    • hit [Accept] when done
  • Launch Template
    • at the moment we have 1 to choose from, can pick any you have, fields show up.
  • Job Type
    • at the moment we have 1 to choose from, can pick any you have, fields show up.
  • Hit [Submit]

Actually Launching

  • Launch by hand
    • left menu
    • Active for exp
    • Click on campaign layer name
    • Launch Campaign Jobs Now
  • Launch via cron (show uBooNE Electron Lifetime for example)
    • left menu
    • Active for exp
    • Click on campaign layer name
    • Clikc on Schedule Future Job Launches
    • Fill in entry form for crontab

Watching what goes on

  • left menu
  • Active for exp
  • Click on Campaign Layer name
  • Several options
    • By submission-jobs: (control?)-click Submission Time Bars
      • Page shows time bars for each submission
      • Can click through to jobs from that submission
      • Can click through again to job triage
    • By submission-files: Campaign Submission Files
      • Can click through to specific file listings

Debugging launch problems

  • Make a debug Job Type
    • clone existing one
    • name it Debug-xxx
    • stick an "printenv ; echo" in front of command you usually run
    • switch your campaign to use the Debug-xxx Job Type
  • launch may be working due to kerberos ticket interactively
    • log into account
    • kdestroy
    • test setup/launch using exp. proxy