Feature list for fife_launch / fife_wrap

Note: this includes jobsub features

[-- command line]

  • override config file entries -O
  • --dry_run
  • --jobname=name set job name
  • --stage=x -- add [stage_x] option overrides
  • --fake use fake_jobsub instead
  • --config_dump -- dump our configuration for debugging
  • --arg_push/del -- modify args in [executable] section


  • include other config files
  • set macros to use elsewhere


  • set values for jobsub_submit -e [duplicate]

[executable] (_n)

  • set executable(s) and args
  • set stdout/stderr to file

[job_output] (_n)

  • dest=DEST Specify destination for copyBackOutput
  • dest_uniq_rename Copy to unique-ified dest dir, rename to dest directory afterwards
  • rename=RENAME Specify output file rename after Art runs
  • addoutput=ADDOUTPUT glob pattern to match and call addOutputFile on
  • declare_metadata use given program to extract and declare metadata
  • add_metadata=ADD_METADATA single metadata field key=value to add when declaring output files
  • metadata_extractor=METADATA_EXTRACTOR use given program to extract and declare metadata
  • metadata_extractor_unquote=METADATA_EXTRACTOR_UNQUOTE use given program to extract and declare metadata
  • add_to_dataset=ADD_TO_DATASET Add files to named dataset using Dataset.Tag
  • filter_metadata=FILTER_METADATA fields to filter out from metadata extractor, comma separated
  • dataset_exclude=DATASET_EXCLUDE glob pattern of output files to exclude from the add_to_dataset=dataset
  • add_location Add locations of output files to SAM
  • outputlist=OUTPUTLIST ifdh cp -f file to run to copy out outputs
  • hash=HASH hash depth argument to ifdh copyBackOutput


  • find_setups look in the 'usual places' for the ups setups script at startup
  • export=EXPORT export environment variable before running (duplicates env_pass ?)
  • setup=SETUP_ARGS setup product before running
  • setup-local setup all ups products in $INPUT_TAR_FILE directory
  • source=SOURCE source setup file before running
  • modify file loop:
    • ifdh_art executable can run the ifdh_art getNextFile loop
    • getconfig get config files as inputs from SAM (i.e. for MonteCarlo simulation). Conflicts with config
    • multifile Fetch files in wrapper and run executable once per file
    • dynamic_lifetime=n for multifile and getconfig loops quit if less time than this left
    • no-checksum don't do checksums when adding metadata
    • no_delete_after_copy set to not delete files after copying them out
  • quals=QUALS Set qualifiers for ifdh_art setup
  • vers=VERS Set version for ifdh_art setup
  • config=CONFIG Specify config file (fcl) for art executable
  • mix=MIX mixing script to run after fetching each input
  • with_gdb run executable under gdb and print stack trace
  • debug Turn on debugging
  • inputfile=INPUTFILE Input file to copy to job area before running the executable (duplicates submit.f drop?)
  • inputtar=INPUTTAR Input tar file to copy to job area and unpack running the executable (not as good as submit.tar_file_name, drop/merge?)
  • inputlist=INPUTLIST ifdh cp -f file to run to fetch inputs (duplicates submit.TIMEOUT ... drop?)
  • cvmfs-revision=CVMFS_REVISION check for this cvmfs revision on /cvmfs/$
  • self_destruct_timer=SELF_DESTRUCT_TIMER After this many seconds, suicide the job so we get output back (duplicates submit.xx, drop/merge?)
  • prescript=PRESCRIPT script to run before Art executable
  • postscript=POSTSCRIPT script to run after Art executable
  • userscript=USERSCRIPT extra user script to run after main executable


  • limit=LIMIT Set SAM project file delivery limit
  • schema=SCHEMA Set SAM file delivery schema
  • appname=APPNAME application name for SAM
  • appfamily=APPFAMILY application family for SAM
  • appvers=APPVERS application version for SAM


Note: these are all (except n_files_per_job) literally jobsub features. You can repeat them in the ini file by using line, line_1, line_2, etc. as actually repeating "line" entries overrides in .ini files...

  • n_files_per_job compute value for N (number of jobs) below based on number of files in dataset_definition.
  • G <Group/Experiment/Subgroup>, --group=<Group/Experiment/Subgroup> Group/Experiment/Subgroup for priorities and accounting
  • role=<VOMS Role> VOMS Role for priorities and accounting
  • jobsub-server=<JobSub Server> Alternate location of JobSub server to use
  • dropbox-server=<Dropbox Server> Alternate location of Dropbox server to use
  • json-config=<JSON submit config file> a JSON dict file of jobsub_submit options and values
  • tarball-exclusion-file=<tarball exclusion regex file> A file of python regex's to exclude from tarballs. Use with --tar_file_name tardir:// Default exclusion list includes .git and .svn directories, pdf, eps, log files, and human readable images.
  • debug Print debug messages including server contacted, http response, response time
  • jobid-output-only Return only jobsub jobid in response to a successful submission
  • h, --help Show this help message and exit
  • timeout NUMBER[UNITS] kill user job if still running after NUMBER[UNITS] of time . UNITS may be `s' for seconds (the default), `m' for minutes, `h' for hours or `d' h for days.
  • expected-lifetime 'short'|'medium'|'long'|NUMBER[UNITS] Expected lifetime of the job. Used to match against resources advertising that they have REMAINING_LIFETIME seconds left. The shorter your EXPECTED_LIFTIME is, the more resources (aka slots, cpus) your job can potentially match against and the quicker it should start. If your job runs longer than EXPECTED_LIFETIME it may be killed by the batch system. If your specified EXPECTED_LIFETIME is too long your job may take a long time to match against a resource a sufficiently long REMAINING_LIFETIME. Valid inputs for this parameter are 'short', 'medium', 'long', or NUMBER[UNITS] of time. IF [UNITS] is omitted, value is NUMBER seconds. Allowed values for UNITS are 's', 'm', 'h', 'd' representing seconds, minutes, etc.The values for 'short','medium',and 'long' are configurable by Grid Operations, they currently are '3h' , '8h' , and '85200s' but this may change in the future. Default value of EXPECTED_LIFETIME is currently '8h' .
  • maxConcurrent MAXCONCURRENT max number of jobs running concurrently at given time. Use in conjunction with -N option to protect a shared resource. Example: jobsub -N 1000 -maxConcurrent 20 will only run 20 jobs at a time until all 1000 have completed. This is implemented by running the jobs in a DAG. Normally when jobs are run with the -N option, they all have the same $CLUSTER number and differing, sequential $PROCESS numbers, and many submission scripts take advantage of this. When jobs are run with this option in a DAG each job has a different $CLUSTER number and a $PROCESS number of 0, which may break scripts that rely on the normal -N numbering scheme for $CLUSTER and $PROCESS. Groups of jobs run with this option will have the same $JOBSUBPARENTJOBID, each individual job will have a unique and sequential $JOBSUBJOBSECTION. Scripts may need modification to take this into account
  • disk NUMBER[UNITS] Request worker nodes have at least NUMBER[UNITS] of disk space. If UNITS is not specified default is 'KB' (a typo in earlier versions said that default was 'MB', this was wrong). Allowed values for UNITS are 'KB','MB','GB', and 'TB'
  • memory NUMBER[UNITS] Request worker nodes have at least NUMBER[UNITS] of memory. If UNITS is not specified default is 'MB'. Allowed values for UNITS are 'KB','MB','GB', and 'TB'
  • cpu NUMBER request worker nodes have at least NUMBER cpus
  • drain mark this job to be allowed to be drained or killed during downtimes
  • schedd SCHEDD name of alternate schedd to submit to
  • OS OS specify OS version of worker node. Example --OS=SL5 Comma seperated list '--OS=SL4,SL5,SL6' works as well . Default is any available OS
  • show-parsing print out how command line was parsed into argv list and exit. Useful for seeing how quotes in options are parsed)
  • generate-email-summary generate and mail a summary report of completed/failed/removed jobs in a DAG
  • email-to NOTIFY_USER email address to send job reports/summaries to (default is $)
  • G ACCOUNTINGGROUP, --group ACCOUNTINGGROUP Group/Experiment for priorities and accounting
  • subgroup SUBGROUP Subgroup for priorities and accounting. See Jobsub_submit#Groups-Subgroups-Quotas-Priorities for more documentation on using --subgroup to set job quotas and priorities
  • v, --verbose dump internal state of program (useful for debugging)
  • resource-provides RESOURCE_PROVIDES request specific resources by changing condor jdf file. For example: --resource-provides=CVMFS=OSG will add +CVMFS="OSG" to the job classad attributes and '&&(CVMFS=="OSG")' to the job requirements
  • Q, --mail_never never send mail about job results (default)
  • q, --mail_on_error send mail only when job fails due to error
  • M, --mail_always always mail when job completes or fails
  • version Show program's version number and exit
  • g, --grid run job on the OSG grid. Other flags can modify target sites to include public or private clouds
  • c APPEND_REQUIREMENTS, --append_condor_requirements APPEND_REQUIREMENTS append condor requirements
  • overwrite_condor_requirements OVERWRITEREQUIREMENTS overwrite default condor requirements with supplied requirements
  • override OVERRIDE OVERRIDE override some other value: --override 'requirements' 'gack==TRUE' would produce the same condor command file as --overwrite_condor_requirements 'gack==TRUE' if you want to use this option, test it first with -n to see what you get as output
  • e ENV_VAR, --environment ENV_VAR -e ADDED_ENVIRONMENT exports this variable with its local value to worker node environment. For example export FOO="BAR"; jobsub -e FOO <more stuff> guarantees that the value of $FOO on the worker node is "BAR" . Alternate format which does not require setting the env var first is the -e VAR=VAL, idiom which sets the value of $VAR to 'VAL' in the worker environment. The -e option can be used as many times in one jobsub_submit invocation as desired
  • site COMMA,SEP,LIST,OF,SITES submit jobs to these sites
  • blacklist COMMA,SEP,LIST,OF,SITES ensure that jobs do not land at these sites
  • n, --no_submit generate condor_command file but do not submit
  • N NUM submit N copies of this job. Each job will have access to the environment variable $PROCESS that provides the job number (0 to NUM-1), equivalent to the number following the decimal point in the job ID (the '2' in 134567.2).
  • l "line", --lines "line" [Expert option] Add "line" to the Condor submission (.cmd) file, typically as a classad attribute. See the HTCondor documentation for more.
  • L JOBLOGFILE, --log_file JOBLOGFILE Log file to hold log output from job.
  • compress_log Compress --log_file output .
  • use_gftp use grid-ftp to transfer file back
  • tar_file_name dropbox://PATH/TO/TAR_FILE tardir://PATH/TO/DIRECTORY specify TAR_FILE or DIRECTORY to be transferred to worker node. TAR_FILE will be copied to an area specified in the jobsub server configuration, transferred to the job and unpacked there. TAR_FILE will be accessible to the user job on the worker node via the environment variable $INPUT_TAR_FILE. The unpacked contents will be in the same directory as $INPUT_TAR_FILE.
  • f INPUT_FILE at runtime, INPUT_FILE will be copied to directory $CONDOR_DIR_INPUT on the execution node. Example :-f /grid/data/minerva/my/input/ will be copied to $CONDOR_DIR_INPUT/ Specify as many -f INPUT_FILE_1 -f INPUT_FILE_2 args as you need. To copy file at submission time instead of run time, use -f dropbox://INPUT_FILE to copy the file.
  • d OUTPUT_DIR_ARRAY OUTPUT_DIR_ARRAY -d <tag> <dir> Writable directory $CONDOR_DIR_<tag> will exist on the execution node. After job completion, its contents will be moved to <dir> automatically Specify as many <tag>/<dir> pairs as you need.
  • dataset_definition DATASET_DEFINITION SAM dataset definition used in a Directed Acyclic Graph (DAG)
  • project_name PROJECT_NAME optional project name for SAM DAG