Jobsub command reference

NOTE: This applies to the submission node only.


To submit jobs with jobsub you need to:

  • make sure you are registered with a Fermilab experiment
  • setup your grid proxy initially


General Options

--version             show program's version number and exit
-h, --help            show this help message and exit
--OS=OS             specify OS version of worker node. Default is SL5,
and SL6 is supported. Comma seperated list '--
OS=SL4,SL5,SL6' works as well
Group/Experiment/Subgroup for priorities and
-v, --verbose       dump internal state of program (useful for debugging)
-M, --mail_always   send mail when job completes or fails
-q, --mail_on_error
send mail only when job fails due to error (default)
-Q, --mail_never    never send mail (default is to send mail on error)
-T, --test_queue    Submit as a test job.  Job will run with highest
possible priority, but you can only have one such
job in the queue at a time.
-g, --grid          run job on the FNAL GP  grid. Other flags can modify
target sites to include other areas of the Open
Science Grid
--nowrapfile        do not generate shell wrapper for fermigrid
operations. (default is to generate a wrapfile)
-c APPEND_REQUIREMENTS, --append_condor_requirements=APPEND_REQUIREMENTS
append condor requirements
overwrite default condor requirements with supplied
override some other value: --override 'requirements'
'gack==TRUE' would produce the same condor command
file as --overwrite_condor_requirements 'gack==TRUE'
if you want to use this option, test it first with -n
to see what you get as output
-C                  execute on grid from directory you are currently in
-e ADDED_ENVIRONMENT exports this variable and its
local value to worker node environment. For example
export FOO="BAR"; jobsub -e FOO <more stuff>
guarantees that the value of $FOO on the worker node
is "BAR" . Can use this option as many times as
submit to different host
--site=SITE         submit jobs to this site
-p                  use parrot to run on afs (only makes sense with -a
--pOff              turn parrot off explicitly (this is the default)
-n, --no_submit     generate condor_command file but do not submit
--opportunistic     submit opportunistically to Fermigrid GP Grid and CDF
Grid. This option will allow you to potentially get
more slots than your Fermigrid quota, but these slots
are subject to preemption
-N QUEUECOUNT       submit N copies of this job. Each job will
have access to the environment variable
$PROCESS that provides the job number (0 to
<num>-1), equivalent to the decimal point in
the job ID (the '2' in 134567.2).
location of X509_USER_PROXY (expert mode)
-l LINES, --lines=LINES
[Expert option] Add the line <line> to the
Condor submission (.cmd) file. See Condor
help for more.

File Options

-a, --needafs       run on afs using parrot (this is discouraged)
Log file to hold log output from job.
--no_log_buffer     write log file directly to disk. Default is to copy it
back after job is completed. This option is useful
for debugging but can be VERY DANGEROUS as joblogfile
typically is sent to bluearc. Using this option
incorrectly can cause all grid submission systems at
FNAL to become overwhelmed resulting in angry admins
hunting you down, so USE SPARINGLY.
--use_gftp          use grid-ftp to transfer file back
create self extracting tarball from contents of
INPUT_TAR_DIR. This tarball will be run on the worker
node with arguments you give to your_script
name of tarball to submit, created from
--input_tar_dir if specified
overwrite TAR_FILE_NAME when creating tarfile using
-f INPUT_DIR_ARRAY  -f <file>         input file <file> will be copied to
$CONDOR_DIR_INPUT on the execution node.
Example :-f /grid/data/minerva/my/input/
will be copied to $CONDOR_DIR_INPUT/
Specify as many -f file1 -f file2 args as you need.
-d<tag> <dir> Writable directory $CONDOR_DIR_<tag>
exist on the execution node. After job completion,
its contents will be moved to <dir> automatically
Specify as many <tag>/<dir> pairs as you need.

SAM Options

SAM dataset definition used in a Directed Acyclic
Graph (DAG)
optional project name for SAM DAG

Nova Specific Options
--SMU steer jobs to HPC.SMU grid site

-i RELDIR           release_directory for Nova Software
-t TESTRELDIR       release_directory for test Nova Software
-r REL              release_version for  Nova Software


You can have as many instances of -c, -d, -e, -f, -l and -y as you need.
The -d directory mapping works on non-Grid nodes, too.
export IFDH_VERSION=some_version then use -e IFDH_VERSION to use 
the (some_version) release of ifdh for copying files in and out
with -d and -f flags instead of the current ifdh version in KITS
More documentation is available at
and on this machine at