Jobsub command reference¶
To submit jobs with jobsub you need to:
- make sure you are registered with a Fermilab experiment
- setup your grid proxy initially
--version show program's version number and exit
-h, --help show this help message and exit
--OS=OS specify OS version of worker node. Default is SL5,
and SL6 is supported. Comma seperated list '--
OS=SL4,SL5,SL6' works as well
-G ACCOUNTINGGROUP, --group=ACCOUNTINGGROUP
Group/Experiment/Subgroup for priorities and
-v, --verbose dump internal state of program (useful for debugging)
-M, --mail_always send mail when job completes or fails
send mail only when job fails due to error (default)
-Q, --mail_never never send mail (default is to send mail on error)
-T, --test_queue Submit as a test job. Job will run with highest
possible priority, but you can only have one such
job in the queue at a time.
-g, --grid run job on the FNAL GP grid. Other flags can modify
target sites to include other areas of the Open
--nowrapfile do not generate shell wrapper for fermigrid
operations. (default is to generate a wrapfile)
-c APPEND_REQUIREMENTS, --append_condor_requirements=APPEND_REQUIREMENTS
append condor requirements
overwrite default condor requirements with supplied
override some other value: --override 'requirements'
'gack==TRUE' would produce the same condor command
file as --overwrite_condor_requirements 'gack==TRUE'
if you want to use this option, test it first with -n
to see what you get as output
-C execute on grid from directory you are currently in
-e ADDED_ENVIRONMENT, --environment=ADDED_ENVIRONMENT
-e ADDED_ENVIRONMENT exports this variable and its
local value to worker node environment. For example
export FOO="BAR"; jobsub -e FOO <more stuff>
guarantees that the value of $FOO on the worker node
is "BAR" . Can use this option as many times as
submit to different host
--site=SITE submit jobs to this site
-p use parrot to run on afs (only makes sense with -a
--pOff turn parrot off explicitly (this is the default)
-n, --no_submit generate condor_command file but do not submit
--opportunistic submit opportunistically to Fermigrid GP Grid and CDF
Grid. This option will allow you to potentially get
more slots than your Fermigrid quota, but these slots
are subject to preemption
-N QUEUECOUNT submit N copies of this job. Each job will
have access to the environment variable
$PROCESS that provides the job number (0 to
<num>-1), equivalent to the decimal point in
the job ID (the '2' in 134567.2).
-x X509_USER_PROXY, --X509_USER_PROXY=X509_USER_PROXY
location of X509_USER_PROXY (expert mode)
-l LINES, --lines=LINES
[Expert option] Add the line <line> to the
Condor submission (.cmd) file. See Condor
help for more.
-a, --needafs run on afs using parrot (this is discouraged)
-L JOBLOGFILE, --log_file=JOBLOGFILE
Log file to hold log output from job.
--no_log_buffer write log file directly to disk. Default is to copy it
back after job is completed. This option is useful
for debugging but can be VERY DANGEROUS as joblogfile
typically is sent to bluearc. Using this option
incorrectly can cause all grid submission systems at
FNAL to become overwhelmed resulting in angry admins
hunting you down, so USE SPARINGLY.
--use_gftp use grid-ftp to transfer file back
create self extracting tarball from contents of
INPUT_TAR_DIR. This tarball will be run on the worker
node with arguments you give to your_script
name of tarball to submit, created from
--input_tar_dir if specified
overwrite TAR_FILE_NAME when creating tarfile using
-f INPUT_DIR_ARRAY -f <file> input file <file> will be copied to
$CONDOR_DIR_INPUT on the execution node.
Example :-f /grid/data/minerva/my/input/file.xxx
will be copied to $CONDOR_DIR_INPUT/file.xxx
Specify as many -f file1 -f file2 args as you need.
-d<tag> <dir> Writable directory $CONDOR_DIR_<tag>
exist on the execution node. After job completion,
its contents will be moved to <dir> automatically
Specify as many <tag>/<dir> pairs as you need.
SAM dataset definition used in a Directed Acyclic
optional project name for SAM DAG
Nova Specific Options
--SMU steer jobs to HPC.SMU grid site¶
-i RELDIR release_directory for Nova Software
-t TESTRELDIR release_directory for test Nova Software
-r REL release_version for Nova Software
You can have as many instances of -c, -d, -e, -f, -l and -y as you need.
The -d directory mapping works on non-Grid nodes, too.
export IFDH_VERSION=some_version then use -e IFDH_VERSION to use
the (some_version) release of ifdh for copying files in and out
with -d and -f flags instead of the current ifdh version in KITS
More documentation is available at
and on this machine at