Information about job submission to OSG sites » History » Version 65
Information about job submission to OSG sites¶
This page captures some of the known quirks about certain sites when submitting jobs there.
What this page is¶
Most OSG sites will work with with the jobsub default requests of 2000 MB of RAM and 35 GB of disk, but at some sites there are some more strict limits. Additionally some sites only support certain experiments as opposed to the entire Fermilab VO. Here we list the OSG sites where users can submit jobs, along with all known cases where either the standard jobsub defaults may not work, or the site only supports certain experiments. Information on it is provided on a best-effort basis and is subject to change without notice.
What this page is NOT¶
This page is NOT a status board or health monitor of the OSG sites. Just because your submission fits in with the guidelines here does not mean that your job will start quickly. Nor does it keep track of downtimes at the remote sites. Its sole purpose is to help you avoid submitting jobs with disk/memory/cpu/site combinations that will never work. Limited offsite monitoring is available from https://fifemon.fnal.gov:3000/dashboard/db/offsite-monitoring
The following table lists the available OSG sites, their Glidein_site name (what you should put in the --site option), what experiment(s) the site will support, and finally and known limitations on disk, memory, or CPU.
NOTE: In many cases you may be able to request more than the jobsub defaults and be fine. We have not done detailed testing at each site to determine what the real limits are. If you do try a site and put in requirements that exceed the jobsub defaults, sometimes a
jobsub_q --better-analyze --jobid=<your job id>
will give you useful information about why a job doesn't start to run
(i.e. it may recommend lowering the disk or memory requirements to a certain value.)
NOTE 2: Under supported experiments, "All" means all experiments except for CDF, D0, and LSST. It does include DES and DUNE.
NOTE 3: The estimated maximum lifetim is just an estimate based on a periodic sampling of glidein lifetimes. It may change from time to time and it does NOT take into account any walltime limitations of the local job queues at the site itself. It also does not guarantee that there are resources available at any given moment to start a job with the longest possible lifetime.
|Site Name||--site option (sorted)||Supported Experiments||Known limitations||Estimated maximum job lifetime|
|Brookhaven National Laboratory||BNL||All||jobsub defaults are OK||24 h|
|Caltech T2||Caltech||All||jobsub defaults are OK; can go up to --memory=3000||25 h|
|Clemson||Clemson||All||jobsub defaults are OK||24 h|
|Cornell||Cornell||All||jobsub defaults are OK||unknown|
|GPGrid||FNAL||All+CDF+LSST+DES+D0||new GPGrid accessed via FNAL Feb 2016, but job could end up on CMS T1||unknown|
|Fermi private cloud||All|| memory up to 7500
|FNAL CMS Tier 1 + GPGrid||FNAL||All||jobsub defaults are OK||48 h|
|Czech Academy of Sciences||FZU||NOvA only||request --disk=20000MB or less||unknown|
|Harvard||Harvard||NOvA only||jobsub defaults are OK; SL5 only||unknown|
|University of Washington||Hyak_CE||All||available resources vary widely; --memory=1900MB or less is better||3.5 h|
|University of Manchester||Manchester||uboone only||can go up to --memory=8000 via multicore; disk default is OK||between 12-24 h|
|ATLAS Great Lakes Tier 2 (AGLT2)||Michigan||All||jobsub defaults are OK; can go up to --memory=2500MB||unknown but at least 6 h|
|MIT||MIT||All + CDF|| opportunistic access currently disabled
jobsub defaults are OK
|Midwest Tier2||MWT2||All|| jobsub defaults are OK
single core jobs will take a very long time to run if requesting more than 1920 MB of memory.
Running custom 3.x linux kernels, probably need to set UPS_OVERRIDE to setup products.
|Red||Nebraska||All||jobsub defaults are OK||48 h|
|Notre Dame||NotreDame||All||jobsub defaults are OK; can go up to --memory=2500 ; aim for short jobs due to preemption||24 h|
|Tusker/Crane||Omaha||All||jobsub defaults are OK; can go up to --memory=3000||24 h|
|Ohio Supercomputing Center||OSC||NOvA only||jobsub defaults are OK||48 h|
|Southern Methodist University||SMU||NOvA only||jobsub defaults are OK; SL5 only||unknown|
|Southern Methodist||SMU_HPC||NOvA only||single core jobs should request --memory=2500 or less||24 h|
|Syracuse||SU-OG||All|| request --disk=9000MB or less and --memory=2500MB or less
2015/10/15 libXpm.so not installed (may be required by ROOT) on all nodes
|Texas Tech||TTU||All but mu2epro and seaquest|| jobsub defaults are OK
2015/11/20 down since OSG software upgrade
|University of Chicago||UChicago||All||linked with MWT2; recommend --memory=1920 or less for single-core jobs ; be sure to set UPS_OVERRIDE environment variable||5 h|
|University of California, San Diego||UCSD||All||jobsub defaults are OK; can go up to --memory=4000||13 h|
|University of Bern||UNIBE-LHEP||uboone only||2000 MB memory or less; use >1 cpu for more memory||48 h|
|Grid Lab of Wisconsin (GLOW)||Wisconsin||All||jobsub defaults are OK||24 h|
|Western Tier2 (SLAC)||WT2||uboone only||jobsub defaults are OK; can go up to --memory=2500MB||10 days|