Project

General

Profile

jobsub_q

Jobsub_q is an analog of condor_q for querying the job queue on the jobsub-server


$ ]$ jobsub_q --help

Usage: jobsub_q [Client Options]

Options:
  --version             show program's version number and exit

  Client Options:
    -G <Group/Experiment/Subgroup>, --group=<Group/Experiment/Subgroup>
                        Group/Experiment/Subgroup for priorities and
                        accounting
    --jobsub-server=<JobSub Server>
                        Alternate location of JobSub server to use
    --role=<VOMS Role>  VOMS Role for priorities and accounting
    --user=<user ID>    User Id  to query
    --jobid=<Job ID>    Job Ids to query.  Job Ids have format of
                        (cluster).(process)@(schedd_name). If Job id has a
                        decimal point but no process (example)
                        1234.@fifebatch.fnal.gov then ALL job ids with that
                        cluster and schedd name will be returned
    --long              show long listing (like condor_q -l)
    --dag               show dags (like condor_q -dag)
    --hold              show held jobs (like condor_q -hold)
    --run               show running jobs (like condor_q -run)
    --idle              show idle jobs
    --constraint=<constraint>
                        like condor_q -constraint <constraint>
    --summary           provide a summary (like ifront_q)
    --debug             Print debug messages to including server contacted,
                        http response, response time
    --better-analyze    do condor_q -better-analyze on job (must use with
                        --jobid)
    -h, --help          Show this help message and exit

Please direct questions, comments, or problems to the service desk. For help
on --jobid or --constraint see
https://cdcvs.fnal.gov/redmine/projects/jobsub/wiki/Frequently_Asked_Questions

Explanation of Job Statuses

The job statuses displayed by jobsub_q are pulled directly from the underlying batch system, HTCondor. The codes are:

Code Status Letter Code (shows up in jobsub_q)
0 Unexpanded U
1 Idle I
2 Running R
3 Removed X
4 Completed C
5 Held H
6 Transferring_Output >
7 Suspended S

These were pulled from the HTCondor Wiki.

The Unexpanded state means the job is being inserted into the system and should generally be brief. Jobs start in the Idle state and go to Running, but may go back to Idle if the job is interrupted. If the job does revert to Idle, it will eventually go back to Running again, and no action is required from the user. Transferring_output refers to transferring the condor-controlled files, not the user transfers with ifdh. Suspended is a rare state that fifebatch will not recover from, so the user should consider this a terminal failure.

The only status that requires a user to take action is Held (H). In this case, please consult the Fifemon Why are my jobs held? dashboard to determine the hold reason, and take appropriate action (most of the time, jobsub_release or jobsub_rm to release or remove the job).

Example: jobsub_q (no options)

Using jobsub_q without the --group option shows all the queued and running jobs on both servers. If there are a lot of queued jobs you might wait a while, so don't do this.

Instead, look at http://fifemon.fnal.gov/monitor/pool/fifebatchgpvmhead1


$ jobsub_q

JOBSUBJOBID                           OWNER           SUBMITTED     RUN_TIME   ST PRI SIZE CMD
390257.0@fifebatch2.fnal.gov          jbrodsky        12/19 16:18   0+00:00:00 I   0   0.0 gridrun_g4ds.sh_20141219_161854_11359_0_1_wrap.sh 
390258.0@fifebatch2.fnal.gov          jbrodsky        12/19 16:20   0+00:00:00 I   0   0.0 gridrun_g4ds.sh_20141219_162003_11599_0_1_wrap.sh 
2016928.0@fifebatch2.fnal.gov         kherner         06/09 05:39   0+00:00:00 I   0   0.0 submission_test.sh_20150609_053935_18329_0_1_wrap.sh 
2016928.1@fifebatch2.fnal.gov         kherner         06/09 05:39   0+00:00:00 I   0   0.0 submission_test.sh_20150609_053935_18329_0_1_wrap.sh 
2016928.2@fifebatch2.fnal.gov         kherner         06/09 05:39   0+00:00:00 I   0   0.0 submission_test.sh_20150609_053935_18329_0_1_wrap.sh 
2016928.3@fifebatch2.fnal.gov         kherner         06/09 05:39   0+00:00:00 I   0   0.0 submission_test.sh_20150609_053935_18329_0_1_wrap.sh 
2016928.4@fifebatch2.fnal.gov         kherner         06/09 05:39   0+00:00:00 I   0   0.0 submission_test.sh_20150609_053935_18329_0_1_wrap.sh 
2077003.0@fifebatch2.fnal.gov         kherner         06/14 20:26   0+00:00:00 I   0   0.0 submission_test.sh_20150614_202628_6331_0_1_wrap.sh 
2248016.0@fifebatch2.fnal.gov         aluca           06/19 09:10  98+05:15:05 R   0   0.3 condor_dagman                  

[ ~133000 rows deleted for brevity ]

3035805.9993@fifebatch1.fnal.gov      mu2epro         09/25 14:14   0+00:00:00 I   0   0.0 mu2eprodsys.sh_20150925_141422_15265_0_1_wrap.sh 
3035805.9994@fifebatch1.fnal.gov      mu2epro         09/25 14:14   0+00:00:00 I   0   0.0 mu2eprodsys.sh_20150925_141422_15265_0_1_wrap.sh 
3035805.9995@fifebatch1.fnal.gov      mu2epro         09/25 14:14   0+00:00:00 I   0   0.0 mu2eprodsys.sh_20150925_141422_15265_0_1_wrap.sh 
3035805.9996@fifebatch1.fnal.gov      mu2epro         09/25 14:14   0+00:00:00 I   0   0.0 mu2eprodsys.sh_20150925_141422_15265_0_1_wrap.sh 
3035805.9997@fifebatch1.fnal.gov      mu2epro         09/25 14:14   0+00:00:00 I   0   0.0 mu2eprodsys.sh_20150925_141422_15265_0_1_wrap.sh 
3035805.9998@fifebatch1.fnal.gov      mu2epro         09/25 14:14   0+00:00:00 I   0   0.0 mu2eprodsys.sh_20150925_141422_15265_0_1_wrap.sh 
3035805.9999@fifebatch1.fnal.gov      mu2epro         09/25 14:14   0+00:00:00 I   0   0.0 mu2eprodsys.sh_20150925_141422_15265_0_1_wrap.sh 
3035806.0@fifebatch1.fnal.gov         skemboi         09/25 14:14   0+00:00:00 I   0   0.0 meh_001.sh_20150925_141425_15728_0_1_wrap.sh 

133043 jobs; 0 completed, 1 removed, 101336 idle, 28641 running, 3062 held, 0 suspended

Example: jobsub_q --summary

If you want to know the state of the servers, a better option is jobsub_q --summary
It is still a lot of output, but it comes back much faster and is better organized by user/group

Again, look at http://fifemon.fnal.gov/monitor/pool/fifebatchgpvmhead1 instead

$ jobsub_q --summary
Name                                    Machine                            RunningJobs IdleJobs HeldJobs

group_annie.kherner@fifebatch1.fnal.gov fifebatch1.fnal.gov                          0        0        0
group_argoneut.kherner@fifebatch1.fnal.gov fifebatch1.fnal.gov                          0        1        0
group_cdf.adriutti@fifebatch1.fnal.gov     fifebatch1.fnal.gov                          0        0        0
group_cdf.aluca@fifebatch1.fnal.gov        fifebatch1.fnal.gov                          0        0        1
group_cdf.ctosciri@fifebatch1.fnal.gov     fifebatch1.fnal.gov                        100        0        0
group_cdf.dbox@fifebatch1.fnal.gov         fifebatch1.fnal.gov                          0        0        0
group_cdf.dfitz11@fifebatch1.fnal.gov      fifebatch1.fnal.gov                        298        0       15
group_cdf.kotwal@fifebatch1.fnal.gov       fifebatch1.fnal.gov                        163        0       26
group_cdf.ptl@fifebatch1.fnal.gov          fifebatch1.fnal.gov                          0        0        1
group_cdf.seiya@fifebatch1.fnal.gov        fifebatch1.fnal.gov                         50        0        0
group_cdf.sganguly@fifebatch1.fnal.gov     fifebatch1.fnal.gov                          0        0      146
group_darkside.aldenf@fifebatch1.fnal.gov  fifebatch1.fnal.gov                        232     1824        0
group_darkside.kherner@fifebatch1.fnal.gov fifebatch1.fnal.gov                          0        0        0
group_darkside.shawest@fifebatch1.fnal.gov fifebatch1.fnal.gov                          0        0        3
group_des.desgw@fifebatch1.fnal.gov        fifebatch1.fnal.gov                          0        0        0
group_des.kherner@fifebatch1.fnal.gov      fifebatch1.fnal.gov                          0        0        0
group_dzero.snow@fifebatch1.fnal.gov       fifebatch1.fnal.gov                          0        0        1
group_fermilab.kherner@fifebatch1.fnal.gov fifebatch1.fnal.gov                          0        1        0
group_gm2.kherner@fifebatch1.fnal.gov      fifebatch1.fnal.gov                          0        1        0
group_gm2.rfatemi@fifebatch1.fnal.gov      fifebatch1.fnal.gov                          0        1        0
group_lar1nd.kherner@fifebatch1.fnal.gov   fifebatch1.fnal.gov                          0        1        0
group_lariat.acciarri@fifebatch1.fnal.gov  fifebatch1.fnal.gov                          2        0        0
group_lariat.kherner@fifebatch1.fnal.gov   fifebatch1.fnal.gov                          0        1        0
group_lbne.achatter@fifebatch1.fnal.gov    fifebatch1.fnal.gov                         18        0        4
group_lbne.dani83@fifebatch1.fnal.gov      fifebatch1.fnal.gov                          0        0        0
group_lbne.dbrailsf@fifebatch1.fnal.gov    fifebatch1.fnal.gov                          1        0        0
group_lbne.kherner@fifebatch1.fnal.gov     fifebatch1.fnal.gov                          0        0        0
group_lbne.skemboi@fifebatch1.fnal.gov     fifebatch1.fnal.gov                          0        6        0
group_lbne.tianxc@fifebatch1.fnal.gov      fifebatch1.fnal.gov                          5        0        0
group_lbne.tjyang@fifebatch1.fnal.gov      fifebatch1.fnal.gov                          0        0        0
group_lbne.trj@fifebatch1.fnal.gov         fifebatch1.fnal.gov                          0        0        0
group_minerva.Production.minervapro@fifebatch1.fnal.gov fifebatch1.fnal.gov                          0        0        0
group_minerva.goran@fifebatch1.fnal.gov                 fifebatch1.fnal.gov                          0        0        1
group_minerva.kherner@fifebatch1.fnal.gov               fifebatch1.fnal.gov                          0        1        0
group_minerva.laliaga@fifebatch1.fnal.gov               fifebatch1.fnal.gov                          0        0        0
group_minerva.minervapro@fifebatch1.fnal.gov            fifebatch1.fnal.gov                          0        0        0
group_minerva.norrick@fifebatch1.fnal.gov               fifebatch1.fnal.gov                          0        0        0
group_minos.ashley90@fifebatch1.fnal.gov                fifebatch1.fnal.gov                        197        3        0
group_minos.kherner@fifebatch1.fnal.gov                 fifebatch1.fnal.gov                          0        0        0
group_minos.rchen2@fifebatch1.fnal.gov                  fifebatch1.fnal.gov                         34        0        0
group_mu2e.ehrlich@fifebatch1.fnal.gov                  fifebatch1.fnal.gov                        606      495        0
group_mu2e.gandr@fifebatch1.fnal.gov                    fifebatch1.fnal.gov                          0        0        0
group_mu2e.kherner@fifebatch1.fnal.gov                  fifebatch1.fnal.gov                          0        1        0
group_mu2e.mu2epro@fifebatch1.fnal.gov                  fifebatch1.fnal.gov                      12822    15354      928
group_mu2e.oksuzian@fifebatch1.fnal.gov                 fifebatch1.fnal.gov                         10      784        0
group_mu2e.rhbob@fifebatch1.fnal.gov                    fifebatch1.fnal.gov                          0        0      299
group_mu2e.rlc@fifebatch1.fnal.gov                      fifebatch1.fnal.gov                          0        5        0
group_nova.arrieta1@fifebatch1.fnal.gov                 fifebatch1.fnal.gov                          0        0        0
group_nova.biaow@fifebatch1.fnal.gov                    fifebatch1.fnal.gov                          0     1112        0
group_nova.brebel@fifebatch1.fnal.gov                   fifebatch1.fnal.gov                         27        0        0
group_nova.dbox@fifebatch1.fnal.gov                     fifebatch1.fnal.gov                          0        0        0
group_nova.jvasel@fifebatch1.fnal.gov                   fifebatch1.fnal.gov                          0        0        0
group_nova.kherner@fifebatch1.fnal.gov                  fifebatch1.fnal.gov                          0        0        0
group_nova.novapro@fifebatch1.fnal.gov                  fifebatch1.fnal.gov                       2851    31050        0
group_nova.prabhjot@fifebatch1.fnal.gov                 fifebatch1.fnal.gov                          0        0        0
group_nova.xbbu@fifebatch1.fnal.gov                     fifebatch1.fnal.gov                          0        0        0
group_numix.kherner@fifebatch1.fnal.gov                 fifebatch1.fnal.gov                          0        0        0
group_seaquest.kherner@fifebatch1.fnal.gov              fifebatch1.fnal.gov                          0        0        0
group_uboone.jyoti@fifebatch1.fnal.gov                  fifebatch1.fnal.gov                          0        0        0
group_uboone.kherner@fifebatch1.fnal.gov                fifebatch1.fnal.gov                          0        1        0
group_annie.kherner@fifebatch2.fnal.gov                 fifebatch2.fnal.gov                          0        2        0
group_argoneut.kherner@fifebatch2.fnal.gov              fifebatch2.fnal.gov                          0        2        0
group_cdf.adriutti@fifebatch2.fnal.gov                  fifebatch2.fnal.gov                          0        0        0
group_cdf.aluca@fifebatch2.fnal.gov                     fifebatch2.fnal.gov                          0        0        3
group_cdf.cjclarke@fifebatch2.fnal.gov                  fifebatch2.fnal.gov                          0        0        0
group_cdf.ctosciri@fifebatch2.fnal.gov                  fifebatch2.fnal.gov                         95        0        0
group_cdf.dbox@fifebatch2.fnal.gov                      fifebatch2.fnal.gov                          0        0        0
group_cdf.dfitz11@fifebatch2.fnal.gov                   fifebatch2.fnal.gov                          0        0        1
group_cdf.kotwal@fifebatch2.fnal.gov                    fifebatch2.fnal.gov                          0        0       12
group_cdf.marchese@fifebatch2.fnal.gov                  fifebatch2.fnal.gov                          0        0        0
group_cdf.sganguly@fifebatch2.fnal.gov                  fifebatch2.fnal.gov                          0        0        6
group_cdf.vaikunth@fifebatch2.fnal.gov                  fifebatch2.fnal.gov                          0        0        0
group_cdf.vellidis@fifebatch2.fnal.gov                  fifebatch2.fnal.gov                          0        1        0
group_cdf.whiteran@fifebatch2.fnal.gov                  fifebatch2.fnal.gov                          0        0        1
group_cdf.willis@fifebatch2.fnal.gov                    fifebatch2.fnal.gov                          0        1        0
group_coupp.orin@fifebatch2.fnal.gov                    fifebatch2.fnal.gov                          0        0        9
group_darkside.aldenf@fifebatch2.fnal.gov               fifebatch2.fnal.gov                          0        0        0
group_darkside.hqian36@fifebatch2.fnal.gov              fifebatch2.fnal.gov                          0        0        0
group_darkside.kherner@fifebatch2.fnal.gov              fifebatch2.fnal.gov                          0        5        0
group_darkside.lmarini@fifebatch2.fnal.gov              fifebatch2.fnal.gov                          0        0        1
group_darkside.masayuki@fifebatch2.fnal.gov             fifebatch2.fnal.gov                          0        0        4
group_darkside.pagnes28@fifebatch2.fnal.gov             fifebatch2.fnal.gov                          0        1        0
group_des.desgw@fifebatch2.fnal.gov                     fifebatch2.fnal.gov                          0        0        1
group_des.dtucker@fifebatch2.fnal.gov                   fifebatch2.fnal.gov                          0        0        1
group_des.kherner@fifebatch2.fnal.gov                   fifebatch2.fnal.gov                          0        0        0
group_des.yanny@fifebatch2.fnal.gov                     fifebatch2.fnal.gov                          0        0        0
group_dune.kherner@fifebatch2.fnal.gov                  fifebatch2.fnal.gov                          0        1        0
group_dune.php13tkw@fifebatch2.fnal.gov                 fifebatch2.fnal.gov                          0        0        0
group_dune.tjyang@fifebatch2.fnal.gov                   fifebatch2.fnal.gov                          0        1        0
group_dune.trj@fifebatch2.fnal.gov                      fifebatch2.fnal.gov                          0        1        0
group_fermilab.arossi@fifebatch2.fnal.gov               fifebatch2.fnal.gov                        209        0        0
group_fermilab.kherner@fifebatch2.fnal.gov              fifebatch2.fnal.gov                          0        3        0
group_fermilab.mengel@fifebatch2.fnal.gov               fifebatch2.fnal.gov                          0        0        0
group_gm2.fienberg@fifebatch2.fnal.gov                  fifebatch2.fnal.gov                          0        0        0
group_gm2.kherner@fifebatch2.fnal.gov                   fifebatch2.fnal.gov                          0        2        0
group_gm2.liangli@fifebatch2.fnal.gov                   fifebatch2.fnal.gov                          0        0        0
group_gm2.lwelty@fifebatch2.fnal.gov                    fifebatch2.fnal.gov                          9        0        0
group_gm2.nfroemm@fifebatch2.fnal.gov                   fifebatch2.fnal.gov                          0        0        0
group_gm2.schlesr2@fifebatch2.fnal.gov                  fifebatch2.fnal.gov                          6        1        0
group_gm2.sweigart@fifebatch2.fnal.gov                  fifebatch2.fnal.gov                          0        0        0
group_gm2.twalton@fifebatch2.fnal.gov                   fifebatch2.fnal.gov                         16        0        0
group_lar1.dbox@fifebatch2.fnal.gov                     fifebatch2.fnal.gov                          0        0        0
group_lar1nd.kherner@fifebatch2.fnal.gov                fifebatch2.fnal.gov                          0        2        0
group_lariat.acciarri@fifebatch2.fnal.gov               fifebatch2.fnal.gov                          0        0        0
group_lariat.gpulliam@fifebatch2.fnal.gov               fifebatch2.fnal.gov                          3        0        0
group_lariat.johnnyho@fifebatch2.fnal.gov               fifebatch2.fnal.gov                          0        0        0
group_lariat.kherner@fifebatch2.fnal.gov                fifebatch2.fnal.gov                          0        2        0
group_lariat.pkryczyn@fifebatch2.fnal.gov               fifebatch2.fnal.gov                          0        0        0
group_lariat.rbouabid@fifebatch2.fnal.gov               fifebatch2.fnal.gov                          0        0        0
group_lariat.soubasis@fifebatch2.fnal.gov               fifebatch2.fnal.gov                          0        0        0
group_lariat.tjyang@fifebatch2.fnal.gov                 fifebatch2.fnal.gov                          0        0        0
group_lariat.wforeman@fifebatch2.fnal.gov               fifebatch2.fnal.gov                          0        0        0
group_lbne.achatter@fifebatch2.fnal.gov                 fifebatch2.fnal.gov                          0        0        4
group_lbne.dani83@fifebatch2.fnal.gov                   fifebatch2.fnal.gov                       2498     9082        1
group_lbne.dbrailsf@fifebatch2.fnal.gov                 fifebatch2.fnal.gov                          0        0        0
group_lbne.gvsinev@fifebatch2.fnal.gov                  fifebatch2.fnal.gov                          0        0        0
group_lbne.jpdavies@fifebatch2.fnal.gov                 fifebatch2.fnal.gov                          0        0        2
group_lbne.kherner@fifebatch2.fnal.gov                  fifebatch2.fnal.gov                          0        0        0
group_lbne.knguyen@fifebatch2.fnal.gov                  fifebatch2.fnal.gov                          0        0        0
group_lbne.lebrun@fifebatch2.fnal.gov                   fifebatch2.fnal.gov                          0        0        0
group_lbne.ljf26@fifebatch2.fnal.gov                    fifebatch2.fnal.gov                          0        0        0
group_lbne.php13tkw@fifebatch2.fnal.gov                 fifebatch2.fnal.gov                          8        0        0
group_lbne.seturner@fifebatch2.fnal.gov                 fifebatch2.fnal.gov                        141        0        2
group_lbne.skemboi@fifebatch2.fnal.gov                  fifebatch2.fnal.gov                          0        1       50
group_lbne.tjyang@fifebatch2.fnal.gov                   fifebatch2.fnal.gov                          2        0        0
group_marsgm2.strigano@fifebatch2.fnal.gov              fifebatch2.fnal.gov                          0        0        0
group_marslbne.reitzner@fifebatch2.fnal.gov             fifebatch2.fnal.gov                          0        0        0
group_marslbne.strigano@fifebatch2.fnal.gov             fifebatch2.fnal.gov                          0        0        0
group_marsmu2e.vspron@fifebatch2.fnal.gov               fifebatch2.fnal.gov                          0        0        0
group_minerva.Keepup.minervadat@fifebatch2.fnal.gov     fifebatch2.fnal.gov                          0        0        0
group_minerva.Production.minervapro@fifebatch2.fnal.gov fifebatch2.fnal.gov                       1095       44     1237
group_minerva.betan009@fifebatch2.fnal.gov              fifebatch2.fnal.gov                        297       40        0
group_minerva.drimal@fifebatch2.fnal.gov                fifebatch2.fnal.gov                          0        0        0
group_minerva.drut1186@fifebatch2.fnal.gov              fifebatch2.fnal.gov                          0        0        0
group_minerva.evalen@fifebatch2.fnal.gov                fifebatch2.fnal.gov                          0        0        0
group_minerva.goran@fifebatch2.fnal.gov                 fifebatch2.fnal.gov                          0        0        4
group_minerva.kherner@fifebatch2.fnal.gov               fifebatch2.fnal.gov                          0        2        0
group_minerva.kwiley@fifebatch2.fnal.gov                fifebatch2.fnal.gov                          0        0        0
group_minerva.marshalc@fifebatch2.fnal.gov              fifebatch2.fnal.gov                          0        0        0
group_minerva.mateusc@fifebatch2.fnal.gov               fifebatch2.fnal.gov                          0        0       13
group_minerva.minervapro@fifebatch2.fnal.gov            fifebatch2.fnal.gov                          0        0        0
group_minerva.norrick@fifebatch2.fnal.gov               fifebatch2.fnal.gov                          0        0        0
group_minerva.nsteinbe@fifebatch2.fnal.gov              fifebatch2.fnal.gov                          0        0        0
group_minerva.oaltinok@fifebatch2.fnal.gov              fifebatch2.fnal.gov                          0        0        0
group_minerva.renlu23@fifebatch2.fnal.gov               fifebatch2.fnal.gov                          0        0        0
group_minerva.rgalindo@fifebatch2.fnal.gov              fifebatch2.fnal.gov                          0        0        0
group_minos.ashley90@fifebatch2.fnal.gov                fifebatch2.fnal.gov                          0        0        0
group_minos.aurisano@fifebatch2.fnal.gov                fifebatch2.fnal.gov                          0        0        0
group_minos.hyepesra@fifebatch2.fnal.gov                fifebatch2.fnal.gov                          0        0        0
group_minos.kherner@fifebatch2.fnal.gov                 fifebatch2.fnal.gov                          0        2        0
group_minos.kreymer@fifebatch2.fnal.gov                 fifebatch2.fnal.gov                          0        0        0
group_minos.minospro@fifebatch2.fnal.gov                fifebatch2.fnal.gov                          0        0        0
group_minos.mkiveni@fifebatch2.fnal.gov                 fifebatch2.fnal.gov                          2        0        0
group_minos.nav@fifebatch2.fnal.gov                     fifebatch2.fnal.gov                          0        0        0
group_minos.psail@fifebatch2.fnal.gov                   fifebatch2.fnal.gov                          0        0        0
group_minos.simon217@fifebatch2.fnal.gov                fifebatch2.fnal.gov                          0        0        0
group_minos.wingmc@fifebatch2.fnal.gov                  fifebatch2.fnal.gov                          0        0        0
group_mu2e.boyd@fifebatch2.fnal.gov                     fifebatch2.fnal.gov                          0        5        0
group_mu2e.brownd@fifebatch2.fnal.gov                   fifebatch2.fnal.gov                          0        0        0
group_mu2e.coleman@fifebatch2.fnal.gov                  fifebatch2.fnal.gov                          0        0        0
group_mu2e.ehrlich@fifebatch2.fnal.gov                  fifebatch2.fnal.gov                        245     3261        0
group_mu2e.gandr@fifebatch2.fnal.gov                    fifebatch2.fnal.gov                          0        0        0
group_mu2e.gianipez@fifebatch2.fnal.gov                 fifebatch2.fnal.gov                          0        0        0
group_mu2e.hedin@fifebatch2.fnal.gov                    fifebatch2.fnal.gov                          0        0        3
group_mu2e.kherner@fifebatch2.fnal.gov                  fifebatch2.fnal.gov                          0       17        0
group_mu2e.mu2epro@fifebatch2.fnal.gov                  fifebatch2.fnal.gov                          0        0        0
group_mu2e.murat@fifebatch2.fnal.gov                    fifebatch2.fnal.gov                          0        0        0
group_mu2e.oksuzian@fifebatch2.fnal.gov                 fifebatch2.fnal.gov                          0        0        1
group_mu2e.palladin@fifebatch2.fnal.gov                 fifebatch2.fnal.gov                          3     8098        2
group_mu2e.rhbob@fifebatch2.fnal.gov                    fifebatch2.fnal.gov                       1262    27234      234
group_mu2e.rlc@fifebatch2.fnal.gov                      fifebatch2.fnal.gov                          0        4        0
group_mu2e.roehrken@fifebatch2.fnal.gov                 fifebatch2.fnal.gov                        140       75        0
group_mu2e.zchen@fifebatch2.fnal.gov                    fifebatch2.fnal.gov                          0        0        0
group_nova.arrieta1@fifebatch2.fnal.gov                 fifebatch2.fnal.gov                          0        0        0
group_nova.aurisano@fifebatch2.fnal.gov                 fifebatch2.fnal.gov                          0        0        0
group_nova.biaow@fifebatch2.fnal.gov                    fifebatch2.fnal.gov                          0     1112        0
group_nova.bzamoran@fifebatch2.fnal.gov                 fifebatch2.fnal.gov                          0        0        0
group_nova.crisprin@fifebatch2.fnal.gov                 fifebatch2.fnal.gov                          1        0        0
group_nova.dbox@fifebatch2.fnal.gov                     fifebatch2.fnal.gov                          0        0        0
group_nova.dmendez@fifebatch2.fnal.gov                  fifebatch2.fnal.gov                          0        0        0
group_nova.jvasel@fifebatch2.fnal.gov                   fifebatch2.fnal.gov                          0        0        0
group_nova.kalra@fifebatch2.fnal.gov                    fifebatch2.fnal.gov                          0        0        0
group_nova.kherner@fifebatch2.fnal.gov                  fifebatch2.fnal.gov                          0      102        0
group_nova.kmatera@fifebatch2.fnal.gov                  fifebatch2.fnal.gov                          0        0        0
group_nova.kotelnik@fifebatch2.fnal.gov                 fifebatch2.fnal.gov                          0        0        0
group_nova.kuldeepm@fifebatch2.fnal.gov                 fifebatch2.fnal.gov                        147        0        0
group_nova.mylab@fifebatch2.fnal.gov                    fifebatch2.fnal.gov                          2        0        0
group_nova.novapro@fifebatch2.fnal.gov                  fifebatch2.fnal.gov                         90       69        0
group_nova.pavan219@fifebatch2.fnal.gov                 fifebatch2.fnal.gov                          0        0        1
group_nova.philipm1@fifebatch2.fnal.gov                 fifebatch2.fnal.gov                       1505     1255        3
group_nova.sedayath@fifebatch2.fnal.gov                 fifebatch2.fnal.gov                          0        0        0
group_nova.tamsett@fifebatch2.fnal.gov                  fifebatch2.fnal.gov                          0        0       22
group_nova.test.dbox@fifebatch2.fnal.gov                fifebatch2.fnal.gov                          0        0        0
group_nova.tjyoti@fifebatch2.fnal.gov                   fifebatch2.fnal.gov                        878        0        0
group_nova.xbbu@fifebatch2.fnal.gov                     fifebatch2.fnal.gov                        124        0        0
group_numix.cpgrant@fifebatch2.fnal.gov                 fifebatch2.fnal.gov                          0        0        0
group_numix.kherner@fifebatch2.fnal.gov                 fifebatch2.fnal.gov                          0        1        0
group_seaquest.bkerns@fifebatch2.fnal.gov               fifebatch2.fnal.gov                          1        0        0
group_seaquest.kherner@fifebatch2.fnal.gov              fifebatch2.fnal.gov                          0        0        0
group_seaquest.liuk@fifebatch2.fnal.gov                 fifebatch2.fnal.gov                          0        0        0
group_seaquest.production.dannowi1@fifebatch2.fnal.gov  fifebatch2.fnal.gov                          0        0        0
group_testjobs@fifebatch2.fnal.gov                      fifebatch2.fnal.gov                          0        2        0
group_uboone.aschu@fifebatch2.fnal.gov                  fifebatch2.fnal.gov                          0        0        0
group_uboone.bcarls@fifebatch2.fnal.gov                 fifebatch2.fnal.gov                          0        0        2
group_uboone.bkirby@fifebatch2.fnal.gov                 fifebatch2.fnal.gov                          0        0        0
group_uboone.cjen@fifebatch2.fnal.gov                   fifebatch2.fnal.gov                          0        0        0
group_uboone.greenlee@fifebatch2.fnal.gov               fifebatch2.fnal.gov                          0        0        0
group_uboone.jyoti@fifebatch2.fnal.gov                  fifebatch2.fnal.gov                          6        4        0
group_uboone.kalousis@fifebatch2.fnal.gov               fifebatch2.fnal.gov                          0        0        0
group_uboone.kherner@fifebatch2.fnal.gov                fifebatch2.fnal.gov                          0        2        0
group_uboone.kterao@fifebatch2.fnal.gov                 fifebatch2.fnal.gov                          0        0        0
group_uboone.ran@fifebatch2.fnal.gov                    fifebatch2.fnal.gov                          0        0        0
group_uboone.uboonepro@fifebatch2.fnal.gov              fifebatch2.fnal.gov                          0        0        0

 OWNER        RUN   IDLE   HELD     OLDEST_JOB
acciarri        2      0      0     9/25 14:00 0+00:25:08 slicer-Acciarri_Li
achatter       18      0      8     8/21 16:37 7+05:40:47 Nominal_test4_010.
aldenf        232   1824      0     9/25 02:17 0+05:44:45 run_darkart_011144
aluca           3      0      4     6/19 09:10 98+01:24:46 -                 
arossi        209      0      0     9/22 12:20 2+00:54:48 dcache-write-only.
ashley90      197      3      0     9/25 13:06 0+01:10:04 submit_FeldmanIFDH
bcarls          1      0      2     9/14 13:35 11+00:51:39 0                 
betan009      294     40      0     9/24 14:15 0+15:38:26 GaudiAnaStage_0001
biaow           0   2224      0     7/21 16:57 0+00:00:00 biaow-fn1_721-2015
bkerns          1      0      0     9/21 11:32 3+07:27:20 mc_drellyan_C_M011
boyd            0      5      0     9/16 10:04 0+00:00:00 sleep_job.sh_20150
brebel         27      0      0     9/23 13:33 2+00:32:11 brebel-AnalysisSki
crisprin        1      0      0     9/25 10:03 0+04:21:31 crisprin-crisprin_
ctosciri      196      0      0     9/22 19:14 2+19:11:08 0                 
dani83       2499   9080      1     9/24 14:23 0+17:33:25 Run_FastMC_job.sh_
dbrailsf        1      0      0     9/16 06:07 9+07:45:36 process_request.sh
desgw           1      0      1     9/24 09:29 1+04:55:24 0                 
dfitz11       301      0     16     9/17 22:29 7+15:57:10 0                 
dtucker         0      0      1     9/23 15:32 0+00:00:21 job_Nite20150722_E
ehrlich       849   3755      0     9/25 11:25 0+02:23:44 copyback.sh_201509
goran           0      0      5     8/11 16:48 0+01:59:18 GaudiAnaStage_0000
gpulliam        3      0      0     9/25 11:42 0+02:40:39 reco-Test-v01_09_0
hedin           0      0      3     8/25 11:48 0+17:34:01 copyback.sh_201508
jbrodsky        0      2      0     12/19 16:18 0+00:00:00 gridrun_g4ds.sh_20
jobs;           0      0      0     0 completed, 1 suspended         
jpdavies        0      0      2     9/4 09:37 5+12:59:35 FullSimulation.sh_
jyoti         884      4      0     9/24 12:16 1+02:08:59 run_flugg_grid.sh_
kherner         0    153      0     6/9 05:39 0+00:00:00 submission_test.sh
kotwal        180      0     38     7/6 14:48 80+20:32:31 -                 
kuldeepm      147      0      0     9/23 17:02 1+20:42:03 g4numi_job.sh_2015
lmarini         0      0      1     8/11 13:43 0+00:46:33 run_darkart_010981
lwelty          9      0      0     9/21 22:11 3+16:13:23 submit-localreleas
masayuki        0      0      4     8/29 15:48 0+06:00:03 run_od_11102_1.sh_
mateusc         0      0     13     7/20 15:54 0+01:00:05 GaudiAnaStage_0001
minervapro   3471     50   1234     9/23 18:27 1+19:59:23 0                 
mkiveni         2      0      0     9/24 16:31 0+21:43:00 catchup-s2s.sh_201
mu2epro     12875  15226    928     7/27 08:11 57+16:28:24 mu2eprodsys.sh_201
mylab           2      0      0     9/25 11:46 0+02:39:20 reduce_job_numu.sh
novapro      2941  31119      0     6/29 11:47 0+00:00:00 novapro-offsite-pc
oksuzian       10    784      1     9/24 19:55 0+06:59:31 copyback.sh_201509
orin            0      0      9     7/12 16:35 0+00:10:42 process_single_run
pagnes28        0      1      0     7/8 12:53 0+00:00:00 test.sh_20150708_1
palladin        3   8098      2     9/22 16:10 0+15:20:22 copyback.sh_201509
pavan219        0      0      1     9/11 06:36 0+00:00:44 pavan219-pavan_Cos
philipm1     1511   1249      3     6/24 12:00 0+11:56:28 philipm1-cosmicfor
php13tkw        8      0      0     9/24 15:00 0+23:26:37 gen-MUSUN_gen-v04_
ptl             1      0      1     9/3 05:20 22+06:35:19 -                 
rchen2         34      0      0     9/25 10:52 0+03:31:23 submit_systematics
rfatemi         0      1      0     6/8 14:40 0+00:00:00 submit-localreleas
rhbob        1259  27234    533     8/21 11:16 1+14:19:50 copyback.sh_201508
rlc             0      9      0     8/27 06:01 0+00:00:00 osgMonCheck.sh_201
roehrken      140     75      0     9/14 11:37 10+09:48:27 copyback.sh_201509
schlesr2        5      1      0     9/21 15:27 3+04:37:55 submit-localreleas
seiya          51      0      0     9/17 10:18 8+03:57:57 0                 
seturner      143      0      2     9/19 14:40 5+23:46:50 0                 
sganguly       11      0    152     9/7 14:02 18+00:24:26 0                 
shawest         0      0      3     8/11 01:26 0+00:09:39 run_od_8303_47.sh_
skemboi         0      7     50     9/24 15:50 0+00:00:21 kimoi_001.sh_20150
snow            0      0      1     7/6 11:34 0+00:37:06 test02.sh_20150706
tamsett         0      0     22     7/21 05:37 0+18:41:27 caf_ana_job.sh_201
tianxc          5      0      0     9/25 12:16 0+02:08:22 Run_GEVGEN_job.sh_
tjyang          2      1      0     9/24 11:04 1+03:00:01 reco-prodgenie_nue
tjyoti        878      0      0     9/24 12:16 1+02:08:59 run_flugg_grid.sh_
trj             0      1      0     9/25 12:31 0+00:00:00 bprobe2.sh_2015092
twalton        16      0      0     9/15 16:15 9+21:33:26 submit-release.sh_
vellidis        1      1      0     9/24 11:04 1+03:22:58 0                 
whiteran        1      0      1     7/28 16:42 58+20:21:22 -                 
willis          1      1      0     9/25 13:51 0+00:31:43 0                 
xbbu          124      0      0     9/23 17:07 1+20:30:58 g4numi_job.sh_2015

TOTALS      28672 100948   3042 

glidein count: 24044 currently servicing jobs, 576 unclaimed 

Example: narrow the query down with --user:

$ jobsub_q --user boyd
JOBSUBJOBID                           OWNER           SUBMITTED     RUN_TIME   ST PRI SIZE CMD
4157596.0@fifebatch2.fnal.gov         boyd            09/16 10:04   0+00:00:00 I   0   0.0 sleep_job.sh_20150916_100421_18859_0_1_wrap.sh 
4157596.1@fifebatch2.fnal.gov         boyd            09/16 10:04   0+00:00:00 I   0   0.0 sleep_job.sh_20150916_100421_18859_0_1_wrap.sh 
4157596.2@fifebatch2.fnal.gov         boyd            09/16 10:04   0+00:00:00 I   0   0.0 sleep_job.sh_20150916_100421_18859_0_1_wrap.sh 
4157596.3@fifebatch2.fnal.gov         boyd            09/16 10:04   0+00:00:00 I   0   0.0 sleep_job.sh_20150916_100421_18859_0_1_wrap.sh 
4157596.4@fifebatch2.fnal.gov         boyd            09/16 10:04   0+00:00:00 I   0   0.0 sleep_job.sh_20150916_100421_18859_0_1_wrap.sh 

5 jobs; 0 completed, 0 removed, 5 idle, 0 running, 0 held, 0 suspended

Example: Narrow the query down with --group:

$ jobsub_q --group seaquest
JOBSUBJOBID                           OWNER           SUBMITTED     RUN_TIME   ST PRI SIZE CMD
4255478.0@fifebatch2.fnal.gov         bkerns          09/21 11:32   3+07:22:31 R   0  293.0 mc_drellyan_C_M011_S012_0203.sh_20150921_113250_30170_0_1_wrap.sh 

1 jobs; 0 completed, 0 removed, 0 idle, 1 running, 0 held, 0 suspended

Example: Querying the state of a previously submitted job:

On the jobsub_submit page, there is an example of the user submitting a job with jobid . Is it running?

$ jobsub_q -G nova --jobid 269.0@fifebatch2.fnal.gov --debug
JOBSUBJOBID                     OWNER     SUBMITTED     RUN_TIME        ST     PRI      SIZE               CMD
269.0@fifebatch2.fnal.gov    dbox    06/09    14:47    0+00:00:00    I    0    0.0    nova.sh_20140609_144728_12037_0_1_wrap.sh

JOBSUB SERVER CONTACTED     : https://fifebatch.fnal.gov:8443
JOBSUB SERVER RESPONDED     : https://fifebatch2.fnal.gov:8443
JOBSUB SERVER RESPONSE CODE : 200 (Success)
JOBSUB SERVER SERVICED IN   : 1.4852669239 sec
JOBSUB CLIENT FQDN          : fermicloud326.fnal.gov
JOBSUB CLIENT SERVICED TIME : 21/Apr/2015 14:39:04

Its still idle

Example: Determining who submitted using a group account (and from where)

Sometimes a job is submitted using a group account, for example the 'novapro' account for Nova Production jobs. Users with access to such accounts may be curious as to who submitted the jobs.

Lets submit a test job from this account for illustration purposes:

[dbox@fermicloud326 client]$ jobsub_submit -G nova --role Production --jobsub-server https://fifebatch.fnal.gov:8443 file://simple_worker_script.sh 10
/fife/local/scratch/uploads/nova/novapro/2015-04-21_143903.633613_8139

/fife/local/scratch/uploads/nova/novapro/2015-04-21_143903.633613_8139/@simple_worker_script.sh_20150421_143904_7820_0_1.cmd

submitting....

Submitting job(s).

1 job(s) submitted to cluster 1508538.

JobsubJobId of first job: 1508538.0@fifebatch1.fnal.gov

Use job id 1508538.0@fifebatch1.fnal.gov to retrieve output

Verify the job is in the queue under user 'novapro' using jobsub_q:


[dbox@fermicloud326jobsub_q -G nova --jobsub-server https://fifebatch.fnal.gov:8443 --jobid 1508538.0@fifebatch1.fnal.gov --debug
JOBSUBJOBID                           USER          SUBMITTED     RUN_TIME   ST PRI SIZE CMD
1508538.0@fifebatch1.fnal.gov        novapro        04/21 14:39   0+00:00:00 I  0   0.0  @simple_worker_script.sh_20150421_143904_7820_0_1_wrap.sh 

JOBSUB SERVER CONTACTED     : https://fifebatch.fnal.gov:8443
JOBSUB SERVER RESPONDED     : https://fifebatch1.fnal.gov:8443
JOBSUB SERVER RESPONSE CODE : 200 (Success)
JOBSUB SERVER SERVICED IN   : 2.27557110786 sec
JOBSUB CLIENT FQDN          : fermicloud326.fnal.gov
JOBSUB CLIENT SERVICED TIME : 21/Apr/2015 14:40:37
[dbox@fermicloud326 client]$ 

Use the --long option with grep to see the DN of the user that submitted and the IP of the machine he submitted from:

$jobsub_q -G nova --jobsub-server https://fifebatch.fnal.gov:8443 --jobid 1508538.0@fifebatch1.fnal.gov --long | grep -i '^JOBSUB'
JobsubClientDN = "/DC=gov/DC=fnal/O=Fermilab/OU=People/CN=Dennis D. Box/CN=UID:dbox" 
JobsubServerVersion = "1.1.1.1" 
JobsubClientIpAddress = "131.225.154.204" 
$ 

Example: using --better-analyze to get more information on why a job hasn't started


$ jobsub_q --user trj
JOBSUBJOBID                           OWNER           SUBMITTED     RUN_TIME   ST PRI SIZE CMD
4352957.0@fifebatch2.fnal.gov         trj             09/25 12:31   0+00:00:00 I   0   0.0 bprobe2.sh_20150925_123158_30683_0_1_wrap.sh 

1 jobs; 0 completed, 0 removed, 1 idle, 0 running, 0 held, 0 suspended
$ jobsub_q --better-analyze --jobid 4352957.0@fifebatch2.fnal.gov

-- Schedd: fifebatch2.fnal.gov : <131.225.67.139:9615?addrs=131.225.67.139-9615&noUDP&sock=13611_1434>
User priority for trj@fifebatch2.fnal.gov is not available, attempting to analyze without it.
---
4352957.000:  Run analysis summary.  Of 25104 machines,
  20445 are rejected by your job's requirements 
   4659 reject your job because of their own requirements 
      0 match and are already running your jobs 
      0 match but are serving other users 
      0 are available to run your job
    No successful match recorded.
    Last failed match: Fri Sep 25 14:49:01 2015

    Reason for last match failure: no match found 

The Requirements expression for your job is:

    ( ( ( Arch == "X86_64" ) || ( Arch == "INTEL" ) ) &&
      ( target.IS_Glidein == true ) && ( DesiredOS is NULL ||
        stringlistimember(Target.IFOS_installed,DesiredOS) ) &&
      ( stringListsIntersect(toUpper(target.HAS_usage_model),toUpper(my.DESIRED_usage_model)) ) ) &&
    ( TARGET.OpSys == "LINUX" ) && ( TARGET.Disk >= RequestDisk ) &&
    ( TARGET.Memory >= RequestMemory ) && ( TARGET.Cpus >= RequestCpus ) &&
    ( TARGET.HasFileTransfer )

Your job defines the following attributes:

    DESIRED_usage_model = "OPPORTUNISTIC" 
    DesiredOS = "SL6" 
    JobUniverse = 5
    RequestCpus = 1
    RequestDisk = 35000000
    RequestMemory = 2000

The Requirements expression for your job reduces to these conditions:

         Slots
Step    Matched  Condition
-----  --------  ---------
[0]       25104  Arch == "X86_64" 
[3]       25104  target.IS_Glidein == true
[6]       25104  stringlistimember(Target.IFOS_installed,DesiredOS)
[9]        4930  stringListsIntersect(toUpper(target.HAS_usage_model),toUpper(my.DESIRED_usage_model))
[13]      24552  TARGET.Disk >= RequestDisk
[14]       4659  [9] && [13]

Suggestions:

    Condition                         Machines Matched    Suggestion
    ---------                         ----------------    ----------
1   ( ( Arch == "X86_64" ) || ( Arch == "INTEL" ) )
                                      0                   REMOVE
2   ( stringListsIntersect(toUpper(target.HAS_usage_model),"OPPORTUNISTIC") )
                                      4930                 
3   ( TARGET.Disk >= 35000000 )       24552                
4   ( TARGET.Memory >= 2000 )         24987                
5   ( target.IS_Glidein == true )     25104                
6   ( "SL6" is NULL || stringlistimember(Target.IFOS_installed,"SL6") )
                                      25104                
7   ( TARGET.OpSys == "LINUX" )       25104                
8   ( TARGET.Cpus >= 1 )              25104                
9   ( TARGET.HasFileTransfer )        25104                

-- Schedd: fifebatch1.fnal.gov : <131.225.67.102:9615?addrs=131.225.67.102-9615&noUDP&sock=27353_cc3f>

$