Project

General

Profile

Check submission status

To check job submission status the command is:

Project.py --check --campaign ${USER}_<UserCampaign>

An output example of submission status check id the following:

$ Project.py --check  --submission_id 468909
['468909']

##########################

Jobs Summary:
POMS_SUBMISSION_ID: 468909
     CAMPAIGN_NAME: vito_UserAnalysis
        STAGE_NAME: overlay_training_trajcluster
       SAM_PROJECT: vito-fife_wrap_20190625_152358_2667171
             owner: vito
        experiment: uboone
            job_id: 20885625@jobsub02.fnal.gov
              done: False
            failed: 0
           running: 4
              idle: 1
         completed: 1
         cancelled: 0
              held: 0
       logs_url: https://landscape.fnal.gov/lens/view/job/20885625.0@jobsub02.fnal.gov/

File Summary:
Input:
       sam_project: vito-fife_wrap_20190625_152358_2667171
  dataset_def_name: "overlay_v12_training" 
 files_in_snapshot: 4000
    project_status: "running" 
project_start_time: "2019-06-25T20:25:01.921249+00:00" 
  project_end_time: "-" 
file_counts: 
            consumed: 0
              failed: 0
         transferred: 5
process_counts: 
              active: 5
   sam_project_url: http://samweb.fnal.gov:8480/station_monitor/uboone/stations/uboone/projects/vito-fife_wrap_20190625_152358_2667171

Output:
*** No Output files

It is possibly to use a list of submission ids':

Project.py --check --submission_id <SubmissionId1> <SubmissionId2> <SubmissionId3> [...]

$ Project.py --check  --submission_id 468909 468913
['468909', '468913']

##########################

Jobs Summary:
POMS_SUBMISSION_ID: 468909
     CAMPAIGN_NAME: vito_UserAnalysis
        STAGE_NAME: overlay_training_trajcluster
       SAM_PROJECT: vito-fife_wrap_20190625_152358_2667171
             owner: vito
        experiment: uboone
            job_id: 20885625@jobsub02.fnal.gov
              done: False
            failed: 0
           running: 5
              idle: 0
         completed: 1
         cancelled: 0
              held: 0
       logs_url: https://landscape.fnal.gov/lens/view/job/20885625.0@jobsub02.fnal.gov/

File Summary:
Input:
       sam_project: vito-fife_wrap_20190625_152358_2667171
  dataset_def_name: "overlay_v12_training" 
 files_in_snapshot: 4000
    project_status: "running" 
project_start_time: "2019-06-25T20:25:01.921249+00:00" 
  project_end_time: "-" 
file_counts: 
            consumed: 4
              failed: 0
         transferred: 5
process_counts: 
              active: 5
   sam_project_url: http://samweb.fnal.gov:8480/station_monitor/uboone/stations/uboone/projects/vito-fife_wrap_20190625_152358_2667171

Output:
*** No Output files

##########################

Jobs Summary:
POMS_SUBMISSION_ID: 468913
     CAMPAIGN_NAME: vito_UserAnalysis
        STAGE_NAME: overlay_testing_trajcluster
       SAM_PROJECT: vito-fife_wrap_20190625_154520_4045721
             owner: vito
        experiment: uboone
            job_id: 20066927@jobsub01.fnal.gov
              done: False
            failed: 0
           running: 3
              idle: 2
         completed: 0
         cancelled: 0
              held: 0
       logs_url: https://landscape.fnal.gov/lens/view/job/20066927.0@jobsub01.fnal.gov/

File Summary:
Input:
       sam_project: vito-fife_wrap_20190625_154520_4045721
  dataset_def_name: "overlay_v12_testing" 
 files_in_snapshot: 4000
    project_status: "running" 
project_start_time: "2019-06-25T20:46:39.913098+00:00" 
  project_end_time: "-" 
file_counts: 
            consumed: 0
              failed: 0
         transferred: 3
process_counts: 
              active: 3
   sam_project_url: http://samweb.fnal.gov:8480/station_monitor/uboone/stations/uboone/projects/vito-fife_wrap_20190625_154520_4045721

Output:
*** No Output files

In case there are failed or held jobs, --failure_info and --hold_info options provides additional info:

Project.py --check --campaign ${USER}_<UserCampaign> --submission_id <SubmissionId> [--failure_info] [--hold_info]

An output example for failed jobs is the following:

$ Project.py --check  --submission_id 467817 --failure_info 
['467817']

##########################

Jobs Summary:
POMS_SUBMISSION_ID: 467817
     CAMPAIGN_NAME: mazzacan_rachele_test
        STAGE_NAME: CC1u2P_onbeam
       SAM_PROJECT: mazzacan-fife_wrap_20190618_172425_2087686
             owner: mazzacan
        experiment: uboone
            job_id: 19836889@jobsub01.fnal.gov
              done: True
            failed: 4
           running: 0
              idle: 0
         completed: 7
         cancelled: 0
              held: 0
       logs_url: https://landscape.fnal.gov/lens/view/job/19836889.0@jobsub01.fnal.gov/

  Job failure info: -- (this can take a while) --
                    exitCode           clusterID            log URL
                       9     19837095.0@jobsub01.fnal.gov   https://landscape.fnal.gov/lens/view/job/19837095.0@jobsub01.fnal.gov/
                       9     19837097.0@jobsub01.fnal.gov   https://landscape.fnal.gov/lens/view/job/19837097.0@jobsub01.fnal.gov/
                       9     19837098.0@jobsub01.fnal.gov   https://landscape.fnal.gov/lens/view/job/19837098.0@jobsub01.fnal.gov/
                       9     19837100.0@jobsub01.fnal.gov   https://landscape.fnal.gov/lens/view/job/19837100.0@jobsub01.fnal.gov/

File Summary:
Input:
       sam_project: mazzacan-fife_wrap_20190618_172425_2087686
  dataset_def_name: "prod_reco_optfilter_bnb_ccinclusive_v15_mcc8_goodruns2_v4s" 
 files_in_snapshot: 6418
    project_status: "ended incomplete" 
project_start_time: "2019-06-18T22:26:06.099523+00:00" 
  project_end_time: "2019-06-18T22:32:29.088540+00:00" 
file_counts: 
            consumed: 0
              failed: 0
process_counts: 
                 bad: 4
   sam_project_url: http://samweb.fnal.gov:8480/station_monitor/uboone/stations/uboone/projects/mazzacan-fife_wrap_20190618_172425_2087686

Output:
*** No Output files

It is possible to check jobs for all stages in a given POMS Campaign using the --campaign

Project.py --check --campaign ${USER}_<UserCampaign>