Project

General

Profile

Feature #11652

Combine jobsub submit log lines for a job into one

Added by Kevin Retzke over 3 years ago. Updated over 3 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
-
Target version:
Start date:
02/05/2016
Due date:
% Done:

0%

Estimated time:
Stakeholders:
Duration:

Description

Currently each jobsub_submit command will produce two lines in the submit log, one with request info and a second with the result:

2016-02-05 13:05:35,420 [INFO]  [140046694758144:jobsub.py:execute_job_submit_wrapper] 131.225.67.36 fbradasc@FNAL.GOV fbradasc 1.1.9.1  jobsub_submit  --group mu2e  --disk 30GB --resource-provides usage_model=OPPORTUNISTIC,DEDICATED --expected-lifetime 24h --memory 2048MB --OS SL6 -e EXPERIMENT -e MU2EGRID_NCLUSTERJOBS -e MU2EGRID_MU2ESETUP -e IFDH_VERSION -e MU2EGRID_EVENTS_PER_JOB -e MU2EGRID_MASTERFHICL -e MU2EGRID_FCLINPUT_NUMENTRIES -e MU2EGRID_DIR -e MU2EGRID_OUTSTAGE -e MU2EGRID_USERSETUP -e MU2EGRID_USERSERVICES -e MU2EGRID_OUTDIRFMT -e MU2EGRID_SUBMITTER -f stoppedMuonsSingleStage.fcl -N 100 copyback.sh mu2eart.sh 
2016-02-05 13:05:35,420 [INFO]  [140046694758144:jobsub.py:execute_job_submit_wrapper] JobsubJobId of first job: 7232763.0@fifebatch2.fnal.gov

We've started testing sending the submit logs into Elastsearch via Logstash to allow easier lookup and correlation with other job info (e.g. https://fifemon.fnal.gov/monitor-pp/dashboard/db/job-cluster-summary?from=1454664132713&to=1454707332713&var-cluster=4293587), and while nominally it can handle merging multiple log lines, it has been having some trouble in practice.

It would be really helpful if the log entry lines could be merged into one.

History

#1 Updated by Kevin Retzke over 3 years ago

Alternatively, if a unique ID could be included in the related log entries that would help us correlate them.

#2 Updated by Dennis Box over 3 years ago

  • Target version set to v1.2.1

#3 Updated by Dennis Box over 3 years ago

  • Target version changed from v1.2.1 to v1.2

#4 Updated by Dennis Box over 3 years ago

  • Status changed from New to Resolved

#5 Updated by Dennis Box over 3 years ago

  • Status changed from Resolved to Closed


Also available in: Atom PDF