Project

General

Profile

Support #18715

corrupt module summaries in art v2_09_04

Added by Raymond Culbertson over 2 years ago. Updated over 2 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
Infrastructure
Target version:
-
Start date:
01/10/2018
Due date:
% Done:

100%

Estimated time:
Spent time:
Scope:
Internal
Experiment:
Mu2e
SSI Package:
art
Duration:

Description

I think the module and path summaries may be
corrupt in art v2_09_04 - the module
summaries appear to be overwriting the path summary.
An example follows.

  • previous result in art v2_07_03

TrigReport -------End-Path Summary ------------
TrigReport Trig Bit# Run Success Error Name
TrigReport 0 0 10 10 0 end_path

  • current result in art v2_09_04

TrigReport -------End-Path Summary ------------
TrigReport Trig Bit# Run Success Error Name
TrigReport 1 0 10 10 0 0 CaloShowerStepROFromShowerStep
TrigReport 1 0 10 10 0 0 FSHPreStereo
TrigReport 1 0 10 10 0 0 MakeStrawHitPositions
TrigReport 10 10 10 0 0 CaloDigiFromShower

To reproduce, on any machine with /cvmfs/mu2e.opensciencegrid.org,

source /cvmfs/mu2e.opensciencegrid.org/setupmu2e-art.sh
git clone http://cdcvs.fnal.gov/projects/mu2eofflinesoftwaremu2eoffline/Offline.git
cd Offline
git checkout -b temp XYZ
source setup.sh
scons -j 20 >& build.log
mu2e -n 10 -c Analyses/test/genReco.fcl

with XYZ being a commit:
art v4_09_04, with apparent corrupt output: 4531590624
art v2_07_03, with expected output: a633320085

History

#1 Updated by Kyle Knoepfel over 2 years ago

  • Status changed from New to Assigned
  • Assignee set to Kyle Knoepfel

I have been able to reproduce this error. Investigating.

#2 Updated by Kyle Knoepfel over 2 years ago

  • Status changed from Assigned to Feedback

The problem is understood. It is not an art bug, per se, but it is an unanticipated consequence of a change that was made with art 2.08. With art 2.08, the severity level of the logged end-of-job summary changed from "severe" to "warning". This change was necessary to facilitate the routing of MT-diagnostic information to a separate file. However, this change had the unexpected effect of making the summary susceptible to "limiting" of message-logging.

If I dump out the fully-processed configuration, I get the following:

[knoepfel@woof Offline]$ art -c Analyses/test/genReco.fcl --debug-config=processed_genReco.fcl --annotate
** Config output to file 'processed_genReco.fcl' **
Art has completed and will exit with status 1.

[knoepfel@woof Offline]$ grep -A29 "message: " processed_genReco.fcl 
   message: {  # ./Analyses/test/genReco.fcl:128
      destinations: {  # /scratch/knoepfel/mu2e/Offline/fcl/messageService.fcl:78
         log: {  # /scratch/knoepfel/mu2e/Offline/fcl/messageService.fcl:79
            categories: {  # /scratch/knoepfel/mu2e/Offline/fcl/standardMessageDestinations.fcl:111
               ArtReport: {  # /scratch/knoepfel/mu2e/Offline/fcl/standardMessageDestinations.fcl:115
                  limit: -1  # /scratch/knoepfel/mu2e/Offline/fcl/standardMessageDestinations.fcl:118
                  reportEvery: 50  # /scratch/knoepfel/mu2e/Offline/fcl/standardMessageDestinations.fcl:116
               }
               MTdiagnostics: {
                  limit: 0
               }
               default: {  # /scratch/knoepfel/mu2e/Offline/fcl/standardMessageDestinations.fcl:123
                  limit: 10  # /scratch/knoepfel/mu2e/Offline/fcl/standardMessageDestinations.fcl:124
               }
            }
            threshold: "INFO"  # /scratch/knoepfel/mu2e/Offline/fcl/standardMessageDestinations.fcl:108
            type: "cout"  # /scratch/knoepfel/mu2e/Offline/fcl/standardMessageDestinations.fcl:107
         }
         statistics: {  # /scratch/knoepfel/mu2e/Offline/fcl/messageService.fcl:84
            stats: {  # /scratch/knoepfel/mu2e/Offline/fcl/messageService.fcl:84
               categories: {
                  MTdiagnostics: {
                     limit: 0
                  }
               }
               type: "cout"  # /scratch/knoepfel/mu2e/Offline/fcl/standardMessageDestinations.fcl:130
            }
         }
      }
   }

Since the TrigReport category (the category for the end-of-job-summary) is not specified in the message configuration, the default category settings are applied. According to the annotations, a "limit" of '10' is specified at line 124 of Offline/fcl/standardMessageDestinations.fcl.

In order to recover the original behavior, you have a few options:

  1. Change line 124 of the standardMessageDestinations.fcl file to have a value of -1.
  2. Add an explicit 'TrigReport: { limit: -1 }' category to the configuration list of 'categories'.
  3. Ask art to special-case the TrigReport category so that it is always logged and never subject to limiting.

Option 3 requires a stakeholders discussion, which we could certainly do. Please advise.

#3 Updated by Raymond Culbertson over 2 years ago

Thanks very much! Did you try solution 2? I put that in and
it appears in the fhicl dump, parallel to ArtReport, but it doesn't seem
to have any effect.

message: { # ./Analyses/test/genReco.fcl:128
destinations: { # /mu2e/app/users/rlc/head/Offline/fcl/messageService.fcl:78
log: { # /mu2e/app/users/rlc/head/Offline/fcl/messageService.fcl:79
categories: { # /mu2e/app/users/rlc/head/Offline/fcl/standardMessageDestinations.fcl:111
ArtReport: { # /mu2e/app/users/rlc/head/Offline/fcl/standardMessageDestinations.fcl:115
limit: -1 # /mu2e/app/users/rlc/head/Offline/fcl/standardMessageDestinations.fcl:118
reportEvery: 50 # /mu2e/app/users/rlc/head/Offline/fcl/standardMessageDestinations.fcl:116
}
MTdiagnostics: {
limit: 0
}
TrigReport: { # /mu2e/app/users/rlc/head/Offline/fcl/standardMessageDestinations.fcl:127
limit: -1 # /mu2e/app/users/rlc/head/Offline/fcl/standardMessageDestinations.fcl:128
}

#4 Updated by Kyle Knoepfel over 2 years ago

Ahem...sorry...the tag at the front of each line is "TrigReport" but the actual category is "ArtSummary".

#5 Updated by Raymond Culbertson over 2 years ago

Great, that's it - thanks again! For me, this is a good fix. and you close the ticket.

#6 Updated by Kyle Knoepfel over 2 years ago

Okay, thanks for the quick turnaround, Ray. I think I can quickly implement option 3. But if not, I will close this bug anyway.

#7 Updated by Kyle Knoepfel over 2 years ago

  • Tracker changed from Bug to Support
  • Category set to Infrastructure
  • Status changed from Feedback to Closed
  • % Done changed from 0 to 100

Actually, upon reflection, doing this automatically in art is more difficult than I imagined. Will close per your suggestion.



Also available in: Atom PDF