Project

General

Profile

Feature #23131

Add reporting of fragment IDs for missing fragments to the FragmentWatcher module

Added by Kurt Biery 3 months ago. Updated 29 days ago.

Status:
Assigned
Priority:
Normal
Assignee:
Category:
-
Target version:
-
Start date:
08/15/2019
Due date:
% Done:

0%

Estimated time:
Experiment:
-
Co-Assignees:
Duration:

Description

The FragmentWatcher art module (included in artdaq/ArtModules) already has reporting of the fragment IDs of individual Empty fragments. It would be helpful to also have this level of detail in the tracking of Missing fragments.


Related issues

Related to artdaq - Feature #5958: Add the ability to flag various conditions in a RawEventClosed12/18/2012

History

#1 Updated by Kurt Biery 3 months ago

The candidate code changes are on the feature/23131_FragWatcher_MissingFragDetail branch in the artdaq repo. This branch was created based on the v3_06_00 tag.

In addition to adding detailed tracking and reporting of the fragment IDs for missing fragments, I removed blocks of code in the constructor and destructor that dealt with the metrics_pset. This code must have been a hold-over from whatever source code file(s) that I used as a starting point for the FragmentWatcher, but my current sense is that it was very much out-of-place there. The configuration snippet for the FragmentWatcher will likely never include information about metrics.

I also tried to organize the TRACE messages and explain their intention with comments in the code.

#2 Updated by Kurt Biery 3 months ago

I tested these changes in an artdaq-demo system on sbnd-daq33 with the mediumsystem_with_routing_master sample configuration.

Here are sample file metrics entries:

[biery@sbnd-daq33 eventbuilder]$ pwd
/home/nfs/biery/gpmTest/daqlogs/eventbuilder
[biery@sbnd-daq33 eventbuilder]$ egrep 'EventsWith|EventsProcessed' evb_13508_metrics.log
...
Thu Aug 15 08:26:33 2019: FileMetric: EventBuilder3_art1.EventsProcessed: 80 events.
Thu Aug 15 08:26:33 2019: FileMetric: EventBuilder3_art1.EventsWith50PctMissingFragments: 0 events.
Thu Aug 15 08:26:33 2019: FileMetric: EventBuilder3_art1.EventsWith10PctMissingFragments: 16 events.
Thu Aug 15 08:26:33 2019: FileMetric: EventBuilder3_art1.EventsWithMissingFragments: 16 events.
Thu Aug 15 08:26:33 2019: FileMetric: EventBuilder3_art1.EventsWith10PctEmptyFragments: 76 events.
Thu Aug 15 08:26:33 2019: FileMetric: EventBuilder3_art1.EventsWithEmptyFragments: 76 events.
Thu Aug 15 08:26:33 2019: FileMetric: EventBuilder3_art1.EventsWith50PctEmptyFragments: 76 events.

Here are sample logfile messages:

[biery@sbnd-daq33 art]$ pwd
/home/nfs/biery/gpmTest/daqlogs/art
[biery@sbnd-daq33 art]$ grep -A 2 -i msg-w art-sbnd-daq33-2019081508254*.log | tail -15
art-sbnd-daq33-20190815082548-13508.log:%MSG-w EventBuilder3_art1_FragmentWatcher:  DAQ 15-Aug-2019 08:26:18 CDT Booted FragmentWatcher_module.cc:216
art-sbnd-daq33-20190815082548-13508.log-Event 224: total_fragments=9, fragmentIDs for empty_fragments: 1, 2, 3, 4, 5, 6, 8, 9
art-sbnd-daq33-20190815082548-13508.log-%MSG
art-sbnd-daq33-20190815082548-13508.log:%MSG-w EventBuilder3_art1_FragmentWatcher:  DAQ 15-Aug-2019 08:26:18 CDT Booted FragmentWatcher_module.cc:202
art-sbnd-daq33-20190815082548-13508.log-Event 225: total_fragments=9, fragmentIDs for missing_fragments: 7
art-sbnd-daq33-20190815082548-13508.log-%MSG
art-sbnd-daq33-20190815082548-13508.log:%MSG-w EventBuilder3_art1_FragmentWatcher:  DAQ 15-Aug-2019 08:26:18 CDT Booted FragmentWatcher_module.cc:216
art-sbnd-daq33-20190815082548-13508.log-Event 225: total_fragments=9, fragmentIDs for empty_fragments: 1, 2, 3, 4, 5, 6, 8, 9
art-sbnd-daq33-20190815082548-13508.log-%MSG
art-sbnd-daq33-20190815082548-13508.log:%MSG-w EventBuilder3_art1_FragmentWatcher:  DAQ 15-Aug-2019 08:26:18 CDT Booted FragmentWatcher_module.cc:202
art-sbnd-daq33-20190815082548-13508.log-Event 226: total_fragments=9, fragmentIDs for missing_fragments: 7
art-sbnd-daq33-20190815082548-13508.log-%MSG
art-sbnd-daq33-20190815082548-13508.log:%MSG-w EventBuilder3_art1_FragmentWatcher:  DAQ 15-Aug-2019 08:26:18 CDT Booted FragmentWatcher_module.cc:216
art-sbnd-daq33-20190815082548-13508.log-Event 226: total_fragments=9, fragmentIDs for empty_fragments: 1, 2, 3, 4, 5, 6, 8, 9
art-sbnd-daq33-20190815082548-13508.log-%MSG

Here are sample TRACE messages:

19622 08-15 08:26:16.909988  13507  13507   1      EventBuilder1_art1_FragmentWatcher wrn . Event 211: total_fragments=9, fragmentIDs for empty_fragments: 1, 2, 3, 4, 5, 6, 8, 9
19623 08-15 08:26:16.909812  13507  13507   1      EventBuilder1_art1_FragmentWatcher wrn . Event 211: total_fragments=9, fragmentIDs for missing_fragments: 7
19624 08-15 08:26:16.909797  13507  13507   1      EventBuilder1_art1_FragmentWatcher d04 . Event 211: this event: total_fragments=9, missing_fragments=1, empty_fragments=8 (66 events processed)
19724 08-15 08:26:16.795336  13507  13507   1      EventBuilder1_art1_FragmentWatcher d07 . Event 210: events_with_50pct_missing_fragments=0, events_with_50pct_empty_fragments=65
19725 08-15 08:26:16.795326  13507  13507   1      EventBuilder1_art1_FragmentWatcher d07 . Event 210: events_with_10pct_missing_fragments=11, events_with_10pct_empty_fragments=65
19726 08-15 08:26:16.795321  13507  13507   1      EventBuilder1_art1_FragmentWatcher d06 . Event 210: events_with_missing_fragments=11, events_with_empty_fragments=65
19727 08-15 08:26:16.795250  13507  13507   1      EventBuilder1_art1_FragmentWatcher wrn . Event 210: total_fragments=9, fragmentIDs for empty_fragments: 1, 2, 3, 4, 5, 6, 8, 9
19728 08-15 08:26:16.795146  13507  13507   1      EventBuilder1_art1_FragmentWatcher wrn . Event 210: total_fragments=9, fragmentIDs for missing_fragments: 7
19729 08-15 08:26:16.795136  13507  13507   1      EventBuilder1_art1_FragmentWatcher d04 . Event 210: this event: total_fragments=9, missing_fragments=1, empty_fragments=8 (65 events processed)
19739 08-15 08:26:16.794524  13507  13507   1      EventBuilder1_art1_FragmentWatcher d07 . Event 209: events_with_50pct_missing_fragments=0, events_with_50pct_empty_fragments=64

The local changes to the EventBuilder configuration files were the following:

--- a/simple_test_config/mediumsystem_with_routing_master/EventBuilder1.fcl
+++ b/simple_test_config/mediumsystem_with_routing_master/EventBuilder1.fcl
@@ -27,6 +27,7 @@ daq: {
        send_requests: true
        request_port: 3001
        request_address: "227.128.12.26" 
+       multicast_interface_ip: "192.168.230.0" 
        expected_art_event_processing_time_us: 2000000

     routing_token_config: {
@@ -46,8 +47,8 @@ daq: {
        evbFile: {
          metricPluginType: "file" 
          level: 4
-         fileName: "eventbuilder/evb_%UID%_metrics.log" 
-         absolute_file_path: false
+         fileName: "/home/nfs/biery/gpmTest/daqlogs/eventbuilder/evb_%UID%_metrics.log" 
+         absolute_file_path: true
          uniquify: true
        }
   }
@@ -61,12 +62,16 @@ outputs: {

 physics: {
   analyzers: {
-
+    fragmentwatcher: {
+      metrics_reporting_level: 1
+      mode_bitmask: 7
+      module_type: "FragmentWatcher" 
+    }
   }

-  producers: {
-  }
+  producers: { }

+  a1: [ fragmentwatcher ]
   my_output_modules: [ rootNetOutput ]
 }
 source: {

#3 Updated by Eric Flumerfelt 29 days ago

  • Status changed from New to Assigned
  • Co-Assignees Eric Flumerfelt added

Performed code review. I noticed that the fragmentHandles are iterated over twice; it may be more performant to only iterate over them once and do both the missing and empty Fragment checks at that time. Also, it would be nice if the metrics reporting level for at least DETAILED_COUNTS_MODE was different, so that the XML could be sent to a different metric destination.

Code tested and verified.

#4 Updated by Eric Flumerfelt 29 days ago

Thinking more about my comments, I'd almost suggest replacing the mode_bitmask and metrics_reporting_level with "basic_counts_metric_level", "fractional_counts_metric_level", and "detailed_counts_metric_level" and indicate in the documentation that a level of 0 means to disable that reporting, which would be the default for fractional and detailed modes.

#5 Updated by Eric Flumerfelt 25 days ago

  • Related to Feature #5958: Add the ability to flag various conditions in a RawEvent added


Also available in: Atom PDF