Project

General

Profile

Feature #22363

Provide a simple art module to check for missing and empty fragments in the EventBuilder art process

Added by Kurt Biery 8 months ago. Updated 4 months ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
Additional Functionality
Target version:
Start date:
04/12/2019
Due date:
% Done:

0%

Estimated time:
Experiment:
-
Co-Assignees:
Duration:

Description

Folks at protoDUNE have asked for an art module to check for missing and empty fragments.

I have created a simple module to do this, starting from examples from JohnF and WesK. The new module is called FragmentWatcher. I'll describe a little more about it after I commit the code.

History

#1 Updated by Kurt Biery 8 months ago

I've committed the code to the feature/22363_FragmentWatcherModule branch in the artdaq repo. This branch is based on the head of the develop branch, as of a few minutes ago.

This module currently has two modes (and another one that is imagined, but not yet implemented):
  • basic mode - keeps track of number of events processed, events with missing fragments, and events with empty fragments
  • fractional mode - keeps track of the number of events processed, the number of events that have more than 10% of the fragments missing, number with 50% of fragments missing, number with 10% of the fragments empty, and 50% of the fragments empty.
  • detailed_counts mode - I imagine this as being able to publish lists of missing and empty fragments per BoardReader. (something like the strings listed below)
This module currently has two configuration parameters:
  • mode_bitmask - bit 0 enables Basic mode, bit 1 enables Fractional mode (default 0x1)
  • metrics_reporting_level - this is the "level" value that will be used when reporting values to the metric manager.

I can imagine wanting to publish these metrics at slightly different frequencies than typical ones, so this Issue is likely linked to #22355.

Here are sample strings with details:

=====================================================

event_builder_snapshot : {
name: "EventBuilder5"
timestamp: "20190408T124433"
events_built: 105

sender_list: [ "felix501", "felix501", "ssp101", "ssp102" ]
valid_fragment_counts: [ 105, 105, 102, 104 ]
empty_fragment_counts: [ 0, 0, 2, 0 ]
missing_fragment_counts: [ 0, 0, 1, 1 ]
}

=====================================================

<event_builder_snapshot name="EventBuilder5">
<timestamp>20190408T124433</timestamp>
<events_built>105</events_built

&lt;sender_list&gt;
&lt;sender index=0&gt;felix501&lt;/sender&gt;
&lt;sender index=1&gt;felix502&lt;/sender&gt;
&lt;sender index=2&gt;ssp101&lt;/sender&gt;
&lt;sender index=3&gt;ssp102&lt;/sender&gt;
&lt;/sender_list&gt;
&lt;valid_fragment_counts&gt;
&lt;count index=0&gt;105&lt;/count&gt;
&lt;count index=1&gt;105&lt;/count&gt;
&lt;count index=2&gt;102&lt;/count&gt;
&lt;count index=3&gt;104&lt;/count&gt;
&lt;/valid_fragment_counts&gt;
&lt;empty_fragment_counts&gt;
&lt;count index=2&gt;2&lt;/count&gt;
&lt;/empty_fragment_counts&gt;
&lt;missing_fragment_counts&gt;
&lt;count index=2&gt;1&lt;/count&gt;
&lt;count index=3&gt;1&lt;/count&gt;
&lt;/missing_fragment_counts&gt;
&lt;/event_builder_snapshot&gt;

=====================================================

#2 Updated by Kurt Biery 7 months ago

This week, I offered to add some initial tracking of the fragment IDs that were associated with Empty fragments. And, Roland said that such functionality would be useful. So, I've made additions to the FragmentWatcher_module.cc file in the artdaq repo.

The original creation of the FragmentWatcher_module was on the feature/22363_FragmentWatcherModule branch, which was branched from "develop". In order to provide an easier transition to the for_dune-artdaq branch for this latest change, I created a new branch called feature/22363_FragmentWatcherModule_v1.5, which was branched from the for_dune-artdaq branch.

So, at this point, the feature/22363_FragmentWatcherModule branch (without the v1.5 in the name) should be considered obsolete.

I've tested the new version of the FragmentWatcher_module on the mu2edaq cluster, and there are some snippets of TRACE messages and metric reports included below. I have not yet tested these new changes on the protoDUNE cluster, but hopefully we will have a chance to do that soon.

TRACE:
9617 05-07 14:30:48.573703 32039 32039 8 EventBuilder1_art1_FragmentWatcher wrn . Event 409: total_fragments=10, fragmentIDs for empty_fragments: 2

METRIC:
Tue May 7 14:30:53 2019: FileMetric: EventBuilder1_art1.EmptyFragmentSnapshot: <eventbuilder_snapshot app_name="EventBuilder1_art1"><events_processed>146</events_processed><empty_fragment_counts><count fragment_id=2>135</count></empty_fragment_counts></eventbuilder_snapshot> xml_string.

ART LOGFILE (child EB process):
%MSG-w EventBuilder1_art1_FragmentWatcher: FragmentWatcher:basicmetrics@ 07-May-2019 14:30:54 CDT run: 9 subRun: 1 event: 467 FragmentWatcher_module.cc:157
Event 467: total_fragments=10, fragmentIDs for empty_fragments: 2

#3 Updated by Eric Flumerfelt 5 months ago

  • Status changed from Assigned to Reviewed
  • Co-Assignees Eric Flumerfelt added

#4 Updated by Eric Flumerfelt 4 months ago

  • Target version set to artdaq v3_06_00
  • Status changed from Reviewed to Closed
  • Category set to Additional Functionality


Also available in: Atom PDF