Consider not dropping out-of-sequence requests
On the protoDUNE DAQ teststand, it's been observed empirically using artdaq's feature/protodune branch (based off of artdaq v2_03_03) that the EventBuilderCore can get fragments from the timing fragment generator out-of-sequence. This can be seen, e.g., in run 1000504. On np04-srv-012:
grep "type = 5" /log/eventbuilder/run1000504-eventbuilder1.log
reveals that fragment with sequence ID 21 appears before fragment with sequence ID 20.
The significance of this is the following: in EventStore.cc, starting on l. 114 in artdaq v2_03_03, logic exists whereby a request is only sent up to fragment generators running in pull mode if the sequence ID of the request is higher than the previous highest sequence ID of a sent request. As a consequence, these out-of-order fragments result in some requests never getting sent - e.g., in the example above, the fragment with sequence ID 20 never causes the sequence ID 20 request to get sent to the fragment generators. This Redmine Issue is filed to open a discussion on the possibility of removing the requirement that a sent request never have a sequence ID lower than that of a previously sent request.
#1 Updated by Eric Flumerfelt almost 3 years ago
This can be a very hard error case to detect and recover from, as in any system with more than one EventBuilder (especially systems using RoutingMasters), there is no clear progression of sequence IDs. My initial reaction is therefore to figure out how fragments can arrive out-of-sequence, and fix that issue.