Project

General

Profile

Feature #6727

Add a bound on the number of incomplete events that can be kept in the EventStore

Added by Kurt Biery over 5 years ago. Updated almost 3 years ago.

Status:
Closed
Priority:
Normal
Category:
Additional Functionality
Target version:
Start date:
08/01/2014
Due date:
% Done:

100%

Estimated time:
80.00 h
Experiment:
Co-Assignees:
Duration:

Description

Some background:

Currently, the EventStore class has two "insert" methods. One of them gracefully handles the case where the queue to the art thread is full; the other one does not. In the first case, the fragment that is passed in is returned to the caller; in the second, the fragment is dropped on the floor, and a warning message is optionally emitted.

The goal has been to transition to using the more graceful method everywhere, and it may be that this is fairly far along.

At the EventBuilder level, the more graceful EventStore insert() method is used to exert back-pressure to the upstream parts of the DAQ and thereby gracefully handle whatever slowdown is causing the art event queue to be full.

This purpose of this Issue is to request that we extend this sort of functionality to the EventStore itself. That is, we should add a configuration parameter to EventStore that specifies the maximum number of incomplete events that the EventStore is allowed to hold, and when that maximum is reached and the EventStore can't accept another partial event, the EventStore::insert() method should return false and return the passed-in fragment to the caller.

The trick inside the code will be to implement the logic so that if the new fragment completes an event, it is allowed to be added to the EventStore, and if it does not complete an event and the store is full, it is not allowed to be added to the EventStore.

We should include an appropriate number of unit tests in this task to verify that this new functionality is working as expected.


Related issues

Related to artdaq - Feature #3187: Add a timeout for incomplete events in EventStore.Closed12/18/2012

Related to artdaq - Feature #10146: Provide moderate synchronization when sending fragments from multiple BoardReadersClosed09/01/2015

History

#1 Updated by Kurt Biery over 5 years ago

  • Estimated time changed from 50.00 h to 80.00 h

One complication with this request is that we will somehow need to selectively exert back-pressure on some BoardReaders and not others. The current implementation of the graceful EventStore::insert() method and its use in EventBuilderCore exerts backpressure on all BoardReaders at the same time.

How should the improved insert() method tell its caller "Don't send me any more fragments with FragmentID X for a while" and how should EventBuilderCore respond to that? And, how should the EventStore class tell the code that it uses it that "It's OK to resume sending me fragments with FragmentID X now"? And, how to do this without breaking encapsulation?

#2 Updated by Kurt Biery over 4 years ago

  • Target version changed from 576 to v1_12_10

#3 Updated by Kurt Biery over 4 years ago

  • Target version changed from v1_12_10 to v1_12_11

#4 Updated by Kurt Biery over 4 years ago

  • Target version changed from v1_12_11 to v1_12_13

#5 Updated by Kurt Biery over 4 years ago

  • Related to Feature #10146: Provide moderate synchronization when sending fragments from multiple BoardReaders added

#6 Updated by Kurt Biery over 4 years ago

  • Target version changed from v1_12_13 to 821

The implementation of this feature could be rather tricky. As such, it may be best to simply use BoardReader synchronization (Issue #10146).

#7 Updated by Eric Flumerfelt about 3 years ago

  • Category set to Under Consideration
  • Target version deleted (821)

#8 Updated by Eric Flumerfelt almost 3 years ago

  • Assignee set to Eric Flumerfelt

I think this is now much simpler in artdaq v2. I've already modified DataReceiverManager to accept a fragment back from callers, and implemented suppression logic so that individual sources can be turned off. There's also a suppression clear function which unsuppresses all sources. This suppression is independent of the fragment synchronization already enforced by DataReceiverManager.

In the EventStore, insert will be modified to return an enumeration value of { Success, SuccessSuppression, Rejected }:
If the EventStore is not full, it will always return Success.
If the EventStore is full, but the given fragment completed an open event, it will return SuccessSuppression.
If the EventStore is full, and the given fragment would open a new event, or in case of other errors, it will return Rejected.

The EventBuilderCore or Aggregator core will then send the fragment back to the DataReceiverManager, which will in turn suppress that source. (Multi-fragment sources might need additonal logic here.) When EventStore returns Success from insert, then the Core will unsuppress all sources (which will then drain out any rejected fragments).

#9 Updated by Eric Flumerfelt almost 3 years ago

  • Category changed from Under Consideration to Additional Functionality
  • Status changed from New to Resolved
  • Target version set to artdaq Next Release
  • % Done changed from 0 to 100

Added "max_incomplete_events" configuration parameter to EventStore, defaults to 50. artdaq_demo has been modified to generate this parameter.
Code has been tested and works. Warning messages have been implemented to warn users when fragments are being rejected from the EventStore (one per source blocked).

#10 Updated by Eric Flumerfelt almost 3 years ago

  • Status changed from Resolved to Closed
  • Target version changed from artdaq Next Release to v2_01_00


Also available in: Atom PDF