Project

General

Profile

Design notes on buffering and transitions

27-Mar-2013, Kurt Biery and Steve Foulkes

Introduction

At the moment,
  1. there is no limit on the number of partial events that can be kept in the EventStore
  2. there is no limit on the number of events that can be stored in the RawEventQueue (the queue that is used to pass events to the art thread
  3. at end run time (stop), we do not take care to drain data from the hardware modules. This often results in incomplete events being flushed inside the EventBuilders.
  4. we do not have documentation on how various messages will be sent to the art process inside the Aggregator.

For #4, such documentation may naturally be generated as part of the creation of the Aggregator. However, it seems useful to make some notes here.

And, we would like to make some concrete proposals for how to handle #1-3.

Requirements for draining the system at Pause and Stop transitions

How do we want to handle system Pause and Stop requests at the hardware level?

Let's consider a Stop transition first. We have discussed a model where triggers are turned off first, then 1720 readout is stopped, and then the EventBuilders are sent the Stop request. We have implemented this model in Steve's ds50MasterControl.sh script. However, the current stopping of 1495 and 1720 readout is rather abrupt - there is no attempt to read out existing buffers in the 1495 FIFO or the 1720 memory.
  • Do we want this to be more graceful?

Draining the FragmentReceivers

Currently, the FragmentReceivers send all of their fragments to the EventBuilders and then send an "end-of-data" fragment to the EventBuilders.

The model of sending all known fragments is probably the correct one, but we may need to change the special fragments that are sent from "end-of-data" fragments to "end-subrun" or "end-run" fragments.

Draining the buffers and queues in the EventBuilders

We propose to drain all partial events from the EventStore at Pause or Stop time. These events will be marked as incomplete and sent the art process.
  • How should we "mark" art/ROOT events as incomplete. We've recently added an "is complete" flag to the RawEvent class, but this information needs to propogate into the art/ROOT event. I have heard people talk about "adding a product" to indicate some condition or other. Does this mean that we simply add some special data product to the event in the code that pulls the Fragments out of the RawEvent and stores them in an EventPrincipal?

We propose to drain all events from the RawEventQueue at Pause or Stop time.

We propose to drain all events from the art thread at Pause or Stop time.

Draining the buffers and queues in the Aggregator

We propose to drain all partial event sets from the EventStore at Pause or Stop time.

We propose to drain all event sets from the RawEventQueue at Pause or Stop time.

We propose to drain all event sets from the art thread at Pause or Stop time.

Handling stale events in the EventStore

Limiting the size of the RawEventQueue in the EventBuilder and Aggregator

EventStore enhancements to support use in the Aggregator

Sequence ID Modulus Logic

The EventStore needs to be able to group fragments with different Sequence IDs.

Ability to handle multiple runs / subruns

Methods will need to be added to inject special RawEvents (defined below).

Messages injected into art from the EventBuilder

The EventBuilder will communicate changes in run state to the art input module via custom RawEvent messages. The RawEvent messages will contain a single fragment with a custom type (StartRun, EndRun, StartSubRun, EndSubRun). The input module will be aware of these messages and will prob art to configure the run state accordingly.

StartRun

Sent whenever a transition is made between the Ready and the Running state.

EndSubRun

This message can be sent on a couple different transitions:
- When a transition is made between the Running and Paused state
- When a transition is made between the Running and Ready state (stop)
- When a transition is made between the Running and Initialized state (init)
- When a transition is made between the Running and Booted state (shutdown)

Sent whenever a transition is made between the Running and Paused state.

BeginSubRun

This can be sent on two different transitions:
- When a transition is made between the Paused and Running state
- When a transition is made between the Ready and the Running state. The message will be sent after the StartRun message.

EndRun

This message must always follow the EndSubRun message and can be sent on a couple different transitions:
- When a transition is made between the Running and Ready state (stop)
- When a transition is made between the Running and Initialized state (init)
- When a transition is made between the Running and Booted state (shutdown)

Implementation notes

Should these messages be sent in to associated state machine guard methods (e.g. do_start(), do_pause(), do_resume(), do_stop()) or in the entry and exit actions of the appropriate states?
  • Probably the guard conditions so that we have more control of exactly when the messages are sent.

Messages between the EventBuilder and Aggregator

There will be Init/StartRun/StopRun/StartSubRub/StartEndRun messages sent between the EventBuilder and Aggregator. Init messages will be distinguished by their fragment type. It is up to the Aggregator to drop all but one of these which will be passed into the NetMon input module. The other message types need to be passed straight into the input module, avoiding the EventStore.