Framework notes

References (reverse chronological order, TDR at the bottom):


  • As I understand it, the idea is to create tools not for typical physicists on DUNE, but instead for a relatively small core - O(10) - of DUNE software developers. These tools are designed to ease the learning curve of new developers as they come in, to make it easy to swap out implementations, and to provide functionality which you'd expect many different types of DUNE process to need.
  • One possible framework model is to have a set of libraries (buffer, thread, error handling, etc.) and then have a standard "DUNE process" with a standard header, with these libraries already linked in. If developers decide they need a structure not offered by the standard DUNE process, or simply have a jack-of-all-trades, reinvent-the-wheel approach to things, they can still benefit from at least some of the libraries available
  • The work here is DUNE-focused, which may sound trivial, but the point is that normally we shouldn't sacrifice much time or effort modifying the libraries so they do things that other experiments want, but that DUNE doesn't care about
  • We should keep in mind lessons learned from ProtoDUNE. E.g.: it should be easy to observe third party library error messages sent to stdout, and error reporting should ideally report on the cause and not symptoms (including if the cause is a crappy NFS disk, slow network, etc., rather than buggy experiment code)
  • Other thoughts, ideas?

An example of a DUNE process: the Module Level Trigger (MLT)

The MLT is a good example of a process to think about. The reason is that it's one of the more complex DUNE processes. Therefore any framework which makes its construction relatively easier than it could be should also make the construction of simpler processes relatively easier than it could be as well.

The way the MLT works is:
-It receives trigger candidate data from up to 150 trigger candidate generator processes, each associated with an APA in a module
-It receives additional trigger candidate data originating in the photon system
-It receives whatever data the ETI (External Trigger Interface) sends it (multiple formats?)
-It runs algorithms to figure out whether the combination of trigger candidates in some time range and whatever the ETI sent it amounts to a trigger
-If it finds a candidate, it sends a trigger command down to the DFO
-If the DFO determines that the eventbuilders can't handle the rate of trigger commands, the MLT needs to be ready to start prescaling or otherwise respond to the situation

Additionally, like all processes:
-It needs to be alert for when data's no longer coming from an expected source. Here, that could be a trigger candidate generator process being intentionally removed from the partition (see "Someone needs to take APAs currently in use in order to perform calibrations" from the requirements page) or simply crashing
-It should try to handle errors as gracefully as possible (see "A process encounters an unrecoverable error during running" from the requirements page)

In a logical (if not physical) sense, the following is needed:
-Up to 150+ receiver threads, receiving trigger candidates from upstream processes which are sending the candidates asynchronously
-Consequently, either 150 separate buffers for each thread, or fewer buffers which can say "Don't perform I/O, someone's writing to me"
-Another receiver thread, receiving whatever sort of data the ETI is supposed to send
-A receiver thread which is listening for state transitions to the DAQ
-A receiver thread which is listening to be told if one or more of its sources has dropped out
-A receiver thread for when there's a full physics reconfiguration (perhaps the same thread as above?)
-A sender thread, which sends trigger decisions down to the DFO
-A receiver thread, which listens to any important messages the DFO sends it
-A sender thread, which can report metrics without interfering with the primary task of the process (or maybe just broadcast it out, non-blocking)
-An analysis thread, which collates the data from the buffers containing trigger candidates from 150 different trigger candidate generators and whatever data the ETI sends, and which comes up with trigger decisions
-The ability to handle a new configuration coming in while it's still processing old trigger candidates. This might mean that it just waits to fully process the old trigger candidates before changing its variable values, deleting and creating new objects, etc. - or it might mean it launches a SECOND thread which starts in on the newer trigger candidates with the new configuration, while the original thread processes older trigger candidates with its older configuration

Another example: the EventBuilder

This one's useful to look at since it has functionality requirements that don't totally overlap with that of the MLT. E.g.:

-It needs to be able to tell the DFO that it's too busy to accept any triggers
-It needs a buffer which has a concept of "there's a complete event here". The analogue would be artdaq's SharedMemoryEventManager, but not necessarily with shared memory as its physical manifestation
-It needs sender threads which have a request-reply model so it can request raw data from the APAs, presumably with a timeout


Just some thoughts...


A standard dune execution thread could be tied to a particular configuration. It could have built-in convenience functions which provide:
-The run number
-The subrun number
-The detector timestamp
-The wall clock time
-Its own priority
-The process state
-Partition number
-A pointer to the complete configuration (perhaps a config ID like Brett mentions in his epoch paper)
-A (subset?) of the input and output buffers associated with the receiver and sender threads in the process, presumably with thread-specific access rights.
-Other access rights: "you can add to this buffer but not remove anything", "you can remove from this buffer but not add anything", etc.
-It would also be nice to be able to tell a thread, "for data timestamped after this time, change these variables to these new values) without needing to do a full setup (e.g., adding a single trigger type to a trigger table, or changing a single register)


A standard dune buffer could:
-Be constructed with an option that tells it what to do when it fills up, e.g. (A) do nothing, (B) print a warning, (C) throw an exception, (D) immediately abort the program, (E) other ideas?
-Drop an entry if it's been there for more than a certain amount of time, with or without a warning
-Print automatic warning messages if it hits 90% capacity
-Have a 2-d form which supports the concept of an event (again, like SharedMemoryEventManager)
-Abstract out its physical implementation, so it would be easy, e.g., to replace shared memory with RAM with disk space without perhaps even needing to recompile
-Be able to return the newest and oldest timestamped members. Would be very useful when working with the epoch scheme Brett discussed. You'd want it to be easy to pluck this value from across containers, though.

We could offer an interface, and then provide a standard implementation centered around a std::vector.

To that point: it seems like we'd want it to be possible to grant access to the implementation of the buffer, since it's no fun to reimplement (or even just wrap) the numerous functions provided by STL containers (emplace_back, push_back, etc.)

Should a DUNEBuffer only hold elements of DUNEDataType? It seems like the answer is yes, given that the DUNEBuffer I described above assumes its elements have quantities like a detector timestamp

An example buffer interface:

Data type

The reason for this might best be explained by a simple example of what the interface might look like (in quasi-C++ code):

class DUNEDataType {

 virtual std::string type() = 0; // Derived classes could have "TriggerCandidate", "TriggerPrimitive", "BeamlineInfo", etc.
 virtual uint8_t* payload() = 0; 
 virtual uint8_t* header() = 0;
 virtual size_t timestamp() { assert(false && "Developer error: timestamp for datatype <type> either isn't a coherent concept, or should have been implemented but wasn't"); } 

 void mark_corruption_level(corruption_level );
 corruption_level get_corruption_level();  // An enum which could denote "totally fine", "garbage" and "probably somewhere in between". Checksums?

 error_state get_error_state(); // error_state could perhaps include the concept of corruption_level, but then also ideas like "contents failed a selection cut" 

 // Brainstorming - would these deletion functions be useful, if a piece of code decided data was garbage but couldn't be bothered to delete it?
 bool marked_for_deletion();

// More brainstorming. In certain situations, perhaps there could be functionality to access the data types from which this data type was derived. E.g., if this is a TriggerCandidate, the DUNEDataType interface could provide the list of associated TriggerPrimitives. OTOH, it seems tasteless to make DUNEDataType have to know about DUNEBuffer, but that just means there's probably a better way to write this interface
DUNEBuffer get_associated_data(); 


Then, in pseudocode, you might have something like:

DUNEBuffer ContainerOfTriggerCandidates; // Where these are actually TriggerCandidate class instances, and TriggerCandidate derives from DUNEDataType

// Imagine instead of this comment, we have code which fills ContainerOfTriggerCandidates

for trigcand in ContainerOfTriggerCandidates:
 if (trigcand.timestamp() not in thisthread.applicableEpoch) continue
 if (trigcand.ErrorState() != 0) print trigcand.ErrorMessage()
 if (passes_additional_scrutiny(trigcand)):

 if really_good_candidates .gt. number_of_really_good_candidates_needed_for_trigger_decision:
   // ...insert whatever code is needed to form a trigger decision, and break out of this loop...

Error handling

Something like ExceptionHandler, but I'm sure that can be improved on.

Operational monitoring (metrics reporting)

[KAB] We had presentations from Eric and Brett on this in November 2019 (link). Here are some notes from the ensuing discussion:

  • the ability to send monitoring information to configurable destinations (at different rates, with different choices of quantities) is desirable. Also, "it would be nice if the publication interval was not a property of the plugin, but rather of the variable to publish".
  • different people have different opinions on whether it makes sense to summarize the data in the reporting process (possibly in a different thread) or whether to off-load the calculations to a different process. Some of the trade-offs include how much data is sent over the network and how much freedom people have in creating new ways of analyzing the data.
    • can we support both? Could the operational monitoring (C++) interface support reporting of quantities in a granular way and support the optional ability to run analyzer code on those quantities before they are published?
  • there are other options, too. For example, for an event counter
    • publish the full data stream 1,2,3,4,5,6,7….
    • metrics manager receives 1,2,3,4,5,6,7…. and publishes max=7, min=1, avg=3.5, or something alike
    • metrics manager picks variable at a defined interval, e.g. takes 7, and publishes it
    • In the third option, "This approach may loose some granularity, but since spikes or variations happen more or less randomly, the behaviour of the metrics will still be representative.
      The backend part can then perform any processing and transform, e.g., the ev counter in a rate since the start of run or since the last publication. It can aggregate metrics from different sources and make average, median, variant, etc. This can be done precisely if the metric contains the time at the origin and we don’t use the time at which variables have been archived, thus eliminating any systematics caused by the chosen plugin manager."
  • it would be nice if every metric manager exposed a remotely-accessible method that could be invoked to initiate a quasi-instantaneous update of all the registered variables
  • a possible improvement could be allowing users to register more complex data structures
  • Any other metadata besides wall clock time to bundle with a metric? E.g.: process of origin, thread metric was reported on, line of source file the metric was reported on, and in the case of metrics which depend on other metrics, a list of what those metrics are.
  • Two problems with metrics: (1) they make the code that's actually performing the task harder to read, thereby increasing the chance of bugs, etc., (2) they can slow down the overall performance of an application. Are there ways we can safeguard against this?
  • Longer term, is there a way we can have systemwide, rather than process-specific, metrics? E.g., the average time it takes to go from an MLT forming a trigger decision to the time it takes to get the complete event in the storage buffer
  • If we have a DUNE framework process, we could have "baked-in" metrics: e.g., average time/resources dedicated to receiver vs. sender vs. execution threads. Perhaps even available resources on the node the process is running on: RAM, disk space, etc., although this perhaps would best be handled by a service orthogonal to the metric reporting in the software toolkit.
  • Metrics which are specific to code, rather than the dataflow? I'm thinking, e.g., benchmarking ("what's the average time it takes this block of code to execute"). Could get fancy: "what's the average time it takes this block of code to execute as a function of the size of the input data?". Or fancier: "how many warning messages are printed on average in this process/block of code per unit time"?

Initial attempt at a summary table

Functionality Type Library or Framework? Comment
Threads library?
Buffers library? should we list different types separately?
Error handling
Operational monitoring separately consider "processing" and "transport"?
Configuration Receiving and internal handling?
Command reception
State transitions
Data transport
Data format(s)
Message transport
Message format(s)
Data and/or message serialization?

[JCF, this is a copy of Kurt's summary table, but with my notes/ideas rather than his. We can negotiate and merge later]

Functionality Type Library or Framework? Comment
Threads Framework Framework, if you should expect that any code which executes on a thread can access partition number, state, detector time, run number, subrun number, epoch, etc.
Or to rephrase: what thread functionality isn't already available from Boost or the STL which we could isolate in a library?
Buffers library Although a lot of the functionality which justifies creating a DUNE-specific buffer class (e.g., "oldest_data()") implies it would only be used on DUNE datatypes
Error handling library
Operational monitoring
Configuration Library Need an interface which isn't unlike fhicl::pset, even though implementation could be XML, etc.
Command reception Framework Any standard DUNE process should have built-in command reception functionality
State transitions Framework Any standard DUNE process should have built-in state functionality. Btw, what states would we have besides "currently usable in a run" and "not currently usable"?
Data transport Both The idea of receiver and sender threads asynchronously reading from and writing to DUNE buffers seems framework-ish, although we could have a simple interface for the physical layer in a library
Data format(s) Library A library, though you'd be most likely to see DUNEDataTypes in code written for a DUNE framework process
Message transport Would this be something different than a special case of data transport?
Message format(s)
Data and/or message serialization? Library I think. Would we ever need DUNE-specific quantities for this (detector timestamp, etc.)?

A standard DUNE process

[JCF: disclaimer - I take "standard DUNE process" to be synonymous with "framework"]

Very broadly, the processes which make up the DUNE DAQ (TCG (Trigger Candidate Generator), MLT (Module Level Trigger), DFO, etc., operate on a principle like the following: some number of receiver threads asynchronously read data into buffers, then one or more execution threads do things with the data in those buffers, then some number of sender threads send the data out of buffers on an as-available basis. Also, the processes all are remotely issued configuration and control commands. A way to handle this might be to have a "standard DUNE process", which comes with a lot of the details behind the aforementioned features already-implemented, so developers can focus mainly on the execution threads.

Every DUNE process could have, built in to it:
- A thread which waits to be alerted about an upcoming config change, and takes action like the following:
-- If a config change comes in and there's already data timestamped past the datatime associated with the start of the new config, the process could automatically announce the failure and continue on with the current config
-- If a config change comes in and we don't have this problem, the thread could diff with the previous config to figure out what actions to take. E.g., if the only difference is that external source / sink processes were added/subtracted, it would simply add/remove sender/receiver threads accordingly. OTOH, if there were a physics change (like a register value being tweaked, or a new trigger table) then it could wait for the execution thread from the original run to process all the old-timestamped data, and then launch a new thread (or if the change is minor, modify the original one)
-It should be possible to tell the process whether it's permitted to support more than one execution thread at once (e.g., if a new epoch came in and a reconfiguration is needed)

Adding receiver threads and sender threads should be something that can be done through configuration, and not require recompilation. E.g., the relevant part of the config might look like:

receiver: {

label: receiver_for_trigcandgenerator1
associated_buffer: <label of buffer defined somewhere else>
# Then a bunch of hardware-specific stuff (port #'s, IP addresses, timeouts, etc.)

receiver: {

label: receiver_for_trigcandgenerator2
associated_buffer: <label of buffer, perhaps the same one associated with "receiver_for_trigcandgenerator1">
# Hardware specific stuff (port #'s, IP addresses, timeouts - presumably not exactly the same values as for "receiver_for_trigcandgenerator2")


# Here the assumption is that metric sending functionality needs to be explicitly requested, and is not a baked-in part of the "standard DUNE process" 
sender: {

label: metrics_sender
associated_buffer: <label of buffer, obviously not the one(s) associated with the receivers - unclear that we'd need such a buffer, though>
level: <some integer>
# Perhaps we could even have different levels for different software components - e.g., if buffers overflow frequently, we'd give them a higher level than normal

Toolkit components

Each component of the toolkit should have a single well-defined task, ideally expressible in a single sentence. Components of the current framework that perform multiple tasks should be separated.

Example: CommandableFragmentGenerator (CFG) (push mode)

The current CFG receives requests via a RequestReceiver and Fragments via a FragmentGenerator. Fragments are retrieved from the FragmentGenerator and placed in a buffer by a dedicated thread. The main application thread calls one of the applyRequests methods to create an output fragment using the contents of the buffer and returns it from getNext. BoardReaderCore simply takes the Fragments from CFG and gives them to DataSenderManager one at a time.

The discrete components in a more modular framework are:
  1. The RequestReceiver, which receives requests from the network and makes them available to other components
  2. The FragmentGenerator, which retrieves data from hardware and makes them available to other components
  3. A FragmentBuffer, which receives data from the FragmentGenerator and has an interface to inspect Fragments contained in the buffer
  4. A DataRequestor, which uses requests from RequestReceiver to match data in the FragmentBuffer, creating output Fragments and making them available to other components
  5. DataSenderManager, which retrieves Fragments from one of FragmentGenerator, FragmentBuffer, or DataRequestor and sends them to one or mode destinations in broadcast, round-robin, or routed modes.