More sophistication/flexibility should be offered concerning the order of transitions to processes
Motivated by ever-more-complex process layouts (multiple subsystems, DFO usage, etc.) Kurt and I discussed ways to improve the order in which DAQInterface sends transitions to the artdaq processes. Currently, what happens is that transitions are sent to one class of process, then another class, then another class - e.g., on start, to Dispatchers, then to DataLoggers, then to EventBuilders, then to BoardReaders. The order of class depends, of course, on the transition - on stop, the order is the reverse of what's on start. The only additional sophistication is that users have the option of deciding in what order the individual boardreader processes receive commands (see Issue #16637).
However, there are two realities which exist now which didn't exist when this original scheme was developed:
1) The ability to run with multiple subsystems.
2) The addition of new process types, e.g., the DFO
An obvious action to take would be to allow ultimate flexibility to the user on when processes receive transitions. Another action is to think carefully about what sort of default behavior we want when multiple subsystems are involved. We want to make sure that DAQInterface's transition order is able to handle the latest, most complex possible process arrangements.
#1 Updated by John Freeman 6 months ago
Here are some further thoughts. Let's take as a model four subsystems, each containing boardreaders, eventbuilders and dataloggers. We'll label the four subsystems A, B, C and D. Subsystem A is a child of subsystems B and C. Subsystem B is a child of subsystem D. In other words:
D \ B C \ / A
It seems like on the start transition, really the main thing we'd care about is that the boardreaders across all four subsystems get the start transition after all the other process types- would it matter if the eventbuilders got the start transition before the dataloggers, as long as boardreaders remained in the configured state and thus weren't sending data? Currently, the dataloggers receive the start transition, and then afterwards, the eventbuilders receive the start transition - but this distinction may be unnecessary, in which case we could parallelize the eventbuilders/dataloggers' start transition and thereby hopefully speed up the overall start.
The stop transition is a bit trickier. This is because the eventbuilders in A are receiving data from the eventbuilders in B and C, and furthermore, the eventbuilders in B are receiving data from the eventbuilders in D. So I suppose one question is, in this scenario would we want to first stop the eventbuilders in D, then stop the eventbuilders in B and C, and then stop the eventbuilders in A? This would be as opposed to the current behavior, which is where a stop is sent to all eventbuilders across the four subsystems at once.
That question, of course, refers to default behavior. The next question is, what sort of interface should be offered to users to allow them to exercise fine-grained control over the relative order of process stops/starts? Right now, there's the "boardreader_priorities" option introduced in Issue #16637 and documented in the DAQInterface manual at https://cdcvs.fnal.gov/redmine/projects/artdaq-utilities/wiki/The_settings_file_reference . This allows users to set the relative order in which specific boardreaders receive the start transition. Kurt pointed out that we might want a different order on stop, so it seems we could add a "boardreader_priorities_on_stop" option as well. Also, given that some eventbuilders are DFOs, should we introduce an "eventbuilder_priorities" option? Or is the behavior of DFOs well-understood enough that DAQInterface can give it some default transition order, e.g. always before or always after the other eventbuilder types?
Another question is, do we want to give users complete control over the transition order, or is this an example of https://en.wikipedia.org/wiki/You_aren't_gonna_need_it? I could add options called "overriding_priorities" and "overriding_priorities_on_stop" in which users would be allowed to specify any order of transitions to the processes, as long as the set of regexes provided covered all the processes in a run, but would this ever be used or would the options discussed in the above paragraphs be all users would realistically need - and not enough rope to hang themselves by?
#3 Updated by John Freeman 4 months ago
- % Done changed from 0 to 100
- Status changed from New to Resolved
I'm marking this issue as resolved at the head of feature/issue22994_transition_order, commit ecfc47a878053f43e751fc198608c46e3e0e58c5. However, there's a caveat: since this issue is fairly broad, I've decided to splinter off speed improvements related to the parallelization of transitions from this issue, and have it focus solely on offering users greater flexibility in the order in which boardreaders are sent transitions. In particular, this focus is motivated by an email thread started by Bill Badgett on August 15 entitled "Configuration sequencing...", related to SBND's wish to use different transition orders for the various boardreaders depending on whether we're doing config, start or stop.With that said, here's what to expect in the implementation of this issue:
- It continues to be the case that if users set "boardreader_priorities" in the DAQInterface settings file, that the regular expressions in the list assigned to boardreader_priorities will define the relative order boardreaders receive transitions, regardless of transition type. Constructing the list of regular expressions is described in the section of the DAQInterface Manual concerned with the settings file, https://cdcvs.fnal.gov/redmine/projects/artdaq-utilities/wiki/Daqinterface_settings_file_reference_
- However, users now have the option of defining the priorities for individual transition types. If this is done, the parameters in the settings file are called "boardreader_priorities_on_config", "boardreader_priorities_on_start" and "boardreader_priorities_on_stop". Spaces in place of underscores are allowed. Notice that this set of three parameters is mutually exclusive of the traditional "boardreader_priorities" -- that is, you can't both define boardreader_priorities and then also one or more transition-specific boardreader priorities. If you've defined at least one transition-specific priority list, then for any transition not included in the settings file, the order in which commands are sent to artdaq processes is arbitrary - e.g., if you define boardreader_priorities_on_config and boardreader_priorities_on_stop, but not boardreader_priorities_on_start, then it's arbitrary what order boardreaders get sent the "start" transition
- The list of regexes comprising the value of the above parameters must span all boardreaders - e.g., you can't just have "boardreader_priorities_on_config: ssp101" and then try to use ssp101 and ssp102 as boardreaders. As always in regular expressions, ".*" is a catchall.
- For reviewing this issue, you may want to set the debug level in the boot file to 3. It's at this level that DAQInterface informs you of exactly when it sends a transition to each individual artdaq process.
#5 Updated by John Freeman 3 months ago
Disturbingly, based on
git diff 4e08d1a3f229cfb153e7d2a6d74746849b0f8146~1 4e08d1a3f229cfb153e7d2a6d74746849b0f8146
...it seems the logic issue crept in when the develop branch got merged into feature/issue22994_transition_order on October 23. Glad this got caught and corrected, and a good cautionary tale that merging develop into a feature branch can sometimes lead to unintended consequences.