Bug #21919
If a process dies or enters an error state during boot or config, the run should not proceed
100%
Description
As described in Issue #21695, unless explicitly told otherwise DAQInterface will continue running if a process dies or throws an exception. While this is a good idea from the perspective of keeping a run going, the way DAQInterface is currently coded up, not only will it not break out of the "running" state if a process dies or goes into error, it also won't break out of the "configuring" state either! It seems reasonable that if something goes wrong before we even begin running, that's grounds for DAQInterface refusing to enter the running state.
Associated revisions
History
#1 Updated by John Freeman 7 months ago
I'll narrow the scope of this Issue, as I was overly pessimistic about the behavior of DAQInterface in the event of artdaq process problems before running. The remaining concern before running is if an artdaq process we want in a run dies, DAQInterface should refuse to proceed to the run. If, however, an artdaq process undergoing the init or the start transitions dies or throws an exception, DAQInterface will already refuse to proceed to the running state. E.g., if an exception is thrown in a fragment generator's constructor or start() function, you'll see:
Exception: An exception was thrown during the Init transition as at least one of the artdaq processes didn't achieve its desired state
and DAQInterface will enter recovery. And if the boardreader dies in the fragment generator's constructor or start() function, you'll see something like:
error: [Errno 111] Connection refused Exception caught sending Start transition to artdaq process component02 at localhost:11100
#2 Updated by John Freeman 7 months ago
- % Done changed from 0 to 100
- Status changed from New to Resolved
With commit 10c6499c41f9e4b21c2e9aeaab98fb4e1c0beb68 on the branch feature/issue21919_dont_start_run_if_problem, it's now the case that if we're not in the running state, any artdaq process dying will result in DAQInterface returning to the stopped state.
#3 Updated by John Freeman 7 months ago
Crucial point: the change in behavior here applies when we've set DAQINTERFACE_PROCESS_MANAGEMENT_METHOD to "direct".
#4 Updated by Eric Flumerfelt 6 months ago
- Status changed from Resolved to Reviewed
- Co-Assignees Eric Flumerfelt added
Code review and differential testing complete. Merged into develop.
JCF: to satisfy Issue #21919, if any process dies when we're in booted or ready, DAQInterface returns to stopped.