When a critical artdaq process dies AND the trace script returns nonzero, DAQInterface hangs
On December 6th, icarus' run 3894 hung for the following reason:
- A boardreader process died
- As expected, upon realizing this, DAQInterface entered the recover transition, designed to wind down the artdaq processes as cleanly as possible
- Since a trace script was meant to be executed on start, stop and recover (referred to by the DAQINTERFACE_TRACE_SCRIPT variable), DAQInterface called it
- For reasons irrelevant to this issue, the trace script returned nonzero, i.e., an error state
- DAQInterface responded by throwing an exception which abruptly cancelled the recover transition and was uncaught
The next time (A) a critical process dies, and (B) the trace script returns in an error state, DAQInterface should continue winding down cleanly.