Project

General

Profile

Bug #21919

If a process dies or enters an error state during boot or config, the run should not proceed

Added by John Freeman 7 months ago. Updated 6 months ago.

Status:
Reviewed
Priority:
Normal
Assignee:
Category:
-
Target version:
-
Start date:
02/18/2019
Due date:
% Done:

100%

Estimated time:
Experiment:
-
Co-Assignees:
Duration:

Description

As described in Issue #21695, unless explicitly told otherwise DAQInterface will continue running if a process dies or throws an exception. While this is a good idea from the perspective of keeping a run going, the way DAQInterface is currently coded up, not only will it not break out of the "running" state if a process dies or goes into error, it also won't break out of the "configuring" state either! It seems reasonable that if something goes wrong before we even begin running, that's grounds for DAQInterface refusing to enter the running state.

Associated revisions

Revision 10c6499c (diff)
Added by John Freeman 7 months ago

JCF: to satisfy Issue #21919, if any process dies when we're in booted or ready, DAQInterface returns to stopped.

History

#1 Updated by John Freeman 7 months ago

I'll narrow the scope of this Issue, as I was overly pessimistic about the behavior of DAQInterface in the event of artdaq process problems before running. The remaining concern before running is if an artdaq process we want in a run dies, DAQInterface should refuse to proceed to the run. If, however, an artdaq process undergoing the init or the start transitions dies or throws an exception, DAQInterface will already refuse to proceed to the running state. E.g., if an exception is thrown in a fragment generator's constructor or start() function, you'll see:

Exception: An exception was thrown during the Init transition as at least one of the artdaq processes didn't achieve its desired state

and DAQInterface will enter recovery. And if the boardreader dies in the fragment generator's constructor or start() function, you'll see something like:
error: [Errno 111] Connection refused

Exception caught sending Start transition to artdaq process component02 at localhost:11100 

#2 Updated by John Freeman 7 months ago

  • % Done changed from 0 to 100
  • Status changed from New to Resolved

With commit 10c6499c41f9e4b21c2e9aeaab98fb4e1c0beb68 on the branch feature/issue21919_dont_start_run_if_problem, it's now the case that if we're not in the running state, any artdaq process dying will result in DAQInterface returning to the stopped state.

#3 Updated by John Freeman 7 months ago

Crucial point: the change in behavior here applies when we've set DAQINTERFACE_PROCESS_MANAGEMENT_METHOD to "direct".

#4 Updated by Eric Flumerfelt 6 months ago

  • Status changed from Resolved to Reviewed
  • Co-Assignees Eric Flumerfelt added

Code review and differential testing complete. Merged into develop.



Also available in: Atom PDF