Project

General

Profile

Feature #4361

Feature #4030: For DS-50 DAQ/artdaq, support reconfiguration without restarting the MPI program

Implement the selective disablement/renablement of modules [ds50daq-related]

Added by Kurt Biery over 6 years ago. Updated over 5 years ago.

Status:
Closed
Priority:
High
Category:
Navigation
Target version:
Start date:
07/17/2013
Due date:
07/31/2013
% Done:

100%

Estimated time:
Spent time:
Scope:
External
Experiment:
DarkSide
SSI Package:
art
Duration: 15

Description

There is a request from ds50daq folks to allow reconfiguration of the art thread without restarting the artdaq/ds50daq processes (that is, without restarting the MPI program).

Various models for doing this have been discussed, and one of the alternatives is to provide the ability to enable or disable paths. In this model, the superset of all interesting configured paths would be specified at startup, and the ones of interest at any given time would be enabled.

My understanding is that the ability to change parameters within a given art module already exists, although I'll confess that I'm not sure how the provenance tracking works in that case.

This Issue deals with the code changes that allow modules (or paths?) to be enabled and disabled.

A question that remains is whether the enabling and disabling should be limited to times when the system is not Running (that is, between runs).


Related issues

Related to art - Feature #2946: SelectEvents for analyzer modulesClosed10/10/201209/30/2013

Related to art - Feature #3190: Allow for reconfiguration in a DAQ environmentClosed12/18/201209/30/2013

Related to artdaq - Feature #5960: Make use of the ability to reconfigure modules in artClosed04/18/2014

History

#1 Updated by Christopher Green over 6 years ago

  • Category set to Navigation
  • Status changed from New to Feedback
  • Scope set to External
  • Experiment DarkSide added

I have a proposal:

On the art side, we can have a service which will trigger the reconfiguration of paths based on a call from (e.g.) an artdaq service which would (e.g.) receive a message from the run control system. This would consist of a name of a trigger path or end-path module to disable or enable. No new paths may be created, and the module-makeup of a trigger path may not be altered. Disablement of paths would trigger on an event boundary.

Regarding the possible changing of pre-scaling for analyzers: is this still an issue? Currently analyzers do not honor path results anyway (and therefore the result of a pre-scaling filter), unless they do so internally. If this is for a small number of analyzers only, I would suggest implementing a configurable pre-scaling facility within the analyze() (possibly using an algorithm library shared across several modules). We can trigger the reconfiguration of said module or modules through the same mechanism as the path reconfiguration. I would strongly recommend against using standard pre-scaling filters to control the operation of the analyzer(s) even after this facility becomes available because of the effect on the accuracy of the provenance due to the reconfiguration. The configuration of analyzers is not tracked by data product provenances, so this does not have the same implications.

Does this sound acceptable? Please let me know ASAP so I have a firm direction.

#2 Updated by Christopher Green over 6 years ago

On 7/23/13 8:44 PM, Kurt Biery wrote:

Hi Chris,

Hi,

Without understanding all of the details, the art service that you describe sounds reasonable to me. A couple of questions:

is this similar to what people were thinking previously for the way to implement this? (I don't really care; I'm just curious.)

Jim had a very similar idea which would have handled only end paths, but this takes advantage of changes made recently to the way that paths and modules are made and managed in order to disable trigger paths also.

for the artdaq (or other) service that you mention - is that something that would watch for messages on a shared queue or something like that? Basically, I'm curious how the external system would feed the request into art.

That's completely open, actually -- anything that involves polling rather than blocking receives: a module would check for such a message during its analyze() phase, for example, and then poke the art service which will handle the reconfiguration for the next event.

I trust that there are no gotchas in what you describe. That is, we will be able to enable/disable disk writing and individual online monitoring modules. (And, I understand that we may need to make sure that each of the online monitoring modules has their own path in order to get module granularity.)

So do I! You don't need to do separate end paths though, because it's all combined into one path when we parse the configuration: it will be possible to deal with end path modules on a per-module basis.

Alessandro,
Please comment on the subject of pre-scaling for analyzers.
Thanks,
Kurt

#3 Updated by Christopher Green over 6 years ago

  • Status changed from Feedback to Resolved
  • Target version set to 1.08.00
  • % Done changed from 0 to 100
  • SSI Package art added

Implemented per proposal. See End-path module and trigger path disablement for description. The service artdaq/artdaq/ArtModules/InFlightConfiguration{.h,_service.cc} is a suitable skeleton on which to build the messaging and decision-making necessary to actually disable and reconfigure paths and modules.

A forthcoming addendum to this feature, although not necessary to use it: the reconfiguration of a module shall result in an entry in the SQLite metadata database for the file to the effect that a named module was reconfigured, thereby rendering metadata suspect for the rest of the file.

To clarify the situation regarding end-path modules and pre-scalers:

It has been a longstanding feature request to implement path selection for analyzers, but due to manpower and budget constraints this has not been done yet. If it were agreed to be something that DS50 needed and was willing to pay for, I'm sure the timescale would shift. It is certainly possible for a user to implement trigger path information checking on a per-module basis, but this is admittedly fiddly.

#4 Updated by Lynn Garren over 5 years ago

  • Status changed from Resolved to Closed


Also available in: Atom PDF