Feature #20772

Inherit Configuration from Provenance

Added by Alexander Himmel over 2 years ago. Updated over 2 years ago.

Under Discussion
Target version:
Start date:
Due date:
% Done:


Estimated time:
SSI Package:


A problem I run into frequently with the photon simulation and reconstruction in DUNE is the need to have services being configured in consistent ways in separate processing steps. Some examples: keeping a consistent Geometry (Geometry service), needing to know the "pre-scale" factors applied during photon simulation when applying quantum efficiency later (LArProperties service), reconstructing optical signals using visibility they were simulated with (PhotonVisibilityService). The use cases I have thought up all relate to services, but I can imagine this might apply to modules as well.

There is a work-around available now, but I think it's a bad one. For each service, you could write a data product into the (run?) record, and then read it back in when the service is configured. But, I think this is a poor solution, since it requires a lot of extra work which needs to be repeated for each service, and requires having Fhcil parameters which tell the configuration to ignore other fhicl parameters, which means you will store provenance information which doesn't actually correspond to what was used in the job.

I think there's a much cleaner possible framework solution, which is adding fhicl configuration commands which specify that configuration should come from the file provenance. Ideally, after such a command you would then be able to override specific options using the same last-wins system already in use in fhicl.

For example, a fhicl configuration like this:

services: {
  LArPropertiesService: @file::g4.LArPropertiesService
services.LArPropertiesServices.ScintYield: 24500

would tell ART to load the LArPropertiesService configuration used in the g4 stage of the file, and then override one setting. The ability to use "last" if a particular processing stage wasn't specified I think would also be useful, since a user often does not know the detailed processing history of a file they are working with.

This of course is not totally foolproof backwards compatibility -- if a service changes how particular set of fhicl parameters are interpreted, or adds or removes some, then things will still break. But, if we know about this design then we can plan around it when designing and changing service configurations, and I think it is still a much better solution than the alternative.


#1 Updated by Alexander Himmel over 2 years ago

If you'd like me to come present this idea at a Stakeholder's meeting, just let me know and I would be happy to.

#2 Updated by Kyle Knoepfel over 2 years ago

  • Status changed from New to Feedback

A fundamental flaw in this proposal is that services, modules, and sources must be configured before the first file is opened--i.e. the configuration initializes the entire program before it is executed. If consistent configuration is needed, there are substitution and splicing facilities within the FHiCL language that should be used for this purpose. Please stop by WH9SW to discuss.

#3 Updated by Kyle Knoepfel over 2 years ago

  • Status changed from Feedback to Under Discussion
  • SSI Package - added

Alex Himmel, Marc Paterno, Chris Green, and I met to discuss the feasibility of this proposal. Salient points of the discussion:

  • The need to correctly configure an art process to correctly interpret the data stored on disk is fragile. It is a result of relegating too much of the physics algorithms to services. In order to avoid this problem in the future:
    • Relevant physics data should be stored as a data product (in some as-of-yet undetermined data aggregation level), without the need to rely on the configuration settings of a previous process to interpret the physics.
    • Examples of DUNE code should be developed in concert with the SciSoft team to demonstrate healthy use of art constructs, and robust coding patterns, in general.
  • In the short term, a temporary solution will be pursued outside of the art executable, which allows a consistent configuration to be presented to art at the command-line. This would involve wrapping the art executable in some sort of script that would first run config_dumper on the first input file to be read, do some manipulations to ensure a consistent configuration wrt the current process, and then present this post-processed file to the art executable.

An additional note: the grammar of the FHiCL language is independent of any art abstractions. Introducing FHiCL grammar that only has meaning in the context of art would invert the intended dependency. If art were to adopt a configuration-consistency validation mechanism, it would need to be in the form of standard FHiCL parameter assignment. I do not suggest going this route, due to the aforementioned complications listed above, but I wanted to mention this because particular FHiCL syntax was suggested.

Also available in: Atom PDF