Feature #10410
Request option for first-match-in-LD_LIBRARY_PATH for resolving art plugins
0%
Description
Consider the case that LD_LIBRARY_PATH contains two module .so files named lib*_ABC_module.so. Also consider that the .fcl file contains the following module configuration:
label : {
module_type : ABC
}
Present behaviour is that art will scan all for all _module.so files in LD_LIBRARY_PATH and recognize that there are two matches for ABC; it will issue a diagnostic message and perform a graceful shutdown. If the two .so files differ in the * part of the file name, then one can specify which of the two modules is desired by:
label : {
module_type : full_relative_path_to_file_ABC_module.so
}
where the path is relative to LD_LIBRARY_PATH and where all /'s in the path are changed to underscore characters. If the two .so files are indentically named there is no way to tell art which one to load.
We would like to request that a different behaviour be provided as an option that is user selectable ( perhaps from the command line, from within the .fcl file or both ). The different behaviour is for first match wins when resolving a module_type. We would like an informational message saying that
art saw more than one option and that it chose the first one.
The use case we have in mind is that someone is doing development work on a module that is part of the base release; frequently it is very convenient to do this with a partial build. We recognize that this is potentially dangerous but we believe that it is our choice to pick our operating point in the speed vs danger space.
When we first requested the opposite behaviour we were focused on the use case of two packages each containing Test_module.so, or similar. We would like the default to remain with the current behaviour, because we believe that this use case is still important. But we feel that expert users should be free to remove saftey features in the interest of speeding up development.
History
#1 Updated by Kyle Knoepfel almost 4 years ago
First, there is an error in how you attempt to disambiguate module libraries. The correct specification for your particular example is:
label : {
module_type : "full/relative/path/to/file/ABC"
}
Second, what is the particular difficulty with the above specification? If it is trying to figure out what the specification should be, art has the --module-description option which will help you here. To illustrate, if I want to print all of the modules that are available, I type:
[knoepfel@woof build-art-prof]$ art --print-available-modules ======================================================================================================================================= module_type Type Source location --------------------------------------------------------------------------------------------------------------------------------------- [...] HMRunProdProducer producer /home/knoepfel/art/test/Integration/high-memory/HMRunProdProducer_module.cc HMSubRunProdProducer producer /home/knoepfel/art/test/Integration/high-memory/HMSubRunProdProducer_module.cc InputProducer producer /home/knoepfel/art/test/Integration/event-shape/InputProducer_module.cc ***InputProducerNoEvents producer /home/knoepfel/art/test/Integration/event-shape/InputProducerNoEvents_module.cc ***InputProducerNoEvents producer /home/knoepfel/art/test/Integration/run-subrun-shape/InputProducerNoEvents_module.cc InputProducerOnlyEvents producer /home/knoepfel/art/test/Integration/event-shape/InputProducerOnlyEvents_module.cc [...] =======================================================================================================================================
There are two instances of InputProducerNoEvents, as flagged by the ***s that precede them. As you've mentioned, one will need to use the long specification so art can determine which one to use. To determine what the long specification is, use the --module-description option followed by the short specification:
[knoepfel@woof build-art-prof]$ art --module-description InputProducerNoEvents ====================================================================================================module_type: InputProducerNoEvents (or "test/Integration/event-shape/InputProducerNoEvents")
type : producer
source : /home/knoepfel/art/test/Integration/event-shape/InputProducerNoEvents_module.cc
library: /home/knoepfel/scratch/build-art-prof/lib/libtest_Integration_event-shape_InputProducerNoEvents_module.so Allowed configuration
--------------------- [ None provided ]====================================================================================================
module_type: InputProducerNoEvents (or "test/Integration/run-subrun-shape/InputProducerNoEvents")
type : producer
source : /home/knoepfel/art/test/Integration/run-subrun-shape/InputProducerNoEvents_module.cc
library: /home/knoepfel/scratch/build-art-prof/lib/libtest_Integration_run-subrun-shape_InputProducerNoEvents_module.so Allowed configuration
--------------------- [ None provided ]====================================================================================================
where the relevant lines have been highlighted.
I have not consulted with the other art developers on this. But before we discuss this as a group, I wanted to make you aware of some features in art that provide help in handling situations like this, in case you were not already aware of them. Please let us know it the aforementioned features do not alleviate any of the difficulties you have listed as a motivation for wanting a relaxed library-lookup facility.
#2 Updated by Kyle Knoepfel almost 4 years ago
Another thought occurred to me. Whereas an exception is thrown if two modules with the same short spec cannot be disambiguated (due to both being on an LD_LIBRARY_PATH path), the same behavior is not true for FHICL_FILE_PATH. In particular, suppose you have:
/path1/to/some_config.fcl/path2/to/some_config.fcl, andFHICL_FILE_PATH=/path2/to:/path1/to
Whenever you run art -c some_config.fcl, each path on FHICL_FILE_PATH is searched in the specified order for the filename some_config.fcl. The first match wins. No exception is thrown if there are multiple matches.
For your particular use case, if you have a work area in which you want to develop one of the mainline modules, but are worried about having to adjust FHiCL file names in scripts in the base release, which you do not have fully checked out, then the following should work:
- Work on local copy of particular module:
/your/current/work/area/MyModule_module.cc - Get local copy of FHiCL file referenced in base-release script
- Adjust local copy of FHiCL file to avoid name collision
module_type: "long/spec/to/avoid/collision/MyModule"
- Add current work area to top of
FHICL_FILE_PATH:
FHICL_FILE_PATH=/your/current/work/area:$FHICL_FILE_PATH
- Run script
Would this meet your needs?
#3 Updated by Christopher Green almost 4 years ago
- Category set to Infrastructure
- Status changed from New to Feedback
- Estimated time set to 4.00 h
- SSI Package art added
- SSI Package deleted (
)
We would propose to allow this non-default option to be set via command line only: if the relevant configuration option is present in the configuration prior to post-processing, it will be ignored with a warning. Does this sound reasonable?
#4 Updated by Rob Kutschke almost 4 years ago
With one small change that works for us. Our preference is an error that stops execution, not a warning.