Configuration validation and fhiclcpp types » History » Version 26

Version 25 (Kyle Knoepfel, 08/03/2016 12:53 PM) → Version 26/28 (Kyle Knoepfel, 09/09/2016 08:22 AM)

h1. Configuration validation and @fhiclcpp@ types

The purpose of introducing @fhiclcpp@ types is to enable:
* a way to validate a user-provided, run-time configuration against a supported configuration,
* a way to provide a description of a supported configuration for a given module, service, or other plugin library,
* an improvement over using the @'pset.get<T>("key",optional_default)'@ pattern.

The @fhiclcpp@ type system was designed so that users would not need to interact directly with a @fhicl::ParameterSet@ object, but can do so if they desire.


h1. %{color:blue}Table of contents%

* [[Configuration_validation_and_fhiclcpp_types#intro|Introduction]]
** [[Configuration_validation_and_fhiclcpp_types#types|fhiclcpp types vs. FHiCL categories]]
** [[Configuration_validation_and_fhiclcpp_types#depend|Necessary header files]]
** [[Configuration_validation_and_fhiclcpp_types#usage|Usage pattern]]
### [[Configuration_validation_and_fhiclcpp_types#declare|fhiclcpp parameter declaration]]
### [[Configuration_validation_and_fhiclcpp_types#validation|ParameterSet validation]]
### [[Configuration_validation_and_fhiclcpp_types#filling|Parameter value filling]]
### [[Configuration_validation_and_fhiclcpp_types#retrieve|Parameter retrieval]]
** [[Configuration_validation_and_fhiclcpp_types#description|Configuration description]]
** [[Configuration_validation_and_fhiclcpp_types#nonart|For users outside of art]]
* [[Fhiclcpp_examples|Examples]]
* [[fhiclcpp types in detail]]
* [[Auxiliary classes]]
* [[Conditional configuration]]
* [[System details]]


h1(#intro). %{color:blue}Introduction%

h2(#types). @fhiclcpp@ types vs. FHiCL categories

The mapping between the @fhiclcpp@ and FHiCL types is shown in this table:

|{background:#fba}.*@fhiclcpp@ type*|{background:#fba}.*FHiCL category* |{background:#fba}.*FHiCL example*|
|\3{background:#ddd}. _Each of the @fhiclcpp@ types below reside in the @fhicl@ namespace._|
| *@Atom<T>@*
@OptionalAtom<T>@| atom | <pre>parameter : value</pre> |
@OptionalSequence<T>@|/3.sequence |<pre><code class="ruby">sequence : [1, 2, 3]</code></pre>|
@OptionalSequence<T,std::size_t>@|<pre><code class="ruby">sequence : [1, 2, 3] # bounded</code></pre>|
@OptionalTuple<T...>@|<pre><code class="ruby">sequence : [1, "two", true]</code></pre>|
@OptionalTable<T>@|table|<pre><code class="ruby">table : {
parameter : value
sequence : [4,5,6]

Since the FHiCL _sequence_ type is very flexible, three @fhiclcpp@ types have been introduced to improve the semantics of the C++ code. Formally speaking, the types @Sequence<T>@ and @Sequence<T, std::size_t>@ map to the same FHiCL _sequence_ category. However, @Sequence<T,std::size_t>@ is a bounded sequence, which is explained below. The @Optional*@ parameters above are described [[Fhiclcpp_types_in_detail#Optional-parameters-argument-types|here]]. <HERE>.


h2(#depend). Necessary header files

<pre><code class="cpp">
#include "fhiclcpp/types/Atom.h" // fhicl::Atom<T>
#include "fhiclcpp/types/DelegatedParameter.h" // fhicl::DelegatedParameter
#include "fhiclcpp/types/Sequence.h" // fhicl::Sequence<T> & fhicl::Sequence<T,size_t>
#include "fhiclcpp/types/Tuple.h" // fhicl::Tuple<T...>
#include "fhiclcpp/types/TupleAs.h" // fhicl::TupleAs<T(U...)>
#include "fhiclcpp/types/Table.h" // fhicl::Table<T>
#include "fhiclcpp/types/TableFragment.h" // fhicl::TableFragment<T>

#include "fhiclcpp/types/OptionalAtom.h" // fhicl::OptionalAtom<T>
#include "fhiclcpp/types/OptionalDelegatedParameter.h" // fhicl::OptionalDelegatedParameter
#include "fhiclcpp/types/OptionalSequence.h" // fhicl::OptionalSequence<T> & fhicl::OptionalSequence<T,size_t>
#include "fhiclcpp/types/OptionalTuple.h" // fhicl::OptionalTuple<T...>
#include "fhiclcpp/types/OptionalTupleAs.h" // fhicl::OptionalTupleAs<T(U...)>
#include "fhiclcpp/types/OptionalTable.h" // fhicl::OptionalTable<T>

// Headers that can be included
// - 'fhicl::Name', 'fhicl::Comment', and
// 'fhicl::use_if' and 'fhicl::use_unless' are already
// provided by each of the headers above
#include "fhiclcpp/types/Name.h"
#include "fhiclcpp/types/Comment.h"
#include "fhiclcpp/types/ConfigPredicate.h"


h2(#usage). Usage pattern

The order in which the system is used is the following:

# *@fhiclcpp@ parameter declarations*: @fhiclcpp@ parameters are declarated
# *@ParameterSet@ validation*: @fhicl::ParameterSet@ is validated against the @fhiclcpp@ parameter collection
# *Value filling*: Values are filled for @fhiclcpp@ parameters from the @ParameterSet@
# *Value retrieval*: Values are usable in C++ code

Steps 1 and 4 are exposed to the user. Steps 2 and 3 can be exposed to the user, in principle, but we do not encourage the user to do them unless there is a good reason (please consult for guidance).

For _art_ users, please consult [[art:Configuration_validation_and_description|here]]. Users outside of art should consult the link [[Configuration_validation_and_fhiclcpp_types#nonart|below]].


h2(#declare). 1. @fhiclcpp@ parameter declaration

A @fhiclcpp@ parameter declaration follows the pattern:

<pre><code class="cpp">
fhiclcpp_type < arg_type(s) > name { Name("name"), [Optional arguments] };

where @fhiclcpp_type@ is either @Atom@, @Sequence@, @Tuple@, or @Table@.

h3. @arg_type(s)@

The allowed set of @arg_type(s)@ values depends on the specific value of @fhiclcpp_type@ (and described in detail <HERE>). In general, almost any C++ type that is currently supported in a @ParameterSet::get@ call is allowed. The @std@ containers, however, are *not* allowed for any @fhiclcpp@ type:
* @std::array@
* @std::pair@
* @std::tuple@
* @std::vector@

h3. @Name("name")@

All @fhiclcpp@ types must be initialized with a @fhicl::Name@ rvalue as its first argument. The @"name"@ string corresponds to the sequence of characters corresponding to the parameter name in a FHiCL configuration:

name : "some value for name"

In general, we suggest the same character sequence for the variable name @'name'@ as the string supplied in the @fhicl::Name@. For example,
<pre><code class="cpp">
Atom<int> debugValue { Name("threshold") }; // discouraged
is discouraged as the mismatch in @debugValue@ vs. @threshold@ is likely to cause confusion when parameter value retrievals are performed (see <HERE>).

h3. Optional arguments

All optional arguments must be specified after the @fhicl::Name@ argument. Both options below can be specified with the same @fhiclcpp@ parameter declaration.

p. %{color:blue}@fhicl::Comment@%

Each of the types may include a @fhicl::Comment@ rvalue argument, whose string will be printed whenever the description of the configuration is requested:

<pre><code class="cpp">
Atom<int> param {
Name("param"), // 'Name' is NOT optional
Comment("A description") // 'Comment' is optional

p. %{color:blue}@std::function<bool()>@ - conditional configuration predicate%

In some cases, it is desirable to introduce components of a configuration only if the value of a previously-supplied parameter satisfies a given condition (i.e a predicate). This is done by the user providing an @std::function@ object whose function type receives no arguments, and whose return type is @bool@. Details as to how a user can enable this feature are given [[Conditional_configuration|here]].

p. %{color:blue}Parameter value default%

All @fhiclcpp@ types can receive a default parameter value except for @fhicl::Table@ objects. If a FHiCL configuration does not provide the parameter value, then the default value will be used. For example,
<pre><code class="cpp">
Atom<int> param { Name("param"), 4 };
</code></pre>will yield the same behavior as
<pre><code class="cpp">
auto param = pset.get<int>("param", 4);
</code></pre>whenever the value of @"parm"@ is retrieved. A significant difference, however, is that all parameter values are loaded prior to retrieval with the @fhiclcpp@-typed system.


h2(#validation). 2. Configuration validation

The validation feature is designed to catch
* missing parameters as determined by the declared @fhiclcpp@ parameters in the C++ code
* extra parameters in a user's configuration file that are not supported by any corresponding declared @fhiclcpp@ parameters

For @fhiclcpp@ parameters that receive a default value, a corresponding FHiCL parameter that is absent in a configuration file is not an error. Any misspelled FHiCL parameters -- i.e. those that do not match the string supplied in the @fhicl::Name@ argument of the @fhiclcpp@ parameter constructor -- are classified as extra parameters and are thus caught.

Consider the following allowed configuration:

<pre><code class="cpp">
struct Config {
Atom<std::string> name { Name("name"), "Gustav" };

Table<Config> pset { Name("pset") };

An attempt is made to provide a configuration to the file that uses this configuration. However the file FHiCL file is misconfigured:

<pre><code class="ruby">
# config.fcl
pset : {
nam : Mahler # attempt to override default value of "Gustav"
flag : false

The parameter @nam@ is misspelled with respect to the allowed configuration name @"name"@, and @flag@ is not supported. When the configuration validation step is performed, an exception is thrown and the following error message is printed to STDOUT:

Unsupported parameters:

+ flag [ ./config.fcl:3 ]
+ nam [ ./config.fcl:2 ]

The characters in brackets denote the location of the unsupported parameter [ @file-name:line-number@ ]. The listing order of the unsupported parameters is based on the @std::string@ ordering operator, which is lexicographical.


h2(#filling). 3. Parameter value filling

If the validation step (2) is successful -- i.e. the @fhicl::ParameterSet@ object conforms to the supported configuration as declared in the C++ code -- then the values of the parameters are filled using the specified values in the configuration file, or the default values as specified in the C++ code if the corresponding FHiCL parameter has not been provided. After this step, the @fhiclcpp@ parameter now has a value corresponding to the user-provided ones.

The parameter-value-filling step is done internally, the details of which are not necessary to understand for the user.


h2(#retrieve). 4. Parameter retrieval

h3. Standard parameters

For a standard @fhiclcpp@ parameter declared as:
<pre><code class="cpp">
Atom<int> param { Name("param"), 4 };
</code></pre>the integer value of the parameter can be retrieved using the function-call syntax @'()'@:
<pre><code class="cpp">
auto param_value = param(); // 'param_value' is int
</code></pre>assuming the supplied value in the FHiCL configuration can be decoded to the C++ type @int@.

h3. Optional parameters

For an optional @fhiclcpp@ parameter declared as:

<pre><code class="cpp">
OptionalAtom<int> optParam { Name("optParam") };
</code></pre>the integer value of the parameter can be retrieved *if present* in a configuration file by passing a variable by reference using the function-call syntax @'()'@:
<pre><code class="cpp">
int j{};
if ( optParam(j) ) {
// 'j' can now be used

If @'optParam'@ is present with a non-nil value in a configuration file, the value of @optParam(j)@ will return @true@, and @j@ will be filled with the configuration-set value. Otherwise, @j@ retains its initial value ('0' in this case). Note that optional parameters can receive no default value.

h3. General structure

The @fhiclcpp@-typed system has been designed in such a way that the parameter declaration and retrieval syntaxes are similar. For example, the FHiCL configuration:
<pre><code class="ruby">
# FHiCL parameter declarations
settings : {
verbosity : true

# FHiCL value retrieval
verbosity : @local::settings.verbosity
</code></pre>would have a @fhiclcpp@ representation of:
<pre><code class="cpp">
// fhiclcpp parameter declarations
struct Settings{
Atom<bool> verbosity { Name("verbosity") };
Table<Settings> settings { Name("settings") };

// fhiclcpp value retrieval
auto verbosity = settings().verbosity();
</code></pre>Details are [[Fhiclcpp_types_in_detail#return|here]] regarding specific return values for the different @fhiclcpp@ types.


h2(#description). Configuration description

Consider the following @fhiclcpp@ parameter configuration:
<pre><code class="cpp">
struct G4Settings {
Atom<double> energyCutoff { Name("energyCutoff") };
Atom<string> physicsList { Name("physicsList"), "QGSP_BERT" };

struct Config {
Atom<bool> flag { Name("flag"), false };
Atom<int> threshold { Name("threshold"), Comment("ADC count threshold") };
OptionalTable<G4Settings> g4Settings { Name("g4Settings") };
Sequence<string,2u> particles { Name("particles") };
Table<Config> config { Name("config") };

h3. Formation

The description of the allowed FHiCL configuration (as shown above) is formed based on the order of initialization of the class members. In addition to registering the expected key, the expected type and any user-provided defaults are also registered with the @fhiclcpp@ system. The below table shows the order of initialization and the corresponding FHiCL keys and types for each @fhiclcpp@ parameter.

|{background:#fba}.*Order*|{background:#fba}.*C++ variable* |{background:#fba}.*Registered FHiCL key*|{background:#fba}. *Registered FHiCL type*|
|={background:#ddd}. 1|@config@|@"config"@|=. table |
|={background:#ddd}.2|&nbsp;&nbsp;&nbsp;&nbsp;@flag@|@"config.flag"@|=. atom |
|={background:#ddd}.3|&nbsp;&nbsp;&nbsp;&nbsp;@threshold@|@"config.threshold"@|=. atom |
|={background:#ddd}.4|&nbsp;&nbsp;&nbsp;&nbsp;@g4Settings@|@"config.g4Settings"@|=. table |
|={background:#ddd}.5|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;@energyCutoff@|@"config.g4Settings.energyCutoff"@|=. atom |
|={background:#ddd}.6|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;@physicsList@|@"config.g4Settings.physicsList"@|=. atom |
|={background:#ddd}.7|&nbsp;&nbsp;&nbsp;&nbsp;@particles@|@"config.particles"@|=. sequence |
|={background:#ddd}.8|/2.&nbsp;&nbsp;&nbsp;&nbsp;@particles[*]@|/2.@"config.particles[*]"@|=. atom |
|={background:#ddd}.9|=. atom |

The asterisk @'*'@ means that the order of initialization of the individual @Sequence@ elements is unknown and therefore the key assignment needs to be handled specially.

h3. Print-out

For users that have access to the 'config' object, a print-out of the allowed description can be produced using:
<pre><code class="cpp">
Table<Config> config { Name("config") };
config.print_allowed_configuration(std::cout); // Any 'std::ostream' object can be provided

The printed description is sent to STDOUT:
config: {

flag: false # default

# ADC count threshold

threshold: <int>

( g4Settings: {
( energyCutoff: <double>
( physicsList: "QGSP_BERT" # default
( }

particles: [

The open parentheses (@'('@) indicate that the @g4Settings@ table is an optional table, which the user is not required to specify in the configuration file.

For _art_ users, printing out the allowed configuration is described [[art:Art|here]].


h2(#nonart). Non-@art@ users

For users outside of _art_, it is possible to use the @fhiclcpp@ types system. The following pattern is recommended:
<pre><code class="cpp">
namespace {

struct MyConfig {
// fhiclcpp parameter declarations ...

retrieveConfiguration( fhicl::ParameterSet const & pset )
std::set<std::string> ignorable_keys {}; // keys that should be ignored
// by the validation system (can be empty)

fhicl::Table<MyConfig> const result { pset, ignorable_keys }; // performs validation and value setting

return result;

} // anon. namespace

void some_func( fhicl::ParameterSet const & pset )
auto config = retrieveConfiguration( pset );
// 'config' now usable