Project

General

Profile

Configuration validation and fhiclcpp types » History » Version 7

Kyle Knoepfel, 08/26/2015 01:02 PM

1 1 Kyle Knoepfel
h1. Configuration validation and @fhiclcpp@ types
2 1 Kyle Knoepfel
3 1 Kyle Knoepfel
The purpose of introducing @fhiclcpp@ types is to enable:
4 1 Kyle Knoepfel
* a way to validate a user-provided, run-time configuration against a supported configuration,
5 1 Kyle Knoepfel
* a way to provide a description of a supported configuration for a given module, service, or other plugin library,
6 1 Kyle Knoepfel
* an improvement over using the @'pset.get<T>("key",optional_default)'@ pattern. 
7 1 Kyle Knoepfel
8 1 Kyle Knoepfel
The @fhiclcpp@ type system was designed so that users would not need to interact directly with a @fhicl::ParameterSet@ object, but can do so if they desire.
9 1 Kyle Knoepfel
10 1 Kyle Knoepfel
----
11 1 Kyle Knoepfel
12 1 Kyle Knoepfel
h1. %{color:blue}Table of contents%
13 1 Kyle Knoepfel
14 3 Kyle Knoepfel
* [[Configuration_validation_and_fhiclcpp_types#intro|Introduction]]
15 5 Kyle Knoepfel
** [[Configuration_validation_and_fhiclcpp_types#types|fhiclcpp types vs. FHiCL categories]]
16 3 Kyle Knoepfel
** [[Configuration_validation_and_fhiclcpp_types#usage|Usage pattern]]
17 3 Kyle Knoepfel
### [[Configuration_validation_and_fhiclcpp_types#declare|fhiclcpp parameter declaration]]
18 3 Kyle Knoepfel
### [[Configuration_validation_and_fhiclcpp_types#validation|ParameterSet validation]]
19 3 Kyle Knoepfel
### [[Configuration_validation_and_fhiclcpp_types#filling|Parameter value filling]]
20 3 Kyle Knoepfel
### [[Configuration_validation_and_fhiclcpp_types#retrieve|Typical parameter retrieval]]
21 3 Kyle Knoepfel
** [[Configuration_validation_and_fhiclcpp_types#description|Configuration description]]
22 7 Kyle Knoepfel
** [[Configuration_validation_and_fhiclcpp_types#nonart|For users outside of art]]
23 1 Kyle Knoepfel
* [[Fhiclcpp_examples|Examples]]
24 1 Kyle Knoepfel
* [[fhiclcpp types in detail]]
25 1 Kyle Knoepfel
* [[System details]]
26 1 Kyle Knoepfel
* [[Common compile-time errors]]
27 1 Kyle Knoepfel
* [[Upcoming features]]
28 1 Kyle Knoepfel
29 1 Kyle Knoepfel
----
30 1 Kyle Knoepfel
31 1 Kyle Knoepfel
h1(#intro). %{color:blue}Introduction%
32 1 Kyle Knoepfel
33 4 Kyle Knoepfel
h2(#types). @fhiclcpp@ types vs. FHiCL categories
34 1 Kyle Knoepfel
35 1 Kyle Knoepfel
The mapping between the @fhiclcpp@ and FHiCL types is shown in this table:
36 1 Kyle Knoepfel
37 2 Kyle Knoepfel
|{background:#fba}.*@fhiclcpp@ type*|{background:#fba}.*FHiCL category* |{background:#fba}.*FHiCL example*|
38 1 Kyle Knoepfel
|\3{background:#ddd}. _Each of the @fhiclcpp@ types below reside in the @fhicl@ namespace._|
39 1 Kyle Knoepfel
| *@Atom<T>@*| atom | <pre>parameter : value</pre> |
40 1 Kyle Knoepfel
|*@Sequence<T>@*|/3.sequence |<pre><code class="ruby">sequence : [1, 2, 3]</code></pre>|
41 1 Kyle Knoepfel
|*@Sequence<T,std::size_t>@*|<pre><code class="ruby">sequence : [1, 2, 3] # bounded</code></pre>|
42 1 Kyle Knoepfel
|*@Tuple<T...>@*|<pre><code class="ruby">sequence : [1, "two", true]</code></pre>|
43 1 Kyle Knoepfel
|*@Table<T>@*|table|<pre><code class="ruby">table : { 
44 1 Kyle Knoepfel
  parameter : value
45 1 Kyle Knoepfel
  sequence  : [4,5,6]
46 1 Kyle Knoepfel
}</code></pre>|
47 1 Kyle Knoepfel
48 1 Kyle Knoepfel
Since the FHiCL _sequence_ type is very flexible, three @fhiclcpp@ types have been introduced to improve the semantics of the C++ code.  Formally speaking, the types @Sequence<T>@ and @Sequence<T, std::size_t>@ map to the same FHiCL _sequence_ type.  However, @Sequence<T,std::size_t>@ is a bounded sequence, which is explained below.
49 1 Kyle Knoepfel
50 1 Kyle Knoepfel
----
51 1 Kyle Knoepfel
52 1 Kyle Knoepfel
h2(#usage). Usage pattern
53 1 Kyle Knoepfel
54 1 Kyle Knoepfel
The order in which the system is used is the following:
55 1 Kyle Knoepfel
56 1 Kyle Knoepfel
# *@fhiclcpp@ parameter declarations*: @fhiclcpp@ parameters are declarated
57 1 Kyle Knoepfel
# *@ParameterSet@ validation*: @fhicl::ParameterSet@ is validated against the @fhiclcpp@ parameter collection
58 1 Kyle Knoepfel
# *Value filling*: Values are filled for @fhiclcpp@ parameters from the @ParameterSet@
59 1 Kyle Knoepfel
# *Value retrieval*: Values are usable in C++ code
60 1 Kyle Knoepfel
61 1 Kyle Knoepfel
Steps 1 and 4 are exposed to the user.  Steps 2 and 3 can be exposed to the user, in principle, but we do not encourage the user to do them unless there is a good reason (please consult artists@fnal.gov for guidance).  
62 1 Kyle Knoepfel
63 7 Kyle Knoepfel
For _art_ users, please consult <HERE>.  Users outside of art should consult the link [[Configuration_validation_and_fhiclcpp_types#nonart|below]].
64 1 Kyle Knoepfel
65 1 Kyle Knoepfel
----
66 1 Kyle Knoepfel
67 1 Kyle Knoepfel
h2(#declare). 1. @fhiclcpp@ parameter declaration
68 1 Kyle Knoepfel
69 1 Kyle Knoepfel
A @fhiclcpp@ parameter declaration follows the pattern:
70 1 Kyle Knoepfel
71 1 Kyle Knoepfel
<pre><code class="cpp">
72 6 Kyle Knoepfel
fhiclcpp_type < arg_type(s) > name { Name("name"), [Optional arguments] };
73 1 Kyle Knoepfel
</code></pre>
74 1 Kyle Knoepfel
75 1 Kyle Knoepfel
where @fhiclcpp_type@ is either @Atom@, @Sequence@, @Tuple@, or @Table@.  
76 1 Kyle Knoepfel
77 1 Kyle Knoepfel
h3. @arg_type(s)@
78 1 Kyle Knoepfel
79 1 Kyle Knoepfel
The allowed set of @arg_type(s)@ values depends on the specific value of @fhiclcpp_type@ (and described in detail <HERE>).  In general, almost any C++ type that is currently supported in a @ParameterSet::get@ call is allowed.  The @std@ containers, however, are *not* allowed for any @fhiclcpp@ type:
80 1 Kyle Knoepfel
* @std::array@
81 1 Kyle Knoepfel
* @std::pair@
82 1 Kyle Knoepfel
* @std::tuple@
83 1 Kyle Knoepfel
* @std::vector@
84 1 Kyle Knoepfel
85 1 Kyle Knoepfel
(An explanation of this design decision is <HERE>.)
86 1 Kyle Knoepfel
87 6 Kyle Knoepfel
h3. @Name("name")@
88 1 Kyle Knoepfel
89 6 Kyle Knoepfel
All @fhiclcpp@ types must be initialized with a @fhicl::Name@ rvalue as its first argument.  The @"name"@ string corresponds to the sequence of characters corresponding to the parameter name in a FHiCL configuration:
90 1 Kyle Knoepfel
91 1 Kyle Knoepfel
<pre>
92 1 Kyle Knoepfel
name : "some value for name"
93 1 Kyle Knoepfel
</pre>
94 1 Kyle Knoepfel
95 6 Kyle Knoepfel
In general, we suggest the same character sequence for the variable name @'name'@ as the string supplied in the @fhicl::Name@.  For example, 
96 1 Kyle Knoepfel
<pre><code class="cpp">
97 6 Kyle Knoepfel
Atom<int> debugValue { Name("threshold") }; // discouraged
98 1 Kyle Knoepfel
</code></pre>
99 1 Kyle Knoepfel
is discouraged as the mismatch in @debugValue@ vs. @threshold@ is likely to cause confusion when parameter value retrievals are performed (see <HERE>).
100 1 Kyle Knoepfel
101 1 Kyle Knoepfel
h3. Optional arguments
102 1 Kyle Knoepfel
103 6 Kyle Knoepfel
All optional arguments must be specified after the @fhicl::Name@ argument.  Both options below can be specified with the same @fhiclcpp@ parameter declaration.
104 1 Kyle Knoepfel
105 1 Kyle Knoepfel
p. %{color:blue}@fhicl::Comment@%
106 1 Kyle Knoepfel
107 1 Kyle Knoepfel
Each of the types may include a @fhicl::Comment@ rvalue argument, whose string will be printed whenever the description of the configuration is requested:
108 1 Kyle Knoepfel
109 1 Kyle Knoepfel
<pre><code class="cpp">
110 1 Kyle Knoepfel
Atom<int> param { 
111 6 Kyle Knoepfel
  Name("param"),           // 'Name' is NOT optional
112 1 Kyle Knoepfel
  Comment("A description") // 'Comment' is optional
113 1 Kyle Knoepfel
};
114 1 Kyle Knoepfel
</code></pre>
115 1 Kyle Knoepfel
116 1 Kyle Knoepfel
p. %{color:blue}Parameter value default%
117 1 Kyle Knoepfel
118 1 Kyle Knoepfel
All @fhiclcpp@ types can receive a default parameter value except for @fhicl::Table@ objects.  If a FHiCL configuration does not provide the parameter value, then the default value will be used.  For example,
119 1 Kyle Knoepfel
<pre><code class="cpp">
120 6 Kyle Knoepfel
Atom<int> param { Name("param"), 4 };
121 1 Kyle Knoepfel
</code></pre>will yield the same behavior as 
122 1 Kyle Knoepfel
<pre><code class="cpp">
123 1 Kyle Knoepfel
auto param = pset.get<int>("param", 4);
124 1 Kyle Knoepfel
</code></pre>whenever the value of @"parm"@ is retrieved.  A significant difference, however, is that all parameter values are loaded prior to retrieval with the @fhiclcpp@-typed system.
125 1 Kyle Knoepfel
126 1 Kyle Knoepfel
----
127 1 Kyle Knoepfel
128 1 Kyle Knoepfel
h2(#validation). 2. Configuration validation
129 1 Kyle Knoepfel
130 1 Kyle Knoepfel
The validation feature is designed to catch 
131 1 Kyle Knoepfel
* missing parameters as determined by the declared @fhiclcpp@ parameters in the C++ code
132 1 Kyle Knoepfel
* extra parameters in a user's configuration file that are not supported by any corresponding declared @fhiclcpp@ parameters
133 1 Kyle Knoepfel
134 6 Kyle Knoepfel
For @fhiclcpp@ parameters that receive a default value, a corresponding FHiCL parameter that is absent in a configuration file is not an error.  Any misspelled FHiCL parameters -- i.e. those that do not match the string supplied in the @fhicl::Name@ argument of the @fhiclcpp@ parameter constructor -- are classified as extra parameters and are thus caught.
135 1 Kyle Knoepfel
136 1 Kyle Knoepfel
Consider the following allowed configuration:
137 1 Kyle Knoepfel
138 1 Kyle Knoepfel
<pre><code class="cpp">
139 1 Kyle Knoepfel
struct Config {
140 6 Kyle Knoepfel
  Atom<std::string> name { Name("name"), "Gustav" };
141 1 Kyle Knoepfel
};
142 1 Kyle Knoepfel
143 6 Kyle Knoepfel
Table<Config> pset { Name("pset") };
144 1 Kyle Knoepfel
</code></pre>
145 1 Kyle Knoepfel
146 1 Kyle Knoepfel
An attempt is made to provide a configuration to the file that uses this configuration.  However the file FHiCL file is misconfigured:
147 1 Kyle Knoepfel
148 1 Kyle Knoepfel
<pre><code class="ruby">
149 1 Kyle Knoepfel
# config.fcl
150 1 Kyle Knoepfel
pset : {
151 1 Kyle Knoepfel
   nam  : Mahler # attempt to override default value of "Gustav"
152 1 Kyle Knoepfel
   flag : false
153 1 Kyle Knoepfel
}
154 1 Kyle Knoepfel
</code></pre>
155 1 Kyle Knoepfel
156 6 Kyle Knoepfel
The parameter @nam@ is misspelled with respect to the allowed configuration name @"name"@, and @flag@ is not supported.  When the configuration validation step is performed, an exception is thrown and the following error message is printed to STDOUT:
157 1 Kyle Knoepfel
158 1 Kyle Knoepfel
<pre>
159 1 Kyle Knoepfel
Unsupported parameters:
160 1 Kyle Knoepfel
161 1 Kyle Knoepfel
 + flag                           [ ./config.fcl:3 ]
162 1 Kyle Knoepfel
 + nam                            [ ./config.fcl:2 ]
163 1 Kyle Knoepfel
</pre>
164 1 Kyle Knoepfel
165 1 Kyle Knoepfel
The characters in brackets denote the location of the unsupported parameter [ @file-name:line-number@ ].  The listing order of the unsupported parameters is based on the @std::string@ ordering operator, which is lexicographical.
166 1 Kyle Knoepfel
167 1 Kyle Knoepfel
----
168 1 Kyle Knoepfel
169 1 Kyle Knoepfel
h2(#filling). 3. Parameter value filling
170 1 Kyle Knoepfel
171 1 Kyle Knoepfel
If the validation step (2) is successful -- i.e. the @fhicl::ParameterSet@ object conforms to the supported configuration as declared in the C++ code -- then the values of the parameters are filled using the specified values in the configuration file, or the default values as specified in the C++ code if the corresponding FHiCL parameter has not been provided.  After this step, the @fhiclcpp@ parameter now has a value corresponding to the user-provided ones.
172 1 Kyle Knoepfel
173 1 Kyle Knoepfel
The parameter-value-filling step is done internally, the details of which are not necessary to understand for the user.
174 1 Kyle Knoepfel
175 1 Kyle Knoepfel
----
176 1 Kyle Knoepfel
177 1 Kyle Knoepfel
h2(#retrieve). 4. Typical parameter retrieval
178 1 Kyle Knoepfel
179 1 Kyle Knoepfel
For a @fhiclcpp@ parameter declared as:
180 1 Kyle Knoepfel
<pre><code class="cpp">
181 6 Kyle Knoepfel
Atom<int> param { Name("param"), 4 };
182 1 Kyle Knoepfel
</code></pre>the integer value of the parameter can be retrieved using the function call syntax @'()'@:
183 1 Kyle Knoepfel
<pre><code class="cpp">
184 1 Kyle Knoepfel
auto param_value = param(); // 'param_value' is int
185 1 Kyle Knoepfel
</code></pre>assuming the supplied value in the FHiCL configuration can be decoded to the C++ type @int@.
186 1 Kyle Knoepfel
187 1 Kyle Knoepfel
The @fhiclcpp@-typed system has been designed in such a way that the parameter declaration and retrieval syntaxes are similar.  For example, the FHiCL configuration:
188 1 Kyle Knoepfel
<pre><code class="ruby">
189 1 Kyle Knoepfel
# FHiCL parameter declarations
190 1 Kyle Knoepfel
settings : {
191 1 Kyle Knoepfel
  verbosity : true
192 1 Kyle Knoepfel
}
193 1 Kyle Knoepfel
194 1 Kyle Knoepfel
# FHiCL value retrieval
195 1 Kyle Knoepfel
verbosity : @local::settings.verbosity
196 1 Kyle Knoepfel
</code></pre>would have a @fhiclcpp@ representation of:
197 1 Kyle Knoepfel
<pre><code class="cpp">
198 1 Kyle Knoepfel
// fhiclcpp parameter declarations
199 1 Kyle Knoepfel
struct Settings{
200 6 Kyle Knoepfel
  Atom<bool> verbosity { Name("verbosity") };
201 1 Kyle Knoepfel
};
202 6 Kyle Knoepfel
Table<Settings> settings { Name("settings") };
203 1 Kyle Knoepfel
204 1 Kyle Knoepfel
// fhiclcpp value retrieval
205 1 Kyle Knoepfel
auto verbosity = settings().verbosity();
206 1 Kyle Knoepfel
</code></pre>Details are [[Fhiclcpp_types_in_detail#return|here]] regarding specific return values for the different @fhiclcpp@ types.
207 1 Kyle Knoepfel
208 1 Kyle Knoepfel
----
209 1 Kyle Knoepfel
210 1 Kyle Knoepfel
h2(#description). Configuration description
211 1 Kyle Knoepfel
212 1 Kyle Knoepfel
Consider the following @fhiclcpp@ parameter configuration:
213 1 Kyle Knoepfel
<pre><code class="cpp">
214 1 Kyle Knoepfel
struct G4Settings {
215 6 Kyle Knoepfel
  Atom<double> energyCutoff { Name("energyCutoff") };
216 6 Kyle Knoepfel
  Atom<string> physicsList  { Name("physicsList"), "QGSP_BERT" };
217 1 Kyle Knoepfel
};
218 1 Kyle Knoepfel
219 1 Kyle Knoepfel
struct Config {
220 6 Kyle Knoepfel
  Atom<bool>          flag       { Name("flag"), false };
221 6 Kyle Knoepfel
  Atom<int>           threshold  { Name("threshold"), Comment("ADC count threshold") };
222 6 Kyle Knoepfel
  Table<G4Settings>   g4Settings { Name("g4Settings") }; 
223 6 Kyle Knoepfel
  Sequence<string,2>  particles  { Name("particles")  };
224 1 Kyle Knoepfel
};
225 6 Kyle Knoepfel
Table<Config> config { Name("config") };
226 1 Kyle Knoepfel
</code></pre>
227 1 Kyle Knoepfel
228 1 Kyle Knoepfel
h3. Formation
229 1 Kyle Knoepfel
230 1 Kyle Knoepfel
The description of the allowed FHiCL configuration (as shown above) is formed based on the order of initialization of the class members.  In addition to registering the expected key, the expected type and any user-provided defaults are also registered with the @fhiclcpp@ system.  The below table shows the order of initialization and the corresponding FHiCL keys and types for each @fhiclcpp@ parameter.
231 1 Kyle Knoepfel
232 1 Kyle Knoepfel
|{background:#fba}.*Order*|{background:#fba}.*C++ variable* |{background:#fba}.*Registered FHiCL key*|{background:#fba}. *Registered FHiCL type*|
233 1 Kyle Knoepfel
|={background:#ddd}. 1|@config@|@"config"@|=. table |
234 1 Kyle Knoepfel
|={background:#ddd}.2|&nbsp;&nbsp;&nbsp;&nbsp;@flag@|@"config.flag"@|=. atom |
235 1 Kyle Knoepfel
|={background:#ddd}.3|&nbsp;&nbsp;&nbsp;&nbsp;@threshold@|@"config.threshold"@|=. atom |
236 1 Kyle Knoepfel
|={background:#ddd}.4|&nbsp;&nbsp;&nbsp;&nbsp;@g4Settings@|@"config.g4Settings"@|=. table |
237 1 Kyle Knoepfel
|={background:#ddd}.5|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;@energyCutoff@|@"config.g4Settings.energyCutoff"@|=. atom |
238 1 Kyle Knoepfel
|={background:#ddd}.6|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;@physicsList@|@"config.g4Settings.physicsList"@|=. atom |
239 1 Kyle Knoepfel
|={background:#ddd}.7|&nbsp;&nbsp;&nbsp;&nbsp;@particles@|@"config.particles"@|=. sequence |
240 1 Kyle Knoepfel
|={background:#ddd}.8|/2.&nbsp;&nbsp;&nbsp;&nbsp;@particles[*]@|/2.@"config.particles[*]"@|=. atom |
241 1 Kyle Knoepfel
|={background:#ddd}.9|=. atom |
242 1 Kyle Knoepfel
243 1 Kyle Knoepfel
The asterisk @'*'@ means that the order of initialization of the individual @Sequence@ elements is unknown and therefore the key assignment needs to be handled specially.
244 1 Kyle Knoepfel
245 1 Kyle Knoepfel
h3. Print-out
246 1 Kyle Knoepfel
247 1 Kyle Knoepfel
For users that have access to the 'config' object, a print-out of the allowed description can be produced using:
248 1 Kyle Knoepfel
<pre><code class="cpp">
249 1 Kyle Knoepfel
Table<Config> config { Name("config") };
250 1 Kyle Knoepfel
config.print_reference(std::cout); // Any 'std::ostream' object can be provided
251 1 Kyle Knoepfel
</code></pre>
252 1 Kyle Knoepfel
253 1 Kyle Knoepfel
The printed description is sent to STDOUT:
254 1 Kyle Knoepfel
<pre>
255 1 Kyle Knoepfel
   config : {
256 1 Kyle Knoepfel
257 1 Kyle Knoepfel
      flag : false  # default
258 1 Kyle Knoepfel
259 1 Kyle Knoepfel
      # ADC count threshold
260 1 Kyle Knoepfel
261 1 Kyle Knoepfel
      threshold : <int>
262 1 Kyle Knoepfel
263 1 Kyle Knoepfel
      g4Settings : {
264 1 Kyle Knoepfel
265 1 Kyle Knoepfel
         energyCutoff : <double>
266 1 Kyle Knoepfel
267 1 Kyle Knoepfel
         physicsList : "QGSP_BERT"  # default
268 1 Kyle Knoepfel
      }
269 1 Kyle Knoepfel
270 1 Kyle Knoepfel
      particles : [
271 1 Kyle Knoepfel
         <string>,
272 1 Kyle Knoepfel
         <string>
273 1 Kyle Knoepfel
      ]
274 1 Kyle Knoepfel
   }
275 1 Kyle Knoepfel
</pre>
276 1 Kyle Knoepfel
277 1 Kyle Knoepfel
For _art_ users, printing out the allowed configuration is described <HERE>.
278 7 Kyle Knoepfel
279 7 Kyle Knoepfel
----
280 7 Kyle Knoepfel
281 7 Kyle Knoepfel
h2(#nonart). Non-@art@ users
282 7 Kyle Knoepfel
283 7 Kyle Knoepfel
For users outside of _art_, it is possible to use the @fhiclcpp@ types system.  The following pattern is recommended:
284 7 Kyle Knoepfel
<pre><code class="cpp">
285 7 Kyle Knoepfel
namespace {
286 7 Kyle Knoepfel
287 7 Kyle Knoepfel
  struct MyConfig {
288 7 Kyle Knoepfel
    // fhiclcpp parameter declarations ...
289 7 Kyle Knoepfel
  };
290 7 Kyle Knoepfel
291 7 Kyle Knoepfel
  fhicl::Table<MyConfig>
292 7 Kyle Knoepfel
  retrieveConfiguration( fhicl::ParameterSet const & pset )
293 7 Kyle Knoepfel
  {
294 7 Kyle Knoepfel
     std::set<std::string> ignorable_keys {}; // keys that should be ignored 
295 7 Kyle Knoepfel
                                              // by the validation system (can be empty)
296 7 Kyle Knoepfel
297 7 Kyle Knoepfel
     fhicl::Table<MyConfig> const result { pset, ignorable_keys }; // performs validation and value setting
298 7 Kyle Knoepfel
299 7 Kyle Knoepfel
     return result;
300 7 Kyle Knoepfel
  }
301 7 Kyle Knoepfel
302 7 Kyle Knoepfel
} // anon. namespace
303 7 Kyle Knoepfel
304 7 Kyle Knoepfel
void some_func( fhicl::ParameterSet const & pset )
305 7 Kyle Knoepfel
{
306 7 Kyle Knoepfel
  auto config = retrieveConfiguration( pset );
307 7 Kyle Knoepfel
  // 'config' now usable
308 7 Kyle Knoepfel
}
309 7 Kyle Knoepfel
310 7 Kyle Knoepfel
</code></pre>